Stream uploads directly into the CDC chunk store (no spool, single write)

Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.

The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):

- Each batch of distinct chunks is pinned-or-classified by ONE
  `UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
  be reclaimed mid-upload), and only chunks the store doesn't have are
  written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
  then one batched INSERT, then the manifest. Identical concurrent
  uploads are resolved at the manifest INSERT via ON CONFLICT (the
  loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
  chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
  MD5/SHA-256 checksums are computed by a stream tee — the post-upload
  re-read of the assembled file is gone.

All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.

Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.

Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:06:33 +00:00
parent 7157454afd
commit e3f04d58aa
34 changed files with 1864 additions and 2440 deletions
Generated
+5 -1
View File
@@ -1906,6 +1906,11 @@ name = "fastcdc"
version = "4.0.1" version = "4.0.1"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77af40d8a8dadb92dc178569a5f5edb5f3056e98255c2de48ab5d59a52892e0c" checksum = "77af40d8a8dadb92dc178569a5f5edb5f3056e98255c2de48ab5d59a52892e0c"
dependencies = [
"async-stream",
"tokio",
"tokio-stream",
]
[[package]] [[package]]
name = "fastrand" name = "fastrand"
@@ -3824,7 +3829,6 @@ dependencies = [
"lightningcss", "lightningcss",
"lru", "lru",
"md-5 0.11.0", "md-5 0.11.0",
"memmap2",
"mimalloc", "mimalloc",
"mime_guess", "mime_guess",
"mockall", "mockall",
+1 -2
View File
@@ -68,8 +68,7 @@ azure_storage = { version = "0.21", default-features = false, features = ["enabl
azure_storage_blobs = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] } azure_storage_blobs = { version = "0.21", default-features = false, features = ["enable_reqwest_rustls", "hmac_rust"] }
aes-gcm = "0.10.3" aes-gcm = "0.10.3"
lru = "0.16.4" lru = "0.16.4"
fastcdc = "4.0.0" fastcdc = { version = "4.0.0", features = ["tokio"] }
memmap2 = "0.9.10"
lettre = { version = "0.11.18", default-features = false, features = ["smtp-transport", "tokio1-rustls-tls", "rustls-native-certs", "builder"] } lettre = { version = "0.11.18", default-features = false, features = ["smtp-transport", "tokio1-rustls-tls", "rustls-native-certs", "builder"] }
idna = "1.1" idna = "1.1"
smol_str = { version = "0.3.2", features = ["serde"] } smol_str = { version = "0.3.2", features = ["serde"] }
+1 -2
View File
@@ -14,8 +14,7 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
| `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Whole-file size ceiling, in bytes (10 GB on 64-bit, 1 GB on 32-bit). Applies to BOTH direct PUTs (per-request body) and chunked uploads (declared `total_size`, checked upfront at session creation). | | `OXICLOUD_MAX_UPLOAD_SIZE` | `10737418240` | Whole-file size ceiling, in bytes (10 GB on 64-bit, 1 GB on 32-bit). Applies to BOTH direct PUTs (per-request body) and chunked uploads (declared `total_size`, checked upfront at session creation). |
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | `1073741824` | Per-request cap for non-chunked PUT bodies, in bytes (1 GiB). Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol (resumable on failure). See [Storage Fine Tuning](./storage-fine-tuning.md). | | `OXICLOUD_DIRECT_PUT_MAX_BYTES` | `1073741824` | Per-request cap for non-chunked PUT bodies, in bytes (1 GiB). Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol (resumable on failure). See [Storage Fine Tuning](./storage-fine-tuning.md). |
| `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Per-chunk cap, independent of `OXICLOUD_MAX_UPLOAD_SIZE` (whole-file cap). See [Storage Fine Tuning](./storage-fine-tuning.md). | | `OXICLOUD_CHUNK_MAX_BYTES` | `104857600` | Maximum size of a single chunked-upload PUT in bytes (100 MB). Per-chunk cap, independent of `OXICLOUD_MAX_UPLOAD_SIZE` (whole-file cap). See [Storage Fine Tuning](./storage-fine-tuning.md). |
| `OXICLOUD_UPLOAD_TMPDIR` | (OS temp dir) | Spool directory for non-chunked PUT bodies (`/api/files/upload`, WebDAV PUT). Point at a real-disk path on the same FS as `.blobs/` to avoid tmpfs OOMKill and make blob promotion an atomic rename. See [Storage Fine Tuning](./storage-fine-tuning.md). | | `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Direct (non-chunked) uploads stream straight into the blob store and need no spool directory. Placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Root directory for chunked-upload sessions (REST + NextCloud). Same-FS / NVMe placement guidance: see [Storage Fine Tuning](./storage-fine-tuning.md). |
| `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. | | `OXICLOUD_REUSE_PORT` | `false` | Enable `SO_REUSEPORT` so multiple processes can share the same port. **Disabled by default** — a second accidental instance will fail with "address already in use". Enable only for deliberate multi-worker setups (process supervisor, rolling restart). Not supported on Windows. |
## Database ## Database
+83 -216
View File
@@ -1,62 +1,56 @@
# Storage Fine Tuning # Storage Fine Tuning
This page is for sysadmins who want to tune **where** OxiCloud spools This page is for sysadmins who want to tune **where** OxiCloud places
upload bodies and **why** the placement matters for throughput and upload data on disk and **why** the placement matters for throughput
memory. The defaults work; the gains from a tuned layout are and memory. The defaults work; the gains from a tuned layout are
significant on busy instances or constrained containers. significant on busy instances or constrained containers.
## The upload lifecycle in 30 seconds ## The upload lifecycle in 30 seconds
Every upload moves through two stages:
``` ```
┌─── direct (single-PUT) upload ────────┐ ┌─── direct (single-PUT / multipart) ───┐
client ─┤ ├──► OxiCloud accepts the client ─┤ │ Streamed DIRECTLY into the
└─── multi-chunk upload │ bytes into a SPOOL on │ ├──► content-addressable store:
(`/api/uploads` / │ local disk. │ │ CDC chunking + BLAKE3 +
`/dav/uploads/...`) │ │ │ dedup checks happen while
│ Direct upload → OXICLOUD_UPLOAD_TMPDIR │ │ the bytes arrive. No spool
│ Chunked upload → OXICLOUD_CHUNK_DIR │ │ file, no re-read; chunks
│ │ │ the store already has are
▼ │ │ never written at all.
│ │
└─── multi-chunk upload ────────────────┤ Chunk PARTS accumulate on
(`/api/uploads` / │ disk under OXICLOUD_CHUNK_DIR
`/dav/uploads/...`) │ until /complete, which
│ streams them (in order)
▼ through the same CDC path.
┌─────────────────────────┐ ┌─────────────────────────┐
│ Once the upload is │ │ STORAGE BACKEND │
│ complete (and verified │ │ • local FS (.blobs/) │
│ if a checksum was │ │ • S3-compatible │
│ supplied), OxiCloud │ │ • Azure Blob │
│ MOVES the assembled │
│ blob into the configured│
│ STORAGE BACKEND: │
│ │
│ • local FS (.blobs/) │
│ • S3-compatible │
│ • Azure Blob │
└─────────────────────────┘ └─────────────────────────┘
``` ```
Two practical consequences: Two practical consequences:
- **The spool/chunk directories see write-heavy churn** during uploads — - **Direct uploads no longer use a spool directory.** Each uploaded
fast disk (NVMe) and sufficient free space matter more here than on byte is written at most once — straight into the blob backend as a
the final storage backend. CDC chunk. Re-uploads and edited files write only the chunks the
- **The promotion from spool → storage is a `rename(2)` whenever store doesn't already have.
source and destination share a filesystem** (i.e. when the backend - **The chunk-session directory sees write-heavy churn** during
is `local` and the spool dir is on the same FS as `.blobs/`). On multi-chunk uploads — fast disk (NVMe) and sufficient free space
remote backends (S3, Azure) the promotion is always a network matter more here than on the final storage backend.
upload from the local spool; placement of the spool still matters
for intake throughput but the "same FS" rule doesn't apply.
## Upload size caps — what each one bounds ## Upload size caps — what each one bounds
Three independent caps control how large an upload OxiCloud will Three independent caps control how large an upload OxiCloud will
accept. Pick them with disk and tmpfs sizing in mind: the spool/chunk accept. Pick them with disk sizing in mind: the chunk directory must
directories must be able to hold the worst case (cap × concurrent be able to hold the worst case (cap × concurrent sessions).
uploads).
| Variable | Default | What it caps | When it fires | | Variable | Default | What it caps | When it fires |
|---|---|---|---| |---|---|---|---|
| `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. | | `OXICLOUD_MAX_UPLOAD_SIZE` | 10 GB | **Whole-file ceiling.** Applies to both direct PUT (per-body) and chunked uploads (declared `total_size`). The absolute upper bound on any single file in OxiCloud. | Chunked: at `POST /api/uploads` against the JSON-declared `total_size`, before any chunk is uploaded. Direct PUT: indirectly via `OXICLOUD_DIRECT_PUT_MAX_BYTES`, which is expected to be ≤ `OXICLOUD_MAX_UPLOAD_SIZE`. |
| `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `POST /api/files/upload`, `PUT /webdav/...`, and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. | | `OXICLOUD_DIRECT_PUT_MAX_BYTES` | 1 GiB | **Non-chunked PUT body.** Per-request cap for `PUT /webdav/...` and `PUT /remote.php/dav/files/.../...`. Set below `OXICLOUD_MAX_UPLOAD_SIZE` so larger files are pushed onto the chunked protocol — which is resumable on failure. | During body streaming, as a per-frame accumulator. Excess → 413 with a "use chunked upload" hint. |
| `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. | | `OXICLOUD_CHUNK_MAX_BYTES` | 100 MB | **Per-chunk body** in a chunked-upload session (`PATCH /api/uploads/{id}` or `PUT /remote.php/dav/uploads/.../chunk`). Independent of the whole-file cap — a 5 GB file in 100 MB chunks is 50 PATCHes each bounded by this. | During chunk-body streaming. Excess → 413. |
### Recommendation: prefer chunked uploads for large files ### Recommendation: prefer chunked uploads for large files
@@ -64,244 +58,117 @@ uploads).
The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below The defaults (`OXICLOUD_DIRECT_PUT_MAX_BYTES` = 1 GiB, well below
`OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric. `OXICLOUD_MAX_UPLOAD_SIZE` = 10 GB) are deliberately asymmetric.
Files between those two caps can only succeed via the chunked Files between those two caps can only succeed via the chunked
protocol. Three reasons to keep them that way: protocol. The reason is **resilience**: a direct PUT at 95 % of 5 GB
that drops loses everything (the partially ingested chunks are
reclaimed by GC, but the client must restart from byte 0). The same
drop on a chunked upload loses one ~5 MB chunk; the client retries
that chunk and continues. NextCloud desktop and the OxiCloud web UI
already switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
- **Resilience.** A direct PUT at 95 % of 5 GB that drops loses ### Disk sizing
everything. The same drop on a chunked upload loses one ~5 MB
chunk; the client retries that chunk and continues.
- **Memory + disk pressure.** Direct PUT spools the full body to
disk per request. Ten concurrent 5 GB direct PUTs use up to 50 GB
of transient spool disk. Chunked spreads each upload across many
small PATCHes; per-request resource use stays bounded by
`OXICLOUD_CHUNK_MAX_BYTES`.
- **Convention.** NextCloud desktop and the OxiCloud web UI already
switch to chunked at ~10 MB (`CHUNKED_UPLOAD_THRESHOLD`).
### Why caps matter for tmpfs sizing OxiCloud streams bodies frame-by-frame, so **RAM** is bounded
(~10 MB per in-flight upload for the CDC ingest buffers) regardless
of the caps. **Disk space** scales with the caps:
OxiCloud streams bodies frame-by-frame, so **RAM** is bounded to one - **Direct PUT / multipart**: no transient spool. Bytes land directly
HTTP frame (~64 KB) per request regardless of the caps. **Disk space**, in the blob backend as deduplicated chunks; worst-case extra disk
however, scales with the caps: per upload is the file's own (deduplicated) size — the same space
the stored file occupies afterwards.
- **Chunked upload**: each in-flight session accumulates its chunk
parts under `OXICLOUD_CHUNK_DIR` until `/complete` streams them
into the blob store and the session is cleaned up. Worst case disk
per session = **file_size** (the parts); total =
`OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
- **Direct PUT**: each in-flight upload spools the full body to disk | Settings | Chunked worst case (5 sessions) | Safe on 4 GB volume? |
under `OXICLOUD_UPLOAD_TMPDIR` until promotion. Worst case disk = |---|---|---|
`OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs`. | Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB | 50 GB | ❌ overflows |
- **Chunked upload**: each in-flight session accumulates chunks | `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB | 2.5 GB | ✅ fits |
under `OXICLOUD_CHUNK_DIR`, then assembles them into a single temp
file before promotion. Worst case disk per session = **2 ×
file_size** (chunks + assembled file); total disk =
`2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions`.
The chunked formula uses `OXICLOUD_MAX_UPLOAD_SIZE` because that's ### Don't put `OXICLOUD_CHUNK_DIR` on tmpfs
what bounds the declared `total_size` at session creation. The
direct-PUT formula uses the smaller `OXICLOUD_DIRECT_PUT_MAX_BYTES`
since that's what bounds each direct PUT body.
### Sizing examples In many container setups the OS temp dir is **tmpfs** — RAM-backed
storage that counts against the cgroup memory limit. A few concurrent
A 4 GB tmpfs serving a small team (5 concurrent direct PUTs OR 5 multi-GB chunked sessions on tmpfs will wake the OOMKiller long
concurrent chunked sessions): before the uploads finish. Point `OXICLOUD_CHUNK_DIR` at a real-disk
directory in containers.
| Settings | Direct-PUT worst case | Chunked worst case | Safe on 4 GB tmpfs? |
|---|---|---|---|
| Defaults: `OXICLOUD_MAX_UPLOAD_SIZE`=10 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=100 MB | 5 GiB (5 × 1 GiB) | 100 GB (5 × 2 × 10 GB) | ❌ chunked overflows |
| `OXICLOUD_MAX_UPLOAD_SIZE`=500 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=100 MB, `OXICLOUD_CHUNK_MAX_BYTES`=20 MB | 500 MB | 5 GB | ⚠ direct PUT fits, chunked still overflows |
| `OXICLOUD_MAX_UPLOAD_SIZE`=300 MB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=50 MB, `OXICLOUD_CHUNK_MAX_BYTES`=10 MB | 250 MB | 3 GB | ✅ both fit |
A real-disk volume (cheap, large):
| Settings | Direct-PUT worst case | Chunked worst case | Comment |
|---|---|---|---|
| Defaults (see row above) | 5 GiB | 100 GB | Fine on a 200+ GB volume; almost any real-disk setup |
| `OXICLOUD_MAX_UPLOAD_SIZE`=100 GB, `OXICLOUD_DIRECT_PUT_MAX_BYTES`=5 GiB, `OXICLOUD_CHUNK_MAX_BYTES`=500 MB | 25 GiB | 1 TB | Plausible for video archives; needs a dedicated upload volume |
### Choosing tmpfs vs real disk
| Constraint | Choice |
|---|---|
| `OXICLOUD_DIRECT_PUT_MAX_BYTES × concurrent_direct_PUTs + 2 × OXICLOUD_MAX_UPLOAD_SIZE × concurrent_chunked_sessions ≤ free RAM × 0.5` | tmpfs OK (fast, atomic with `.blobs/` if also tmpfs) |
| Worst case exceeds half free RAM | **real disk** — same filesystem as `.blobs/` ideal |
| Container with cgroup memory limit | **real disk** — tmpfs spool counts against the cgroup limit and triggers OOMKill |
| Multi-GB uploads expected | **real disk** — even small concurrency on tmpfs runs out of space |
| Small-file workload only (≤ 50 MB), high concurrency | tmpfs gives a noticeable intake speedup |
The defaults (`OXICLOUD_MAX_UPLOAD_SIZE`=10 GB,
`OXICLOUD_DIRECT_PUT_MAX_BYTES`=1 GiB,
`OXICLOUD_CHUNK_MAX_BYTES`=100 MB) assume **real disk**. Don't run
the defaults against tmpfs unless you've sized it for the worst case.
## TL;DR ## TL;DR
| Variable | Default | Purpose | | Variable | Default | Purpose |
|---|---|---| |---|---|---|
| `OXICLOUD_STORAGE_PATH` | `./storage` | Where `.blobs/` lives (the canonical content store) | | `OXICLOUD_STORAGE_PATH` | `./storage` | Where `.blobs/` lives (the canonical content store) |
| `OXICLOUD_UPLOAD_TMPDIR` | OS temp dir | Where non-chunked PUT bodies are spooled |
| `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Where chunked-upload sessions accumulate | | `OXICLOUD_CHUNK_DIR` | `{STORAGE_PATH}/.uploads` | Where chunked-upload sessions accumulate |
The two rules that matter most: The two rules that matter most:
1. **Put all three on the same filesystem.** Blob promotion is an 1. **Keep `OXICLOUD_CHUNK_DIR` off tmpfs** (the default in many
atomic `rename(2)` when source and destination share an FS — cheap containers) — chunk parts count against the cgroup memory limit
and crash-safe. Across filesystems it becomes a full `read + write +
unlink`, multiplying the IO and widening the durability window.
2. **Don't leave the spool dir on tmpfs** (the default in many
containers). Spool bodies count against the cgroup memory limit
and can trigger OOMKill on multi-GB uploads. and can trigger OOMKill on multi-GB uploads.
2. **NVMe for the chunk dir pays off** on deployments with heavy
large-file traffic: each chunk PUT writes a file and the progress
bitmap, and `/complete` reads them all back in order.
## Where each upload surface spools ## Where each upload surface writes
OxiCloud has several entry points that accept request bodies. They
land in different places by default:
| Surface | Default destination | Configurable via | | Surface | Default destination | Configurable via |
|---|---|---| |---|---|---|
| REST chunked PUT (`PATCH /api/uploads/{id}`) | `{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN` | `OXICLOUD_CHUNK_DIR` | | REST chunked PUT (`PATCH /api/uploads/{id}`) | `{STORAGE_PATH}/.uploads/{upload_id}/chunk_NNNNNN` | `OXICLOUD_CHUNK_DIR` |
| REST chunked assemble (during `/complete`) | `{STORAGE_PATH}/.uploads/{upload_id}/assembled` | `OXICLOUD_CHUNK_DIR` |
| NextCloud chunked PUT (`PUT /dav/uploads/.../chunk`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}` | `OXICLOUD_CHUNK_DIR` | | NextCloud chunked PUT (`PUT /dav/uploads/.../chunk`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}` | `OXICLOUD_CHUNK_DIR` |
| NextCloud chunked assemble (during `MOVE`) | `{STORAGE_PATH}/.uploads/nextcloud/{user}/{upload_id}/.assembled` | `OXICLOUD_CHUNK_DIR` | | Direct PUT / multipart / WOPI / chunked `/complete` | straight into the blob backend (CDC chunks) | `OXICLOUD_STORAGE_PATH` (local backend) |
| Native WebDAV PUT (`PUT /webdav/{path}`) | OS temp dir (`/tmp`) | `OXICLOUD_UPLOAD_TMPDIR` | | Final blob storage | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |
| NextCloud single-file PUT (`PUT /dav/files/.../{path}`) | OS temp dir | `OXICLOUD_UPLOAD_TMPDIR` |
| REST multipart upload (`POST /api/files/upload`) | `{STORAGE_PATH}/.dedup_temp/upload-{uuid}` | `OXICLOUD_STORAGE_PATH` (subdir is hard-wired) |
| Final blob storage (after fsync + rename) | `{STORAGE_PATH}/.blobs/{ab}/{abc…}.blob` | `OXICLOUD_STORAGE_PATH` |
## Why placement matters The local blob backend stages each chunk write under
`{STORAGE_PATH}/.dedup_temp/` and promotes it with an atomic
### 1. Same filesystem ⇒ promotion is a rename `rename(2)` — both directories live under `OXICLOUD_STORAGE_PATH`,
so same-filesystem placement (and therefore atomic promotion) is
OxiCloud uses **content-addressable storage**: the final blob path is automatic and not separately configurable.
derived from the file's BLAKE3 hash, which can only be known after the
last byte arrives. So every upload writes to a temp location first,
then **promotes** the temp file to `.blobs/{ab}/{abc…}.blob` by way of
a `rename(2)` call.
- **Same FS:** `rename` is atomic, O(1), no data copy. Total upload
cost = body bytes received + one rename syscall. Crash-safe — the
blob either exists at the final path or doesn't.
- **Cross-FS:** the kernel can't `rename(2)` across filesystems. The
blob backend falls back to `fs::copy + fs::remove_file` (visible in
`local_blob_backend.rs` as the EXDEV handler). Total cost = body
bytes received + one full file copy. Doubles the IO bandwidth used
per upload and widens the durability window.
### 2. Spool off tmpfs
`tempfile::NamedTempFile::new()` (used when `OXICLOUD_UPLOAD_TMPDIR`
is unset) honors `$TMPDIR`, which in many container setups points at
**tmpfs** — RAM-backed storage. A 2 GB upload spool then consumes 2 GB
of memory until the rename promotes it to disk.
In a Kubernetes pod with a 4 GB memory limit, the OOMKiller wakes up
long before the upload finishes. With `OXICLOUD_UPLOAD_TMPDIR` pointed
at a real-disk directory, the spool's memory footprint stays at ~one
HTTP frame regardless of file size.
### 3. NVMe for the hot path
The chunked-upload session directory sees a LOT of small writes —
each chunk PUT writes a file, the progress bitmap is rewritten after
each PUT, the assemble step reads them all back in order. Pointing
`OXICLOUD_CHUNK_DIR` at an NVMe device is a substantial win on
deployments that handle large file uploads, even if the final blob
storage is on slower disk.
The same applies to `OXICLOUD_UPLOAD_TMPDIR` (single-file PUTs).
A common high-throughput layout:
- **NVMe** (small, fast): `OXICLOUD_CHUNK_DIR`, `OXICLOUD_UPLOAD_TMPDIR`
- **HDD or NAS** (large, cheap): `OXICLOUD_STORAGE_PATH`/`.blobs/`
Trade-off: the rename optimization (rule 1) DOESN'T apply across
filesystems. If you split the hot path off the blob filesystem, every
upload pays a full file copy on promotion. You have to choose
between **fast intake** and **zero-copy promotion**.
| Goal | Layout | Cost per upload |
|---|---|---|
| Fastest possible intake | NVMe chunk dir + HDD blobs | 1× write to NVMe + 1× read NVMe + 1× write to HDD (copy) |
| Lowest IO + crash safety | NVMe everything OR HDD everything | 1× write to disk + 1 rename (~0 cost) |
| Default (do nothing) | Everything under `STORAGE_PATH` on whatever FS that is | Depends on `STORAGE_PATH` placement |
For most deployments **"same FS everywhere"** wins. The NVMe-split is
useful when intake latency dominates the user experience and you can
afford the doubled IO.
## Recommended layouts ## Recommended layouts
### Single-disk box (most common) ### Single-disk box (most common)
Defaults are fine. Optionally set `OXICLOUD_UPLOAD_TMPDIR` to keep Defaults are fine:
the PUT spool off `/tmp`:
```bash ```bash
OXICLOUD_STORAGE_PATH=/var/lib/oxicloud OXICLOUD_STORAGE_PATH=/var/lib/oxicloud
OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/.spool
# OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads # OXICLOUD_CHUNK_DIR unset → /var/lib/oxicloud/.uploads
``` ```
All three on the same filesystem → rename promotion → atomic and fast.
### Container with constrained memory ### Container with constrained memory
Critical: make sure neither spool sits on tmpfs. Critical: make sure the chunk dir doesn't sit on tmpfs.
```bash ```bash
OXICLOUD_STORAGE_PATH=/data OXICLOUD_STORAGE_PATH=/data
OXICLOUD_UPLOAD_TMPDIR=/data/.spool
OXICLOUD_CHUNK_DIR=/data/.uploads OXICLOUD_CHUNK_DIR=/data/.uploads
``` ```
If you can't mount a writable `/data`, at minimum bind-mount a real
volume at the spool dirs.
### Split-disk (NVMe intake + HDD blobs) ### Split-disk (NVMe intake + HDD blobs)
```bash ```bash
OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud # .blobs/ + .dedup_temp/ OXICLOUD_STORAGE_PATH=/mnt/hdd/oxicloud # .blobs/ + .dedup_temp/
OXICLOUD_UPLOAD_TMPDIR=/mnt/nvme/oxi-spool
OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks OXICLOUD_CHUNK_DIR=/mnt/nvme/oxi-chunks
``` ```
Faster intake; pays a copy on promotion. Worth it when uploads are Chunk parts land on NVMe (fast PUTs, fast `/complete` read-back);
many small files (NVMe IOPS dominates) or when intake latency directly the deduplicated chunks are written once to the HDD-backed blob
hits user-visible UX. store as `/complete` streams through them.
## Sharing the spool and chunk directories ## Sharing the chunk directory
Pointing `OXICLOUD_UPLOAD_TMPDIR` and `OXICLOUD_CHUNK_DIR` at the The REST and NC chunked surfaces can share `OXICLOUD_CHUNK_DIR` by
**same directory** is supported by design. Each writer tags its design. Each writer tags its output so they never interfere:
output so the surfaces never interfere with each other:
| Writer | On-disk name pattern | | Writer | On-disk name pattern |
|---|---| |---|---|
| PUT spool (single-file uploads) | `.tmpXXXXXXXX` — files (not directories), random suffix |
| REST chunked sessions | `oxi-chunk-{uuid}/` — directories with a well-known prefix | | REST chunked sessions | `oxi-chunk-{uuid}/` — directories with a well-known prefix |
| NC chunked subtree | `nextcloud/{user}/{uuid}/` — under its own root subdir | | NC chunked subtree | `nextcloud/{user}/{uuid}/` — under its own root subdir |
The 24-hour orphan-session cleanup loop filters strictly on the The 24-hour orphan-session cleanup loop filters strictly on the
`oxi-chunk-` prefix, so it can NEVER delete a non-OxiCloud directory `oxi-chunk-` prefix, so it can NEVER delete a non-OxiCloud directory
that happens to live alongside chunked sessions. The PUT spool's that happens to live alongside chunked sessions.
`.tmpXXXX` files are files (not directories) and the NC subtree's
`nextcloud/` root has its own name — both are invisible to the
cleanup loop.
**Recommendation:** for new deployments, use separate directories
anyway (the defaults `.spool/` and `.uploads/` already do this) —
it makes disk-usage attribution clearer and keeps IOPS isolated when
both are busy. Shared directories are safe to use when disk layout
forces it.
## What's NOT yet configurable
- **REST multipart upload directory** (`POST /api/files/upload`) is
hard-wired to `{STORAGE_PATH}/.dedup_temp/`. It can't be moved
separately. Same-FS placement is automatic.
- **WOPI PutFile spool** (Office editor saves) uses the bare OS temp
dir without honoring `OXICLOUD_UPLOAD_TMPDIR`. This is a known
inconsistency and on the hardening backlog.
- **Per-user / per-drive spool directories** — all users share the
same `OXICLOUD_CHUNK_DIR` root today. Multi-tenant isolation
through separate spool dirs isn't supported.
## Quick verification ## Quick verification
+5 -13
View File
@@ -55,22 +55,14 @@ OXICLOUD_SERVER_HOST=127.0.0.1
#OXICLOUD_CHUNK_MAX_BYTES=104857600 #OXICLOUD_CHUNK_MAX_BYTES=104857600
# ── Upload spool directories ───────────────────────────────────────── # ── Upload spool directories ─────────────────────────────────────────
# Where in-flight uploads land BEFORE being promoted into final blob # Direct (non-chunked) uploads stream straight into the blob store —
# storage. Same-filesystem placement (with `.blobs/`) makes the # no spool directory. Only chunked-upload sessions accumulate on disk.
# promotion an atomic rename(2); NVMe placement speeds up intake.
# See docs/config/storage-fine-tuning.md for layout examples. # See docs/config/storage-fine-tuning.md for layout examples.
# Directory for non-chunked PUT spool tempfiles. Default: OS temp dir
# ($TMPDIR / /tmp), often tmpfs (RAM) in containers — writing a large
# upload there fills page-cache that counts against the cgroup memory
# limit and can OOMKill the process. Point this at a real-disk path
# (same FS as the storage backend is ideal).
#OXICLOUD_UPLOAD_TMPDIR=/var/lib/oxicloud/tmp
# Root directory for chunked-upload sessions (REST + NextCloud chunked # Root directory for chunked-upload sessions (REST + NextCloud chunked
# share this root). Default: {STORAGE_PATH}/.uploads. Pointing this at # share this root). Default: {STORAGE_PATH}/.uploads. Avoid tmpfs in
# NVMe accelerates the chunk-write + assembly loop; pointing it at the # containers (parts count against the cgroup memory limit); NVMe
# same FS as `.blobs/` makes blob promotion atomic. # placement accelerates chunk PUTs and the /complete streaming pass.
#OXICLOUD_CHUNK_DIR=/var/lib/oxicloud/.uploads #OXICLOUD_CHUNK_DIR=/var/lib/oxicloud/.uploads
# How often (seconds) the background sweep reconciles each user's cached # How often (seconds) the background sweep reconciles each user's cached
+19 -5
View File
@@ -98,6 +98,21 @@ pub struct UploadStatusResponseDto {
pub is_complete: bool, pub is_complete: bool,
} }
/// A completed upload session, ready to be streamed into the blob store.
///
/// `chunk_paths` lists every chunk file in assembly order — the caller
/// concatenates them as one byte stream (CDC chunking + hashing happen in
/// that single pass). The files stay on disk until `finalize_upload`, so a
/// failed completion (e.g. checksum mismatch) remains retryable.
#[derive(Debug, Clone)]
pub struct CompletedUploadParts {
pub chunk_paths: Vec<PathBuf>,
pub filename: String,
pub folder_id: Option<String>,
pub content_type: String,
pub total_size: u64,
}
/// Port for chunked/resumable upload operations. /// Port for chunked/resumable upload operations.
/// ///
/// Implementations manage upload sessions, chunk storage, reassembly, /// Implementations manage upload sessions, chunk storage, reassembly,
@@ -137,16 +152,15 @@ pub trait ChunkedUploadPort: Send + Sync + 'static {
user_id: Uuid, user_id: Uuid,
) -> Result<UploadStatusResponseDto, DomainError>; ) -> Result<UploadStatusResponseDto, DomainError>;
/// Assemble all chunks into the final file. /// Validate completion and hand back the ordered chunk parts.
/// ///
/// Returns `(assembled_file_path, filename, folder_id, content_type, total_size, blake3_hash)`. /// No assembled file is written — the caller streams the parts straight
/// The hash is computed during assembly (hash-on-write), eliminating a /// into the content-addressable store.
/// second sequential read of the assembled file.
async fn complete_upload( async fn complete_upload(
&self, &self,
upload_id: &str, upload_id: &str,
user_id: Uuid, user_id: Uuid,
) -> Result<(PathBuf, String, Option<String>, String, u64, String), DomainError>; ) -> Result<CompletedUploadParts, DomainError>;
/// Finalize upload: clean up the session and temporary files. /// Finalize upload: clean up the session and temporary files.
async fn finalize_upload(&self, upload_id: &str, user_id: Uuid) -> Result<(), DomainError>; async fn finalize_upload(&self, upload_id: &str, user_id: Uuid) -> Result<(), DomainError>;
-12
View File
@@ -80,18 +80,6 @@ pub struct DedupStatsDto {
/// duplicate storage automatically. Multiple file references can /// duplicate storage automatically. Multiple file references can
/// point to the same physical blob. /// point to the same physical blob.
pub trait DedupPort: Send + Sync + 'static { pub trait DedupPort: Send + Sync + 'static {
/// Store content with deduplication (streaming from file).
///
/// If `pre_computed_hash` is provided (e.g. hash-on-write from the handler),
/// the file will NOT be re-read to calculate the hash — saving one full
/// sequential read of the file.
async fn store_from_file(
&self,
source_path: &Path,
content_type: Option<String>,
pre_computed_hash: Option<String>,
) -> Result<DedupResultDto, DomainError>;
/// Check if a blob with the given hash exists. /// Check if a blob with the given hash exists.
async fn blob_exists(&self, hash: &str) -> bool; async fn blob_exists(&self, hash: &str) -> bool;
+29 -53
View File
@@ -1,6 +1,5 @@
use bytes::Bytes; use bytes::Bytes;
use futures::Stream; use futures::Stream;
use std::path::Path;
use std::pin::Pin; use std::pin::Pin;
use std::sync::Arc; use std::sync::Arc;
use uuid::Uuid; use uuid::Uuid;
@@ -17,74 +16,51 @@ use crate::domain::services::authorization::Permission;
// Upload port // Upload port
// ───────────────────────────────────────────────────── // ─────────────────────────────────────────────────────
/// A blob already stored in the content-addressable chunk store, carrying
/// exactly ONE reference that the receiving method takes ownership of.
///
/// Produced by the upload-ingest layer (`interfaces::upload_ingest`), which
/// streams the request body straight into the CDC dedup store. Methods that
/// accept a `StoredBlob` either attach the reference to a file row or
/// release it on failure — callers never need to compensate themselves.
#[derive(Debug, Clone)]
pub struct StoredBlob {
/// BLAKE3 of the full content (manifest / blob key).
pub hash: String,
/// Content size in bytes.
pub size: u64,
/// `false` when the content already existed (dedup hit) — forwarded to
/// lifecycle hooks so e.g. thumbnails aren't regenerated for known blobs.
pub is_new_blob: bool,
}
/// Primary port for file upload operations. /// Primary port for file upload operations.
/// ///
/// **All upload paths converge on streaming-to-disk** — no method accepts /// **All upload paths converge on streaming-into-the-chunk-store** — content
/// `Vec<u8>` for content. Even `create_file` / `update_file` (WebDAV /// never passes through this port; it is ingested by the interface layer
/// helpers that receive `&[u8]`) spool to a temp file internally so that /// (CDC chunking + hashing while the body arrives, no spool file) and only
/// peak RAM stays at ~256 KB regardless of file size. /// the resulting [`StoredBlob`] reference travels through here.
///
/// - Normal uploads: handler spools multipart to temp file → `upload_file_streaming`
/// - Chunked uploads: chunks already on disk → `upload_file_from_path`
/// - WebDAV PUT (new): handler streams to temp file → `update_file_streaming`
/// - WebDAV PUT (small/compat): `create_file` / `update_file` spool internally
pub trait FileUploadUseCase: Send + Sync + 'static { pub trait FileUploadUseCase: Send + Sync + 'static {
/// Upload from a temp file already on disk (true streaming, ~256 KB RAM). /// Register a new file row pointing at an already-ingested blob.
/// ///
/// When `pre_computed_hash` is `Some`, the blob store skips the hash /// Takes ownership of the blob's reference (released on failure).
/// re-read — the handler already computed it during the multipart spool.
async fn upload_file_streaming( async fn upload_file_streaming(
&self, &self,
name: String, name: String,
folder_id: Option<String>, folder_id: Option<String>,
content_type: String, content_type: String,
temp_path: &Path, blob: StoredBlob,
size: u64,
pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError>; ) -> Result<FileDto, DomainError>;
/// Upload from a file already assembled on disk (chunked uploads). /// Replace the content of the file at `path` with an already-ingested
/// blob, or create the file when it doesn't exist (WebDAV/WOPI PUT).
/// ///
/// Same as `upload_file_streaming` but with a separate name for clarity. /// Takes ownership of the blob's reference (released on failure).
async fn upload_file_from_path(
&self,
name: String,
folder_id: Option<String>,
content_type: String,
file_path: &Path,
pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError>;
/// Creates a new file at the specified path (for WebDAV)
async fn create_file(
&self,
parent_path: &str,
filename: &str,
content: &[u8],
content_type: &str,
) -> Result<FileDto, DomainError>;
/// Updates the content of an existing file (for WebDAV)
async fn update_file(
&self,
path: &str,
content: &[u8],
content_type: &str,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError>;
/// Streaming update — spools body to a temp file with incremental hash,
/// then atomically replaces the file content via dedup store.
///
/// Peak RAM: ~256 KB regardless of file size.
/// Used by WebDAV PUT for large files.
async fn update_file_streaming( async fn update_file_streaming(
&self, &self,
path: &str, path: &str,
temp_path: &Path, blob: StoredBlob,
size: u64,
content_type: &str, content_type: &str,
pre_computed_hash: Option<String>,
modified_at: Option<i64>, modified_at: Option<i64>,
) -> Result<FileDto, DomainError>; ) -> Result<FileDto, DomainError>;
} }
+14 -16
View File
@@ -251,21 +251,21 @@ pub struct CopyFolderTreeResult {
/// Secondary port for file **writing**. /// Secondary port for file **writing**.
/// ///
/// Covers: upload (buffered + streaming), move, delete, update, /// Covers: upload registration, move, delete, update, and deferred
/// and deferred registration for the write-behind cache. /// registration for the write-behind cache.
pub trait FileWritePort: Send + Sync + 'static { pub trait FileWritePort: Send + Sync + 'static {
/// Streaming upload — saves a file from a temp file already on disk. /// Register a file row pointing at a blob already stored in the
/// content-addressable chunk store.
/// ///
/// When `pre_computed_hash` is provided, the dedup service skips the /// Takes ownership of one blob reference: on any failure the reference
/// hash re-read — zero extra I/O beyond the initial spool. /// is released before the error is returned.
async fn save_file_from_temp( async fn save_file_with_blob(
&self, &self,
name: String, name: String,
folder_id: Option<String>, folder_id: Option<String>,
content_type: String, content_type: String,
temp_path: &std::path::Path, blob_hash: &str,
size: u64, size: u64,
pre_computed_hash: Option<String>,
) -> Result<File, DomainError>; ) -> Result<File, DomainError>;
/// Moves a file to another folder. /// Moves a file to another folder.
@@ -281,22 +281,20 @@ pub trait FileWritePort: Send + Sync + 'static {
/// Deletes a file. /// Deletes a file.
async fn delete_file(&self, id: &str) -> Result<(), DomainError>; async fn delete_file(&self, id: &str) -> Result<(), DomainError>;
/// Streaming update — replaces file content from a temp file on disk. /// Atomically swap a file's content to a blob already stored in the
/// content-addressable chunk store.
/// ///
/// When `pre_computed_hash` is provided, the dedup service skips the /// Takes ownership of one blob reference (released on failure); the
/// hash re-read — zero extra I/O beyond the initial spool. /// previous content's reference is dropped after the swap.
/// Peak RAM: ~256 KB regardless of file size.
/// ///
/// Returns `(new_blob_hash, updated_at_epoch)` — everything a caller /// Returns `(new_blob_hash, updated_at_epoch)` — everything a caller
/// needs to rebuild the fresh entity/ETag from a `File` it already /// needs to rebuild the fresh entity/ETag from a `File` it already
/// holds, without re-reading the row it just updated. /// holds, without re-reading the row it just updated.
async fn update_file_content_from_temp( async fn update_file_content_with_blob(
&self, &self,
file_id: &str, file_id: &str,
temp_path: &std::path::Path, blob_hash: &str,
size: u64, size: u64,
content_type: Option<String>,
pre_computed_hash: Option<String>,
modified_at: Option<i64>, modified_at: Option<i64>,
) -> Result<(String, i64), DomainError>; ) -> Result<(String, i64), DomainError>;
+25 -189
View File
@@ -1,15 +1,13 @@
use std::path::{Path, PathBuf};
use std::sync::Arc; use std::sync::Arc;
use crate::application::dtos::file_dto::FileDto; use crate::application::dtos::file_dto::FileDto;
use crate::application::ports::file_lifecycle::FileLifecycleHook; use crate::application::ports::file_lifecycle::FileLifecycleHook;
use crate::application::ports::file_ports::FileUploadUseCase; use crate::application::ports::file_ports::{FileUploadUseCase, StoredBlob};
use crate::application::ports::storage_ports::{FileReadPort, FileWritePort}; use crate::application::ports::storage_ports::{FileReadPort, FileWritePort};
use crate::application::services::storage_usage_service::StorageUsageService; use crate::application::services::storage_usage_service::StorageUsageService;
use crate::common::errors::DomainError; use crate::common::errors::DomainError;
use crate::infrastructure::repositories::pg::FileBlobReadRepository; use crate::infrastructure::repositories::pg::FileBlobReadRepository;
use crate::infrastructure::repositories::pg::FileBlobWriteRepository; use crate::infrastructure::repositories::pg::FileBlobWriteRepository;
use crate::infrastructure::services::dedup_service::DedupService;
use crate::infrastructure::services::file_content_cache::FileContentCache; use crate::infrastructure::services::file_content_cache::FileContentCache;
use tracing::{debug, info, warn}; use tracing::{debug, info, warn};
@@ -34,20 +32,16 @@ fn extract_username_from_path(path: &str) -> Option<String> {
/// Service for file upload operations. /// Service for file upload operations.
/// ///
/// **Every upload path converges on streaming-to-disk** — there is no /// Content never passes through this service: the interface layer streams
/// `Vec<u8>` buffer path. /// the request body straight into the CDC chunk store (no spool file, no
/// /// full-body buffering) and hands over a [`StoredBlob`] reference. This
/// - **Normal uploads**: handler spools multipart to temp file → `upload_file_streaming` /// service registers the metadata row, keeps caches coherent and fires
/// - **Chunked uploads**: chunks already on disk → `upload_file_from_path` /// lifecycle hooks. Blob-reference ownership is consumed by the write
/// - **WebDAV PUT (large)**: handler streams body to temp file → `update_file_streaming` /// port, which releases it on failure — callers never compensate.
/// - **WebDAV PUT (small / compat)**: `create_file` / `update_file` spool `&[u8]`
/// to a temp file internally, then call the streaming path.
///
/// Peak RAM usage during any upload: ~256 KB (streaming hash) regardless of file size.
pub struct FileUploadService { pub struct FileUploadService {
/// Write port — handles save, streaming, deferred registration /// Write port — registers file rows against ingested blobs.
file_write: Arc<FileBlobWriteRepository>, file_write: Arc<FileBlobWriteRepository>,
/// Read port — needed for WebDAV create_file / update_file /// Read port — needed for WebDAV/WOPI update-by-path.
file_read: Option<Arc<FileBlobReadRepository>>, file_read: Option<Arc<FileBlobReadRepository>>,
/// Optional storage usage tracking /// Optional storage usage tracking
storage_usage_service: Option<Arc<StorageUsageService>>, storage_usage_service: Option<Arc<StorageUsageService>>,
@@ -55,9 +49,6 @@ pub struct FileUploadService {
content_cache: Option<Arc<FileContentCache>>, content_cache: Option<Arc<FileContentCache>>,
/// Single lifecycle dispatcher — fires on_file_created / on_file_updated. /// Single lifecycle dispatcher — fires on_file_created / on_file_updated.
file_lifecycle_hook: Option<Arc<dyn FileLifecycleHook>>, file_lifecycle_hook: Option<Arc<dyn FileLifecycleHook>>,
/// Directory for spool temp files (`&[u8]` upload variants). When `Some`,
/// keeps spools off tmpfs/RAM so they don't count against the cgroup limit.
upload_temp_dir: Option<PathBuf>,
} }
impl FileUploadService { impl FileUploadService {
@@ -69,7 +60,6 @@ impl FileUploadService {
storage_usage_service: None, storage_usage_service: None,
content_cache: None, content_cache: None,
file_lifecycle_hook: None, file_lifecycle_hook: None,
upload_temp_dir: None,
} }
} }
@@ -84,16 +74,9 @@ impl FileUploadService {
storage_usage_service: None, storage_usage_service: None,
content_cache: None, content_cache: None,
file_lifecycle_hook: None, file_lifecycle_hook: None,
upload_temp_dir: None,
} }
} }
/// Configures the spool directory for the `&[u8]` upload variants.
pub fn with_upload_temp_dir(mut self, dir: Option<PathBuf>) -> Self {
self.upload_temp_dir = dir;
self
}
/// Configures the content cache for invalidation on file updates. /// Configures the content cache for invalidation on file updates.
pub fn with_content_cache(mut self, cache: Arc<FileContentCache>) -> Self { pub fn with_content_cache(mut self, cache: Arc<FileContentCache>) -> Self {
self.content_cache = Some(cache); self.content_cache = Some(cache);
@@ -117,11 +100,6 @@ impl FileUploadService {
// ── private helpers ────────────────────────────────────────── // ── private helpers ──────────────────────────────────────────
/// Create a spool temp file, honoring the configured upload temp dir.
fn new_temp(&self) -> std::io::Result<tempfile::NamedTempFile> {
crate::common::temp::new_spool_temp_file(self.upload_temp_dir.as_deref())
}
/// Optionally update storage usage after a successful upload. /// Optionally update storage usage after a successful upload.
fn maybe_update_storage_usage(&self, file: &FileDto) { fn maybe_update_storage_usage(&self, file: &FileDto) {
if let Some(storage_service) = &self.storage_usage_service { if let Some(storage_service) = &self.storage_usage_service {
@@ -146,172 +124,37 @@ impl FileUploadService {
} }
impl FileUploadUseCase for FileUploadService { impl FileUploadUseCase for FileUploadService {
/// Streaming upload from a temp file on disk. /// Register a new file row pointing at an already-ingested blob.
///
/// Peak RAM: ~256 KB (hash calculation) regardless of file size.
/// The temp file is consumed (moved/deleted) by the blob store.
async fn upload_file_streaming( async fn upload_file_streaming(
&self, &self,
name: String, name: String,
folder_id: Option<String>, folder_id: Option<String>,
content_type: String, content_type: String,
temp_path: &Path, blob: StoredBlob,
size: u64,
pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError> { ) -> Result<FileDto, DomainError> {
let (file, is_new_blob) = self let file = self
.file_write .file_write
.save_file_from_temp_with_dedup( .save_file_with_blob(name.clone(), folder_id, content_type, &blob.hash, blob.size)
name.clone(),
folder_id,
content_type,
temp_path,
size,
pre_computed_hash,
)
.await?; .await?;
let dto = FileDto::from(file); let dto = FileDto::from(file);
info!( info!(
"📡 STREAMING UPLOAD: {} ({} bytes, ID: {})", "📡 STREAMING UPLOAD: {} ({} bytes, ID: {})",
name, size, dto.id name, blob.size, dto.id
); );
self.maybe_update_storage_usage(&dto); self.maybe_update_storage_usage(&dto);
if let Some(hook) = &self.file_lifecycle_hook { if let Some(hook) = &self.file_lifecycle_hook {
hook.on_file_created(&dto.id, &dto.content_hash, &dto.mime_type, is_new_blob); hook.on_file_created(&dto.id, &dto.content_hash, &dto.mime_type, blob.is_new_blob);
} }
Ok(dto) Ok(dto)
} }
/// Upload from a file already on disk (chunked uploads). /// Swap the content of the file at `path` to an already-ingested blob,
async fn upload_file_from_path( /// creating the file when it doesn't exist (WebDAV/NextCloud/WOPI PUT).
&self,
name: String,
folder_id: Option<String>,
content_type: String,
file_path: &Path,
pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError> {
let size = tokio::fs::metadata(file_path)
.await
.map_err(|e| {
DomainError::internal_error(
"FileUpload",
format!("Failed to read file metadata: {}", e),
)
})?
.len();
self.upload_file_streaming(
name,
folder_id,
content_type,
file_path,
size,
pre_computed_hash,
)
.await
}
/// Creates a file at a specific path (for WebDAV PUT on new resource).
///
/// Spools the in-memory `&[u8]` to a temp file with hash-on-write,
/// then delegates to the streaming path. Peak RAM: the caller's
/// buffer + ~256 KB for the hasher.
async fn create_file(
&self,
parent_path: &str,
filename: &str,
content: &[u8],
content_type: &str,
) -> Result<FileDto, DomainError> {
// Look up the folder ID by folder path
let parent_id = if !parent_path.is_empty() {
if let Some(file_read) = &self.file_read {
file_read.get_folder_id_by_path(parent_path).await.ok()
} else {
None
}
} else {
None
};
// Spool to temp file + hash
let temp = self
.new_temp()
.map_err(|e| DomainError::internal_error("FileUpload", format!("temp file: {e}")))?;
tokio::fs::write(temp.path(), content)
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("write temp: {e}")))?;
let hash = DedupService::hash_file(temp.path())
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("hash: {e}")))?;
let (file, is_new_blob) = self
.file_write
.save_file_from_temp_with_dedup(
filename.to_string(),
parent_id,
content_type.to_string(),
temp.path(),
content.len() as u64,
Some(hash),
)
.await?;
let dto = FileDto::from(file);
self.maybe_update_storage_usage(&dto);
if let Some(hook) = &self.file_lifecycle_hook {
hook.on_file_created(&dto.id, &dto.content_hash, &dto.mime_type, is_new_blob);
}
Ok(dto)
}
/// Updates an existing file's content, or creates it if not found (for WebDAV PUT).
///
/// Spools the in-memory `&[u8]` to a temp file with hash-on-write,
/// then delegates to the streaming update/create path.
async fn update_file(
&self,
path: &str,
content: &[u8],
content_type: &str,
modified_at: Option<i64>,
) -> Result<FileDto, DomainError> {
// Spool to temp file + hash
let temp = self
.new_temp()
.map_err(|e| DomainError::internal_error("FileUpload", format!("temp file: {e}")))?;
tokio::fs::write(temp.path(), content)
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("write temp: {e}")))?;
let hash = DedupService::hash_file(temp.path())
.await
.map_err(|e| DomainError::internal_error("FileUpload", format!("hash: {e}")))?;
self.update_file_streaming(
path,
temp.path(),
content.len() as u64,
content_type,
Some(hash),
modified_at,
)
.await
}
/// Streaming update — replaces file content from a temp file on disk.
///
/// Uses `update_file_content_from_temp` which passes the pre-computed hash
/// to dedup, avoiding a second full read of the file.
/// For new files (not found at `path`), falls back to `upload_file_streaming`.
///
/// Peak RAM: ~256 KB regardless of file size.
async fn update_file_streaming( async fn update_file_streaming(
&self, &self,
path: &str, path: &str,
temp_path: &Path, blob: StoredBlob,
size: u64,
content_type: &str, content_type: &str,
pre_computed_hash: Option<String>,
modified_at: Option<i64>, modified_at: Option<i64>,
) -> Result<FileDto, DomainError> { ) -> Result<FileDto, DomainError> {
// Try to find the existing file first // Try to find the existing file first
@@ -321,14 +164,7 @@ impl FileUploadUseCase for FileUploadService {
let file_id = file.id().to_string(); let file_id = file.id().to_string();
let (new_hash, updated_at) = self let (new_hash, updated_at) = self
.file_write .file_write
.update_file_content_from_temp( .update_file_content_with_blob(&file_id, &blob.hash, blob.size, modified_at)
&file_id,
temp_path,
size,
Some(content_type.to_string()),
pre_computed_hash,
modified_at,
)
.await?; .await?;
// Invalidate content cache — file content has changed. // Invalidate content cache — file content has changed.
if let Some(cc) = &self.content_cache { if let Some(cc) = &self.content_cache {
@@ -343,7 +179,7 @@ impl FileUploadUseCase for FileUploadService {
parts.id, parts.id,
parts.name, parts.name,
parts.storage_path, parts.storage_path,
size, blob.size,
parts.mime_type, parts.mime_type,
parts.folder_id, parts.folder_id,
parts.created_at, parts.created_at,
@@ -381,15 +217,15 @@ impl FileUploadUseCase for FileUploadService {
None None
}; };
let (created, is_new_blob) = self let is_new_blob = blob.is_new_blob;
let created = self
.file_write .file_write
.save_file_from_temp_with_dedup( .save_file_with_blob(
filename.to_string(), filename.to_string(),
parent_id, parent_id,
content_type.to_string(), content_type.to_string(),
temp_path, &blob.hash,
size, blob.size,
pre_computed_hash,
) )
.await?; .await?;
let dto = FileDto::from(created); let dto = FileDto::from(created);
@@ -6,7 +6,7 @@
use bytes::Bytes; use bytes::Bytes;
use futures::Stream; use futures::Stream;
use std::collections::HashMap; use std::collections::HashMap;
use std::path::{Path, PathBuf}; use std::path::PathBuf;
use std::pin::Pin; use std::pin::Pin;
use std::sync::Mutex; use std::sync::Mutex;
use uuid::Uuid; use uuid::Uuid;
@@ -169,14 +169,13 @@ impl MockFileWritePort {
} }
impl FileWritePort for MockFileWritePort { impl FileWritePort for MockFileWritePort {
async fn save_file_from_temp( async fn save_file_with_blob(
&self, &self,
_name: String, _name: String,
_folder_id: Option<String>, _folder_id: Option<String>,
_content_type: String, _content_type: String,
_temp_path: &Path, _blob_hash: &str,
_size: u64, _size: u64,
_pre_computed_hash: Option<String>,
) -> Result<File, DomainError> { ) -> Result<File, DomainError> {
unimplemented!() unimplemented!()
} }
@@ -205,13 +204,11 @@ impl FileWritePort for MockFileWritePort {
Ok(()) Ok(())
} }
async fn update_file_content_from_temp( async fn update_file_content_with_blob(
&self, &self,
_file_id: &str, _file_id: &str,
_temp_path: &Path, _blob_hash: &str,
_size: u64, _size: u64,
_content_type: Option<String>,
_pre_computed_hash: Option<String>,
_modified_at: Option<i64>, _modified_at: Option<i64>,
) -> Result<(String, i64), DomainError> { ) -> Result<(String, i64), DomainError> {
Ok((String::new(), 0)) Ok((String::new(), 0))
@@ -554,14 +554,13 @@ impl FileReadPort for MockFileRepository {
} }
impl FileWritePort for MockFileRepository { impl FileWritePort for MockFileRepository {
async fn save_file_from_temp( async fn save_file_with_blob(
&self, &self,
_name: String, _name: String,
_folder_id: Option<String>, _folder_id: Option<String>,
_content_type: String, _content_type: String,
_temp_path: &std::path::Path, _blob_hash: &str,
_size: u64, _size: u64,
_pre_computed_hash: Option<String>,
) -> std::result::Result<File, DomainError> { ) -> std::result::Result<File, DomainError> {
unimplemented!() unimplemented!()
} }
@@ -586,13 +585,11 @@ impl FileWritePort for MockFileRepository {
Ok(()) Ok(())
} }
async fn update_file_content_from_temp( async fn update_file_content_with_blob(
&self, &self,
_file_id: &str, _file_id: &str,
_temp_path: &std::path::Path, _blob_hash: &str,
_size: u64, _size: u64,
_content_type: Option<String>,
_pre_computed_hash: Option<String>,
_modified_at: Option<i64>, _modified_at: Option<i64>,
) -> std::result::Result<(String, i64), DomainError> { ) -> std::result::Result<(String, i64), DomainError> {
Ok((String::new(), 0)) Ok((String::new(), 0))
+4 -19
View File
@@ -228,12 +228,6 @@ pub struct StorageConfig {
/// returns 413 with a "use chunked upload" hint when a direct PUT /// returns 413 with a "use chunked upload" hint when a direct PUT
/// exceeds this cap. Env: `OXICLOUD_DIRECT_PUT_MAX_BYTES`. /// exceeds this cap. Env: `OXICLOUD_DIRECT_PUT_MAX_BYTES`.
pub direct_put_max_bytes: usize, pub direct_put_max_bytes: usize,
/// Directory for upload spool temp files. When `Some`, large uploads are
/// spooled here instead of the OS default temp dir (often tmpfs/RAM in
/// containers, where the spool's page-cache counts against the cgroup
/// memory limit and can trigger OOMKill on large files). Env:
/// `OXICLOUD_UPLOAD_TMPDIR`.
pub upload_temp_dir: Option<PathBuf>,
/// Root directory for chunked-upload sessions. When `Some`, chunks land /// Root directory for chunked-upload sessions. When `Some`, chunks land
/// under `{chunk_dir}/{upload_id}/` (REST) and /// under `{chunk_dir}/{upload_id}/` (REST) and
/// `{chunk_dir}/nextcloud/{user}/{upload_id}/` (NC). When `None`, falls /// `{chunk_dir}/nextcloud/{user}/{upload_id}/` (NC). When `None`, falls
@@ -401,7 +395,6 @@ impl Default for StorageConfig {
max_upload_size: MAX_UPLOAD_SIZE, max_upload_size: MAX_UPLOAD_SIZE,
chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT chunk_max_bytes: 100 * 1024 * 1024, // 100 MB — sane upper bound for a single chunked-upload PUT
direct_put_max_bytes: 1024 * 1024 * 1024, // 1 GiB — pushes larger uploads onto the chunked protocol direct_put_max_bytes: 1024 * 1024 * 1024, // 1 GiB — pushes larger uploads onto the chunked protocol
upload_temp_dir: None,
chunk_dir: None, chunk_dir: None,
usage_reconcile_secs: 600, // 10 minutes usage_reconcile_secs: 600, // 10 minutes
tree_etag_flush_ms: 500, tree_etag_flush_ms: 500,
@@ -1281,18 +1274,10 @@ impl AppConfig {
config.storage.direct_put_max_bytes = val; config.storage.direct_put_max_bytes = val;
} }
// Upload spool directory — keep large upload temp files off tmpfs/RAM // Chunked-upload session root — chunked sessions accumulate disk on
// (otherwise their page-cache counts against the cgroup memory limit). // long uploads (multi-chunk resumable transfers); sysadmins commonly
if let Ok(dir) = env::var("OXICLOUD_UPLOAD_TMPDIR") // want them on fast/local storage (NVMe). This knob lets that be
&& !dir.trim().is_empty() // expressed.
{
config.storage.upload_temp_dir = Some(PathBuf::from(dir.trim()));
}
// Chunked-upload session root — separate from the PUT spool because
// chunked sessions accumulate disk on long uploads (multi-chunk
// resumable transfers) while PUT spool is short-lived. Sysadmins
// commonly want one of them on fast/local storage (NVMe) and the
// other on bulk storage; this knob lets that be expressed.
if let Ok(dir) = env::var("OXICLOUD_CHUNK_DIR") if let Ok(dir) = env::var("OXICLOUD_CHUNK_DIR")
&& !dir.trim().is_empty() && !dir.trim().is_empty()
{ {
+1 -2
View File
@@ -452,8 +452,7 @@ impl AppServiceFactory {
repos.file_read_repository.clone(), repos.file_read_repository.clone(),
) )
.with_content_cache(core.file_content_cache.clone()) .with_content_cache(core.file_content_cache.clone())
.with_file_lifecycle_hook(core.file_lifecycle.clone()) .with_file_lifecycle_hook(core.file_lifecycle.clone()),
.with_upload_temp_dir(self.config.storage.upload_temp_dir.clone()),
); );
let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache( let file_retrieval_service = Arc::new(FileRetrievalService::new_with_cache(
+19 -90
View File
@@ -8,21 +8,25 @@
//! //!
//! Performance: < 1µs for the `infer` check (reads only header bytes, no allocation). //! Performance: < 1µs for the `infer` check (reads only header bytes, no allocation).
use std::path::Path; /// Maximum bytes needed for magic-byte detection. Upload ingestion peeks
use tokio::io::AsyncReadExt; /// this many bytes off the stream before forwarding them unchanged.
pub const MAGIC_BYTES_LEN: usize = 8192;
/// Maximum bytes to read for magic-byte detection.
const MAGIC_BYTES_LEN: usize = 8192;
/// Extract the filename component from a `/`-separated path. /// Extract the filename component from a `/`-separated path.
pub fn filename_from_path(path: &str) -> &str { pub fn filename_from_path(path: &str) -> &str {
path.rsplit('/').next().unwrap_or(path) path.rsplit('/').next().unwrap_or(path)
} }
/// Whether a claimed Content-Type is too generic to trust — these trigger
/// magic-byte detection on the upload path.
pub fn is_generic_mime(claimed: &str) -> bool {
claimed.is_empty() || claimed == "application/octet-stream" || claimed == "binary/octet-stream"
}
/// Refine a claimed MIME type using magic bytes and filename extension. /// Refine a claimed MIME type using magic bytes and filename extension.
/// ///
/// This is a synchronous function — the caller should already have the first /// This is a synchronous function — the caller should already have the first
/// bytes of the file available (or call the async wrapper below). /// bytes of the content available (upload ingestion peeks them in-flight).
/// ///
/// # Arguments /// # Arguments
/// * `buf` — first bytes of the file (at least 8192 for best results) /// * `buf` — first bytes of the file (at least 8192 for best results)
@@ -30,10 +34,7 @@ pub fn filename_from_path(path: &str) -> &str {
/// * `claimed` — the Content-Type sent by the client /// * `claimed` — the Content-Type sent by the client
pub fn refine_content_type(buf: &[u8], filename: &str, claimed: &str) -> String { pub fn refine_content_type(buf: &[u8], filename: &str, claimed: &str) -> String {
// If the client sent a specific type (not generic), trust it // If the client sent a specific type (not generic), trust it
if !claimed.is_empty() if !is_generic_mime(claimed) {
&& claimed != "application/octet-stream"
&& claimed != "binary/octet-stream"
{
return claimed.to_string(); return claimed.to_string();
} }
@@ -52,49 +53,9 @@ pub fn refine_content_type(buf: &[u8], filename: &str, claimed: &str) -> String
claimed.to_string() claimed.to_string()
} }
/// Async helper: reads the first bytes of a file on disk and refines the MIME type.
///
/// Designed for the upload path where the file has been spooled to a temp path.
pub async fn refine_content_type_from_file(
temp_path: &Path,
filename: &str,
claimed: &str,
) -> String {
// Fast path: if the client gave us a specific type, trust it
if !claimed.is_empty()
&& claimed != "application/octet-stream"
&& claimed != "binary/octet-stream"
{
return claimed.to_string();
}
// Read only the first bytes needed for magic detection (not the whole file).
match tokio::fs::File::open(temp_path).await {
Ok(mut file) => {
let mut buf = vec![0u8; MAGIC_BYTES_LEN];
let n = file.read(&mut buf).await.unwrap_or(0);
refine_content_type(&buf[..n], filename, claimed)
}
Err(e) => {
tracing::warn!(
"MIME detection: failed to read {} for magic bytes: {}",
temp_path.display(),
e
);
// Fall back to extension
let guess = mime_guess::from_path(filename);
if let Some(mime) = guess.first() {
return mime.to_string();
}
claimed.to_string()
}
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
use std::io::Write;
// ── refine_content_type (sync) ────────────────────────────── // ── refine_content_type (sync) ──────────────────────────────
@@ -145,47 +106,15 @@ mod tests {
assert_eq!(result, "image/png"); assert_eq!(result, "image/png");
} }
// ── refine_content_type_from_file (async) ─────────────────── // ── is_generic_mime ─────────────────────────────────────────
#[tokio::test] #[test]
async fn from_file_detects_png() { fn generic_mime_detection() {
let mut tmp = tempfile::NamedTempFile::new().unwrap(); assert!(is_generic_mime(""));
let png = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR"; assert!(is_generic_mime("application/octet-stream"));
tmp.write_all(png).unwrap(); assert!(is_generic_mime("binary/octet-stream"));
tmp.flush().unwrap(); assert!(!is_generic_mime("image/png"));
assert!(!is_generic_mime("text/plain"));
let result =
refine_content_type_from_file(tmp.path(), "photo", "application/octet-stream").await;
assert_eq!(result, "image/png");
}
#[tokio::test]
async fn from_file_falls_back_to_extension() {
let mut tmp = tempfile::NamedTempFile::new().unwrap();
tmp.write_all(b"not magic").unwrap();
tmp.flush().unwrap();
let result =
refine_content_type_from_file(tmp.path(), "doc.css", "application/octet-stream").await;
assert_eq!(result, "text/css");
}
#[tokio::test]
async fn from_file_trusts_specific_claimed() {
let result =
refine_content_type_from_file(Path::new("/nonexistent"), "file", "image/webp").await;
assert_eq!(result, "image/webp");
}
#[tokio::test]
async fn from_file_missing_file_falls_back_to_extension() {
let result = refine_content_type_from_file(
Path::new("/nonexistent/file"),
"photo.jpg",
"application/octet-stream",
)
.await;
assert_eq!(result, "image/jpeg");
} }
// ── filename_from_path ────────────────────────────────────── // ── filename_from_path ──────────────────────────────────────
-1
View File
@@ -4,4 +4,3 @@ pub mod errors;
pub mod locale; pub mod locale;
pub mod mime_detect; pub mod mime_detect;
pub mod stubs; pub mod stubs;
pub mod temp;
+8 -59
View File
@@ -24,6 +24,7 @@ use crate::application::dtos::search_dto::{
}; };
use crate::application::ports::file_ports::{ use crate::application::ports::file_ports::{
FileManagementUseCase, FileRetrievalUseCase, FileUploadUseCase, OptimizedFileContent, FileManagementUseCase, FileRetrievalUseCase, FileUploadUseCase, OptimizedFileContent,
StoredBlob,
}; };
use crate::application::ports::folder_ports::FolderUseCase; use crate::application::ports::folder_ports::FolderUseCase;
@@ -149,14 +150,13 @@ impl FileReadPort for StubFileReadPort {
pub struct StubFileWritePort; pub struct StubFileWritePort;
impl FileWritePort for StubFileWritePort { impl FileWritePort for StubFileWritePort {
async fn save_file_from_temp( async fn save_file_with_blob(
&self, &self,
_name: String, _name: String,
_folder_id: Option<String>, _folder_id: Option<String>,
_content_type: String, _content_type: String,
_temp_path: &Path, _blob_hash: &str,
_size: u64, _size: u64,
_pre_computed_hash: Option<String>,
) -> Result<File, DomainError> { ) -> Result<File, DomainError> {
Ok(File::default()) Ok(File::default())
} }
@@ -185,13 +185,11 @@ impl FileWritePort for StubFileWritePort {
Ok(()) Ok(())
} }
async fn update_file_content_from_temp( async fn update_file_content_with_blob(
&self, &self,
_file_id: &str, _file_id: &str,
_temp_path: &Path, _blob_hash: &str,
_size: u64, _size: u64,
_content_type: Option<String>,
_pre_computed_hash: Option<String>,
_modified_at: Option<i64>, _modified_at: Option<i64>,
) -> Result<(String, i64), DomainError> { ) -> Result<(String, i64), DomainError> {
Ok((String::new(), 0)) Ok((String::new(), 0))
@@ -478,40 +476,7 @@ impl FileUploadUseCase for StubFileUploadUseCase {
_name: String, _name: String,
_folder_id: Option<String>, _folder_id: Option<String>,
_content_type: String, _content_type: String,
_temp_path: &Path, _blob: StoredBlob,
_size: u64,
_pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError> {
Ok(FileDto::default())
}
async fn upload_file_from_path(
&self,
_name: String,
_folder_id: Option<String>,
_content_type: String,
_file_path: &Path,
_pre_computed_hash: Option<String>,
) -> Result<FileDto, DomainError> {
Ok(FileDto::default())
}
async fn create_file(
&self,
_parent_path: &str,
_filename: &str,
_content: &[u8],
_content_type: &str,
) -> Result<FileDto, DomainError> {
Ok(FileDto::default())
}
async fn update_file(
&self,
_path: &str,
_content: &[u8],
_content_type: &str,
_modified_at: Option<i64>,
) -> Result<FileDto, DomainError> { ) -> Result<FileDto, DomainError> {
Ok(FileDto::default()) Ok(FileDto::default())
} }
@@ -519,10 +484,8 @@ impl FileUploadUseCase for StubFileUploadUseCase {
async fn update_file_streaming( async fn update_file_streaming(
&self, &self,
_path: &str, _path: &str,
_temp_path: &Path, _blob: StoredBlob,
_size: u64,
_content_type: &str, _content_type: &str,
_pre_computed_hash: Option<String>,
_modified_at: Option<i64>, _modified_at: Option<i64>,
) -> Result<FileDto, DomainError> { ) -> Result<FileDto, DomainError> {
Ok(FileDto::default()) Ok(FileDto::default())
@@ -756,25 +719,11 @@ impl SearchUseCase for StubSearchUseCase {
// DedupPort // DedupPort
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
use crate::application::ports::dedup_ports::{ use crate::application::ports::dedup_ports::{BlobMetadataDto, DedupPort, DedupStatsDto};
BlobMetadataDto, DedupPort, DedupResultDto, DedupStatsDto,
};
pub struct StubDedupPort; pub struct StubDedupPort;
impl DedupPort for StubDedupPort { impl DedupPort for StubDedupPort {
async fn store_from_file(
&self,
_source_path: &Path,
_content_type: Option<String>,
_pre_computed_hash: Option<String>,
) -> Result<DedupResultDto, DomainError> {
Err(DomainError::internal_error(
"DedupService",
"DedupService not initialized",
))
}
async fn blob_exists(&self, _hash: &str) -> bool { async fn blob_exists(&self, _hash: &str) -> bool {
false false
} }
-26
View File
@@ -1,26 +0,0 @@
//! Shared helper for creating upload spool temp files.
//!
//! Upload paths spool the request body to a temp file before deduplication.
//! By default `tempfile` uses the OS temp dir (`std::env::temp_dir()`, i.e.
//! `$TMPDIR` / `/tmp`), which in many container setups is **tmpfs (RAM)**.
//! Writing a multi-hundred-MB upload there fills page-cache that counts
//! against the cgroup memory limit and can OOMKill the process. Pointing the
//! spool at a real-disk directory (`OXICLOUD_UPLOAD_TMPDIR`) keeps the upload
//! footprint proportional to the streaming buffer, not the file size.
use std::path::Path;
use tempfile::NamedTempFile;
/// Create a [`NamedTempFile`], honoring an optional configured spool directory.
///
/// When `dir` is `Some`, the temp file is created there (the directory is
/// created if missing); otherwise the OS default temp dir is used.
pub fn new_spool_temp_file(dir: Option<&Path>) -> std::io::Result<NamedTempFile> {
match dir {
Some(d) => {
std::fs::create_dir_all(d)?;
NamedTempFile::new_in(d)
}
None => NamedTempFile::new(),
}
}
+4 -4
View File
@@ -71,15 +71,15 @@ pub trait FileWriteRepository: Send + Sync + 'static {
content: Vec<u8>, content: Vec<u8>,
) -> Result<File, DomainError>; ) -> Result<File, DomainError>;
/// Streaming upload — saves a file from a temp file already on disk. /// Registers a file row pointing at a blob already stored in the
async fn save_file_from_temp( /// content-addressable chunk store (one blob reference is consumed).
async fn save_file_with_blob(
&self, &self,
name: String, name: String,
folder_id: Option<String>, folder_id: Option<String>,
content_type: String, content_type: String,
temp_path: &std::path::Path, blob_hash: &str,
size: u64, size: u64,
pre_computed_hash: Option<String>,
) -> Result<File, DomainError>; ) -> Result<File, DomainError>;
/// Moves a file to another folder. /// Moves a file to another folder.
@@ -220,26 +220,30 @@ impl FileBlobWriteRepository {
Ok((new_hash.to_string(), updated_at)) Ok((new_hash.to_string(), updated_at))
} }
/// Like [`FileWritePort::save_file_from_temp`] but also returns whether the /// Register a file row pointing at a blob already stored in the chunk
/// blob was genuinely new (`true`) or a dedup hit (`false`). /// store (the upload-ingest layer streamed the content in). Consumes the
/// Used by [`FileUploadService`] to pass `is_new_blob` to lifecycle hooks. /// caller's blob reference: any failure releases it before returning.
pub async fn save_file_from_temp_with_dedup( async fn save_file_with_blob_impl(
&self, &self,
name: String, name: String,
folder_id: Option<String>, folder_id: Option<String>,
content_type: String, content_type: String,
temp_path: &std::path::Path, blob_hash: &str,
size: u64, size: u64,
pre_computed_hash: Option<String>, ) -> Result<File, DomainError> {
) -> Result<(File, bool), DomainError> { let user_id = match self.resolve_user_id(folder_id.as_deref()).await {
let user_id = self.resolve_user_id(folder_id.as_deref()).await?; Ok(user_id) => user_id,
Err(e) => {
let dedup_result = self if let Err(rollback_err) = self.dedup.remove_reference(blob_hash).await {
.dedup tracing::error!(
.store_from_file(temp_path, Some(content_type.clone()), pre_computed_hash) "Blob orphaned after owner resolution failure — hash: {}, err: {}",
.await?; &blob_hash[..12],
let is_new_blob = !dedup_result.was_deduplicated(); rollback_err
let blob_hash = dedup_result.hash().to_string(); );
}
return Err(e);
}
};
// Deadlock victims (40P01) retry before the compensation below runs — // Deadlock victims (40P01) retry before the compensation below runs —
// a successful retry must keep the blob reference alive. The final // a successful retry must keep the blob reference alive. The final
@@ -259,7 +263,7 @@ impl FileBlobWriteRepository {
.bind(&name) .bind(&name)
.bind(&folder_id) .bind(&folder_id)
.bind(user_id) .bind(user_id)
.bind(&blob_hash) .bind(blob_hash)
.bind(size as i64) .bind(size as i64)
.bind(&content_type) .bind(&content_type)
.bind(category_order_for(&name, &content_type)) .bind(category_order_for(&name, &content_type))
@@ -269,7 +273,7 @@ impl FileBlobWriteRepository {
{ {
Ok(row) => row, Ok(row) => row,
Err(e) => { Err(e) => {
if let Err(rollback_err) = self.dedup.remove_reference(&blob_hash).await { if let Err(rollback_err) = self.dedup.remove_reference(blob_hash).await {
tracing::error!( tracing::error!(
"Blob orphaned after failed INSERT — hash: {}, err: {}", "Blob orphaned after failed INSERT — hash: {}, err: {}",
&blob_hash[..12], &blob_hash[..12],
@@ -309,32 +313,23 @@ impl FileBlobWriteRepository {
row.1, row.1,
row.2, row.2,
Some(user_id), Some(user_id),
blob_hash, blob_hash.to_string(),
)?; )?;
Ok((file, is_new_blob)) Ok(file)
} }
} }
impl FileWritePort for FileBlobWriteRepository { impl FileWritePort for FileBlobWriteRepository {
async fn save_file_from_temp( async fn save_file_with_blob(
&self, &self,
name: String, name: String,
folder_id: Option<String>, folder_id: Option<String>,
content_type: String, content_type: String,
temp_path: &std::path::Path, blob_hash: &str,
size: u64, size: u64,
pre_computed_hash: Option<String>,
) -> Result<File, DomainError> { ) -> Result<File, DomainError> {
self.save_file_from_temp_with_dedup( self.save_file_with_blob_impl(name, folder_id, content_type, blob_hash, size)
name, .await
folder_id,
content_type,
temp_path,
size,
pre_computed_hash,
)
.await
.map(|(file, _)| file)
} }
async fn move_file( async fn move_file(
@@ -532,24 +527,18 @@ impl FileWritePort for FileBlobWriteRepository {
Ok(()) Ok(())
} }
async fn update_file_content_from_temp( async fn update_file_content_with_blob(
&self, &self,
file_id: &str, file_id: &str,
temp_path: &std::path::Path, blob_hash: &str,
size: u64, size: u64,
content_type: Option<String>,
pre_computed_hash: Option<String>,
modified_at: Option<i64>, modified_at: Option<i64>,
) -> Result<(String, i64), DomainError> { ) -> Result<(String, i64), DomainError> {
// Streaming: pass pre-computed hash so dedup skips re-reading the file. // The content was already ingested into the chunk store by the
let dedup_result = self // upload-ingest layer; swap_blob_hash consumes its reference and
.dedup // releases it on failure.
.store_from_file(temp_path, content_type, pre_computed_hash)
.await?;
let new_hash = dedup_result.hash().to_string();
let swapped = self let swapped = self
.swap_blob_hash(file_id, &new_hash, size as i64, modified_at) .swap_blob_hash(file_id, blob_hash, size as i64, modified_at)
.await?; .await?;
// The file now maps to a different blob — drop the read-side cache // The file now maps to a different blob — drop the read-side cache
// entry so streaming downloads cannot serve the previous content // entry so streaming downloads cannot serve the previous content
@@ -6,14 +6,14 @@
//! (updated atomically on each chunk) are stored alongside the chunk files. //! (updated atomically on each chunk) are stored alongside the chunk files.
//! On boot the service scans `temp_base_dir` and recovers any active sessions. //! On boot the service scans `temp_base_dir` and recovers any active sessions.
//! - Parallel chunk transfers (up to 6 concurrent) //! - Parallel chunk transfers (up to 6 concurrent)
//! - Automatic reassembly with hash-on-write (BLAKE3) //! - Completion streams the ordered parts straight into the CDC blob store
//! - Expiration cleanup (24 h) //! - Expiration cleanup (24 h)
//! //!
//! Protocol: //! Protocol:
//! 1. POST /api/uploads → Create upload session, get upload_id //! 1. POST /api/uploads → Create upload session, get upload_id
//! 2. PATCH /api/uploads/:id → Upload chunks (parallel OK) //! 2. PATCH /api/uploads/:id → Upload chunks (parallel OK)
//! 3. HEAD /api/uploads/:id → Check progress //! 3. HEAD /api/uploads/:id → Check progress
//! 4. POST /api/uploads/:id/complete → Finalize and assemble //! 4. POST /api/uploads/:id/complete → Stream parts into the blob store
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use dashmap::DashMap; use dashmap::DashMap;
@@ -28,7 +28,8 @@ use tokio::io::AsyncWriteExt;
use uuid::Uuid; use uuid::Uuid;
use crate::application::ports::chunked_upload_ports::{ use crate::application::ports::chunked_upload_ports::{
ChunkUploadResponseDto, ChunkedUploadPort, CreateUploadResponseDto, UploadStatusResponseDto, ChunkUploadResponseDto, ChunkedUploadPort, CompletedUploadParts, CreateUploadResponseDto,
UploadStatusResponseDto,
}; };
use crate::domain::errors::{DomainError, ErrorKind}; use crate::domain::errors::{DomainError, ErrorKind};
@@ -603,7 +604,7 @@ impl ChunkedUploadService {
/// ///
/// Used by the streaming REST PUT path: the handler calls /// Used by the streaming REST PUT path: the handler calls
/// `prepare_chunk` → streams body to disk via /// `prepare_chunk` → streams body to disk via
/// `interfaces::upload_spool::stream_body_to_path` → calls /// `interfaces::upload_ingest::stream_body_to_path` → calls
/// `commit_chunk` to finalise. This lets the body bypass the /// `commit_chunk` to finalise. This lets the body bypass the
/// in-memory `Bytes` allocation entirely (peak heap ~one HTTP /// in-memory `Bytes` allocation entirely (peak heap ~one HTTP
/// frame instead of "chunk size"). /// frame instead of "chunk size").
@@ -936,24 +937,23 @@ impl ChunkedUploadService {
}) })
} }
/// Assemble chunks into final file and return the path + pre-computed BLAKE3 hash. /// Validate completion and return the chunk parts in assembly order.
/// ///
/// **Hash-on-Write**: BLAKE3 is computed while copying chunks into the /// No assembled file is written and nothing is hashed here — the caller
/// assembled file, eliminating the second sequential read that dedup_service /// streams the parts straight into the CDC chunk store, which computes
/// would otherwise need. /// BLAKE3 and dedup-checks each CDC chunk in that single read pass. The
/// /// part files stay on disk until `finalize_upload`, so a completion
/// Returns `(assembled_file_path, filename, folder_id, content_type, total_size, blake3_hash)`. /// that fails downstream (e.g. client checksum mismatch) is retryable.
async fn complete_upload_inner( async fn complete_upload_inner(
&self, &self,
upload_id: &str, upload_id: &str,
user_id: &str, user_id: &str,
) -> Result<(PathBuf, String, Option<String>, String, u64, String), String> { ) -> Result<CompletedUploadParts, String> {
// Verify ownership before assembly // Verify ownership before completion
self.verify_session_owner(upload_id, user_id)?; self.verify_session_owner(upload_id, user_id)?;
// Get session and validate completion. // Get session and validate completion.
// Clone the session data and drop the DashMap ref immediately // Clone the session data and drop the DashMap ref immediately.
// so the shard is not held during the expensive assembly step.
let session = { let session = {
let entry = self let entry = self
.sessions .sessions
@@ -971,118 +971,29 @@ impl ChunkedUploadService {
entry.clone() entry.clone()
}; };
// Assemble file with hash-on-write. // Chunk indices are unique (upload_chunk rejects duplicates) but may
// // have arrived out of order — sort to recover the assembly order.
// The entire loop is offloaded to spawn_blocking because BLAKE3 let mut indices: Vec<usize> = session.chunks.iter().map(|c| c.index).collect();
// hashing is CPU-bound and would otherwise block a Tokio worker, indices.sort_unstable();
// starving all other connections. let chunk_paths: Vec<PathBuf> = indices
// Synchronous I/O is used inside the blocking thread — it avoids
// the async reactor overhead and is actually faster for this
// sequential workload.
let assembled_path = session.temp_dir.join("assembled");
let chunks_meta: Vec<(usize, PathBuf)> = session
.chunks
.iter() .iter()
.map(|c| { .map(|index| session.temp_dir.join(format!("chunk_{:06}", index)))
(
c.index,
session.temp_dir.join(format!("chunk_{:06}", c.index)),
)
})
.collect(); .collect();
let total_size = session.total_size;
let hash = tokio::task::spawn_blocking(move || -> Result<String, String> {
use std::io::{BufWriter as StdBufWriter, Read, Write};
let raw_output = std::fs::OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(&assembled_path)
.map_err(|e| format!("Failed to create assembled file: {e}"))?;
// Pre-allocate assembled file to reduce fragmentation
let _ = raw_output.set_len(total_size);
// 512 KB I/O buffers — 8× fewer syscalls than 64 KB
let mut output = StdBufWriter::with_capacity(524_288, raw_output);
let mut hasher = blake3::Hasher::new();
// For files >10 MB, use multithreaded BLAKE3 hashing (all cores)
const RAYON_THRESHOLD: u64 = 10 * 1024 * 1024;
let use_rayon = total_size > RAYON_THRESHOLD;
// Single 512 KB read buffer reused across all chunks (avoids N allocations)
let mut buf = vec![0u8; 524_288];
for (index, chunk_path) in &chunks_meta {
let mut chunk_file = std::fs::File::open(chunk_path)
.map_err(|e| format!("Failed to open chunk {index}: {e}"))?;
loop {
let n = chunk_file
.read(&mut buf)
.map_err(|e| format!("Failed to read chunk {index}: {e}"))?;
if n == 0 {
break;
}
if use_rayon {
hasher.update_rayon(&buf[..n]);
} else {
hasher.update(&buf[..n]);
}
output.write_all(&buf[..n]).map_err(|e| {
format!("Failed to write chunk {index} to assembled file: {e}")
})?;
}
}
output
.flush()
.map_err(|e| format!("Failed to flush assembled file: {e}"))?;
// ── Durability boundary ────────────────────────────────────
// `flush` drains BufWriter's userspace buffer but leaves the
// bytes in the kernel page cache. Without `sync_all`, a
// power loss between this `complete_upload` returning 2xx
// and the OS writeback timer firing (~5 s default) loses
// the merged blob — and PG's metadata row references a hash
// that no longer exists on disk. Reclaim the BufWriter's
// inner File via `into_inner` so we can `sync_all` it; the
// BufWriter would otherwise drop without flushing on the
// inner handle.
let raw_output = output
.into_inner()
.map_err(|e| format!("into_inner on BufWriter failed: {e}"))?;
raw_output
.sync_all()
.map_err(|e| format!("Failed to fsync assembled file: {e}"))?;
// Clean up chunk files (keep assembled) — already on a blocking thread
for (_index, chunk_path) in &chunks_meta {
let _ = std::fs::remove_file(chunk_path);
}
Ok(hasher.finalize().to_hex().to_string())
})
.await
.map_err(|e| format!("Assembly task panicked: {e}"))??;
let assembled_path = session.temp_dir.join("assembled");
tracing::info!( tracing::info!(
"✅ Assembled chunked upload: {} ({} bytes from {} chunks)", "✅ Chunked upload complete: {} ({} bytes in {} chunks, streamed to CAS)",
session.filename, session.filename,
session.total_size, session.total_size,
session.chunks.len() chunk_paths.len()
); );
Ok(( Ok(CompletedUploadParts {
assembled_path, chunk_paths,
session.filename.clone(), filename: session.filename.clone(),
session.folder_id.clone(), folder_id: session.folder_id.clone(),
session.content_type.clone(), content_type: session.content_type.clone(),
session.total_size, total_size: session.total_size,
hash, })
))
} }
/// Finalize upload: remove session from RAM, then clean disk OUTSIDE lock. /// Finalize upload: remove session from RAM, then clean disk OUTSIDE lock.
@@ -1187,7 +1098,7 @@ impl ChunkedUploadPort for ChunkedUploadService {
&self, &self,
upload_id: &str, upload_id: &str,
user_id: Uuid, user_id: Uuid,
) -> Result<(PathBuf, String, Option<String>, String, u64, String), DomainError> { ) -> Result<CompletedUploadParts, DomainError> {
self.complete_upload_inner(upload_id, &user_id.to_string()) self.complete_upload_inner(upload_id, &user_id.to_string())
.await .await
.map_err(|e| DomainError::new(ErrorKind::InternalError, "ChunkedUpload", e)) .map_err(|e| DomainError::new(ErrorKind::InternalError, "ChunkedUpload", e))
@@ -1458,18 +1369,21 @@ mod tests {
assert_eq!(status.completed_chunks, 2); assert_eq!(status.completed_chunks, 2);
assert!(status.pending_chunks.is_empty()); assert!(status.pending_chunks.is_empty());
// 4. Complete (assemble) // 4. Complete — returns the ordered chunk parts, no assembled file
let (path, filename, _folder, _ct, size, hash) = service let parts = service
.complete_upload_inner(&id, "test-user") .complete_upload_inner(&id, "test-user")
.await .await
.expect("complete"); .expect("complete");
assert_eq!(filename, "test.txt"); assert_eq!(parts.filename, "test.txt");
assert_eq!(size, 1024); assert_eq!(parts.total_size, 1024);
assert!(!hash.is_empty()); assert_eq!(parts.chunk_paths.len(), 2);
assert!(path.exists());
// 5. Verify assembled content // 5. Verify the parts concatenate to the original content in order
let content = fs::read(&path).await.expect("read assembled"); let mut content = Vec::new();
for path in &parts.chunk_paths {
assert!(path.exists(), "chunk part must remain until finalize");
content.extend_from_slice(&fs::read(path).await.expect("read part"));
}
assert_eq!(&content[..512], &[b'A'; 512]); assert_eq!(&content[..512], &[b'A'; 512]);
assert_eq!(&content[512..], &[b'B'; 512]); assert_eq!(&content[512..], &[b'B'; 512]);
File diff suppressed because it is too large Load Diff
@@ -244,12 +244,13 @@ impl BlobStorageBackend for LocalBlobBackend {
// Atomic rename (same filesystem). Falls back to copy+delete for // Atomic rename (same filesystem). Falls back to copy+delete for
// cross-device moves (EXDEV errno 18). // cross-device moves (EXDEV errno 18).
// //
// Durability boundary: the caller is responsible for // Durability boundary: the caller is responsible for having
// having sync_all'd the source file before invoking this // sync_all'd the source file before invoking this function.
// function — both `interfaces::upload_spool::spool_body_to_temp` // (The streaming upload path writes chunks via
// and `chunked_upload_service::complete_upload_inner` // `put_blob_from_bytes_unsynced` + a batched `sync_blobs`
// (the two production producers of `source_path`) now do // sweep instead; this move-based entry point remains for
// so. We fsync the parent of `blob_path` AFTER the rename // whole-file producers such as migration tooling and tests.)
// We fsync the parent of `blob_path` AFTER the rename
// so the dirent change itself becomes durable; without // so the dirent change itself becomes durable; without
// that, a power loss can resurrect the old (unrenamed) // that, a power loss can resurrect the old (unrenamed)
// name even when the file contents survive. // name even when the file contents survive.
@@ -73,7 +73,7 @@ impl NextcloudChunkedUploadService {
/// Store a chunk in the session directory. Buffers `data` in memory — /// Store a chunk in the session directory. Buffers `data` in memory —
/// use [`safe_chunk_path`](Self::safe_chunk_path) + the /// use [`safe_chunk_path`](Self::safe_chunk_path) + the
/// `interfaces/upload_spool::stream_body_to_path` helper to stream the /// `interfaces/upload_ingest::stream_body_to_path` helper to stream the
/// HTTP body directly to disk and avoid materialising the whole chunk /// HTTP body directly to disk and avoid materialising the whole chunk
/// in RAM. /// in RAM.
pub async fn store_chunk( pub async fn store_chunk(
@@ -93,24 +93,14 @@ impl NextcloudChunkedUploadService {
Ok(()) Ok(())
} }
/// Assemble all chunks in numeric order into a temp file, computing /// List the session's chunk files in assembly (numeric) order.
/// the BLAKE3 of the concatenated stream **during** the same read/
/// write pass (hash-on-write).
/// ///
/// Returns `(temp_path, total_size, blake3_hex)`. The caller passes /// The caller streams these directly into the CDC chunk store
/// the hash to the upload service as `pre_computed_hash` so the /// (`interfaces::upload_ingest::stream_from_files`) — chunking, BLAKE3
/// downstream dedup layer never has to re-read the assembled file /// hashing and dedup checks happen in that single read pass, so no
/// to compute it — saving one full file-sized read pass per upload. /// assembled temp file is ever written. The chunk parts stay on disk
/// /// until [`cleanup`](Self::cleanup), keeping completion retryable.
/// The read/hash/write loop runs inside `spawn_blocking` because pub async fn ordered_chunk_paths(&self, user: &str, upload_id: &str) -> Result<Vec<PathBuf>> {
/// BLAKE3 is CPU-bound and would otherwise starve the Tokio worker
/// running other connections; synchronous I/O is used inside the
/// blocking thread because the workload is sequential and the
/// async reactor overhead would only slow it down. For files larger
/// than ~10 MB BLAKE3's Rayon mode parallelises across cores —
/// mirrors what `ChunkedUploadService::complete_upload_inner` does
/// for the REST chunked path.
pub async fn assemble(&self, user: &str, upload_id: &str) -> Result<(PathBuf, u64, String)> {
let session_dir = self.safe_session_dir(user, upload_id)?; let session_dir = self.safe_session_dir(user, upload_id)?;
let mut entries: Vec<String> = Vec::new(); let mut entries: Vec<String> = Vec::new();
@@ -124,8 +114,11 @@ impl NextcloudChunkedUploadService {
.map_err(|e| DomainError::internal_error("ChunkedUpload", e.to_string()))? .map_err(|e| DomainError::internal_error("ChunkedUpload", e.to_string()))?
{ {
let name = entry.file_name().to_string_lossy().to_string(); let name = entry.file_name().to_string_lossy().to_string();
if name == ".file" { // `.file` is the NC-protocol assembly marker; `.assembled` is
continue; // Skip the assembly marker. // the staging file older releases wrote — sessions in flight
// across an upgrade may still contain one.
if name == ".file" || name == ".assembled" {
continue;
} }
entries.push(name); entries.push(name);
} }
@@ -133,74 +126,7 @@ impl NextcloudChunkedUploadService {
// Sort chunks numerically (Nextcloud sends them as "00001", "00002", ...). // Sort chunks numerically (Nextcloud sends them as "00001", "00002", ...).
entries.sort(); entries.sort();
let temp_path = session_dir.join(".assembled"); Ok(entries.iter().map(|n| session_dir.join(n)).collect())
let chunk_paths: Vec<PathBuf> = entries.iter().map(|n| session_dir.join(n)).collect();
let assembled_for_blocking = temp_path.clone();
// Read/hash/write loop runs synchronously on the blocking pool.
// BLAKE3 is computed in the same pass that copies bytes from chunk
// files into the assembled file — no second read after the fact.
let (total_size, hash) =
tokio::task::spawn_blocking(move || -> std::io::Result<(u64, String)> {
use std::io::{BufWriter as StdBufWriter, Read, Write};
let raw_output = std::fs::OpenOptions::new()
.create(true)
.write(true)
.truncate(true)
.open(&assembled_for_blocking)?;
// 512 KB write buffer — 8× fewer syscalls than 64 KB.
let mut output = StdBufWriter::with_capacity(524_288, raw_output);
let mut hasher = blake3::Hasher::new();
let mut buf = vec![0u8; 524_288];
let mut total: u64 = 0;
// Files >10 MB benefit from BLAKE3's multi-threaded mode.
// The threshold matches the REST chunked path's heuristic.
const RAYON_THRESHOLD_PER_FRAME: usize = 128 * 1024;
for chunk_path in &chunk_paths {
let mut chunk_file = std::fs::File::open(chunk_path)?;
loop {
let n = chunk_file.read(&mut buf)?;
if n == 0 {
break;
}
if n >= RAYON_THRESHOLD_PER_FRAME {
hasher.update_rayon(&buf[..n]);
} else {
hasher.update(&buf[..n]);
}
output.write_all(&buf[..n])?;
total += n as u64;
}
}
output.flush()?;
// ── Durability boundary ─────────────────────────────────
// sync_all is the actual fsync; without it, a power loss
// before the kernel writeback timer (~5 s) loses
// acknowledged data. Pull the inner File out of the
// BufWriter so we can sync the underlying handle —
// dropping the BufWriter wouldn't trigger fsync. macOS
// caveat: fsync there flushes to the disk controller
// only; true durability needs F_FULLFSYNC, not exposed
// by std.
let raw_output = output
.into_inner()
.map_err(|e| std::io::Error::other(format!("into_inner: {e}")))?;
raw_output.sync_all()?;
Ok((total, hasher.finalize().to_hex().to_string()))
})
.await
.map_err(|e| {
DomainError::internal_error("ChunkedUpload", format!("assemble task: {e}"))
})?
.map_err(|e| DomainError::internal_error("ChunkedUpload", e.to_string()))?;
Ok((temp_path, total_size, hash))
} }
/// Delete the upload session directory. /// Delete the upload session directory.
@@ -260,9 +186,10 @@ impl NextcloudChunkedUploadService {
let name = entry.file_name().to_string_lossy().to_string(); let name = entry.file_name().to_string_lossy().to_string();
// Filter internal markers — `.file` is the NC-protocol // Filter internal markers — `.file` is the NC-protocol
// assembly trigger target (it never reaches the disk // assembly trigger target (it never reaches the disk
// because MOVE redirects it), `.assembled` is our own // because MOVE redirects it), `.assembled` is the staging
// staging file from `assemble()`. Surfacing either to // file pre-streaming releases wrote (kept for sessions in
// the client would confuse its chunk-count check. // flight across an upgrade). Surfacing either to the
// client would confuse its chunk-count check.
if name == ".file" || name == ".assembled" { if name == ".file" || name == ".assembled" {
continue; continue;
} }
@@ -331,8 +258,18 @@ mod tests {
assert!(!svc.session_exists("alice", "upload-999").await); assert!(!svc.session_exists("alice", "upload-999").await);
} }
/// Concatenate the session's chunk files in assembly order — mirrors
/// what `upload_ingest::stream_from_files` feeds the CDC store.
async fn concat_chunks(svc: &NextcloudChunkedUploadService, user: &str, id: &str) -> Vec<u8> {
let mut out = Vec::new();
for path in svc.ordered_chunk_paths(user, id).await.unwrap() {
out.extend_from_slice(&fs::read(&path).await.unwrap());
}
out
}
#[tokio::test] #[tokio::test]
async fn test_store_and_assemble_chunks() { async fn test_store_and_order_chunks() {
let (svc, _dir) = test_service(); let (svc, _dir) = test_service();
svc.create_session("alice", "upload-002").await.unwrap(); svc.create_session("alice", "upload-002").await.unwrap();
@@ -343,20 +280,14 @@ mod tests {
.await .await
.unwrap(); .unwrap();
let (temp_path, size, hash) = svc.assemble("alice", "upload-002").await.unwrap();
let assembled = fs::read(&temp_path).await.unwrap();
assert_eq!(assembled, b"Hello, World!");
assert_eq!(size, 13);
// BLAKE3("Hello, World!") — proves hash-on-write happens during
// the assemble pass, not via a re-read.
assert_eq!( assert_eq!(
hash, concat_chunks(&svc, "alice", "upload-002").await,
"288a86a79f20a3d6dccdca7713beaed178798296bdfa7913fa2a62d9727bf8f8" b"Hello, World!"
); );
} }
#[tokio::test] #[tokio::test]
async fn test_assemble_chunks_in_sorted_order() { async fn test_chunk_paths_sorted_regardless_of_upload_order() {
let (svc, _dir) = test_service(); let (svc, _dir) = test_service();
svc.create_session("alice", "upload-003").await.unwrap(); svc.create_session("alice", "upload-003").await.unwrap();
@@ -371,16 +302,29 @@ mod tests {
.await .await
.unwrap(); .unwrap();
let (temp_path, size, hash) = svc.assemble("alice", "upload-003").await.unwrap(); // Chunks were stored in order 3,1,2 but must concatenate as "ABC".
let assembled = fs::read(&temp_path).await.unwrap(); assert_eq!(concat_chunks(&svc, "alice", "upload-003").await, b"ABC");
assert_eq!(assembled, b"ABC"); }
assert_eq!(size, 3);
// BLAKE3("ABC") — confirms sort happened (chunks were stored in #[tokio::test]
// order 3,1,2 but the hash matches "ABC", not "CAB" or "BAC"). async fn test_internal_markers_excluded_from_chunk_paths() {
assert_eq!( let (svc, _dir) = test_service();
hash, svc.create_session("alice", "upload-005").await.unwrap();
"d1717274597cf0289694f75d96d444b992a096f1afd8e7bbfa6ebb1d360fedfc"
); svc.store_chunk("alice", "upload-005", "00001", b"data")
.await
.unwrap();
// Stale staging file from a pre-streaming release.
svc.store_chunk("alice", "upload-005", ".assembled", b"old")
.await
.unwrap();
let paths = svc
.ordered_chunk_paths("alice", "upload-005")
.await
.unwrap();
assert_eq!(paths.len(), 1, "markers must be filtered out");
assert!(paths[0].ends_with("00001"));
} }
#[tokio::test] #[tokio::test]
@@ -4,7 +4,7 @@
//! - POST /api/uploads → Create upload session //! - POST /api/uploads → Create upload session
//! - PATCH /api/uploads/:id → Upload a chunk //! - PATCH /api/uploads/:id → Upload a chunk
//! - HEAD /api/uploads/:id → Get upload status //! - HEAD /api/uploads/:id → Get upload status
//! - POST /api/uploads/:id/complete → Assemble and finalize //! - POST /api/uploads/:id/complete → Stream parts into the blob store
//! - DELETE /api/uploads/:id → Cancel upload //! - DELETE /api/uploads/:id → Cancel upload
use axum::{ use axum::{
@@ -27,7 +27,7 @@ use crate::common::di::AppState;
use crate::domain::services::authorization::Permission; use crate::domain::services::authorization::Permission;
use crate::interfaces::errors::AppError; use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::AuthUser; use crate::interfaces::middleware::auth::AuthUser;
use crate::interfaces::upload_spool::stream_body_to_path; use crate::interfaces::upload_ingest::{self, stream_body_to_path};
/// Request body for creating an upload session /// Request body for creating an upload session
#[derive(Debug, Deserialize, ToSchema)] #[derive(Debug, Deserialize, ToSchema)]
@@ -64,35 +64,32 @@ pub struct CompleteUploadResponse {
/// Optional body for `POST /api/uploads/{id}/complete`. /// Optional body for `POST /api/uploads/{id}/complete`.
/// ///
/// When the client supplies `checksum`, the server compares it against /// When the client supplies `checksum`, the server compares it against
/// the assembled file's hash BEFORE promoting the blob to storage — /// the streamed content's hash BEFORE the file row is created — failure
/// failure aborts the upload atomically (no orphaned blob, no DB row). /// releases the blob reference and returns 400, with the chunk parts
/// This is the end-to-end integrity check: per-chunk MD5 proves each /// kept on disk for a retry. This is the end-to-end integrity check:
/// chunk arrived intact, but only the final hash catches assembly / /// per-chunk MD5 proves each chunk arrived intact, but only the final
/// promotion bugs and mis-ordered chunks. /// hash catches mis-ordered or corrupted assemblies.
/// ///
/// **`blake3` is highly recommended** — it's the algorithm the server /// **`blake3` is highly recommended** — it's the content-addressing
/// already runs over the assembled file during hash-on-write /// algorithm of the blob store itself, so verification is a string
/// assembly, so verification is a string comparison with zero extra /// comparison against the hash the store already computed, and the
/// I/O and zero extra CPU. It's also the same algorithm the server /// value the client sends equals the `content_hash` they'd later read
/// uses for blob-storage addressing, so the value the client sends /// back from `GET /api/files/{id}`. `md5` and `sha256` are accepted
/// equals the `content_hash` they'd later read back from /// for legacy client tooling; they are computed by an in-flight tee
/// `GET /api/files/{id}`. `md5` and `sha256` are accepted for /// during the same streaming pass — no extra disk read either way.
/// compatibility with legacy client tooling but each triggers a
/// second hash pass over the assembled file (~30–100 ms depending
/// on size).
/// ///
/// `Default` keeps the existing wire shape: clients that POST with no /// `Default` keeps the existing wire shape: clients that POST with no
/// body get today's behavior (no verification, server just returns /// body get today's behavior (no verification, server just returns
/// what it computed). /// what it computed).
#[derive(Debug, Default, Deserialize, ToSchema)] #[derive(Debug, Default, Deserialize, ToSchema)]
pub struct CompleteUploadRequest { pub struct CompleteUploadRequest {
/// Lowercase hex digest the client expects the assembled file to /// Lowercase hex digest the client expects the streamed content to
/// hash to. Compared case-insensitively. Omit to skip verification. /// hash to. Compared case-insensitively. Omit to skip verification.
pub checksum: Option<String>, pub checksum: Option<String>,
/// Algorithm name. `blake3` is the recommended choice (default — /// Algorithm name. `blake3` is the recommended choice (default —
/// matches the server's hash-on-write algorithm, zero extra cost). /// matches the blob store's content-addressing algorithm). `md5`,
/// `md5`, `sha256` / `sha-256` are accepted but trigger an extra /// `sha256` / `sha-256` are accepted and computed in-flight.
/// hash pass. Unknown values return 400. /// Unknown values return 400.
pub checksumalg: Option<String>, pub checksumalg: Option<String>,
} }
@@ -307,70 +304,15 @@ impl ChunkedUploadHandler {
} }
} }
/// Compute the requested checksum of the assembled file.
///
/// For `Blake3` the server already has the hash from hash-on-write
/// assembly — we just return it (zero I/O, zero CPU). For `Md5` and
/// `Sha256` we re-read the assembled file on the blocking pool and
/// hash it; the cost (~30–100 ms for typical files) is the trade-off
/// for accepting non-default algorithms.
async fn compute_assembled_hash(
assembled_path: &std::path::Path,
alg: ChecksumAlg,
blake3_already_computed: &str,
) -> Result<String, std::io::Error> {
match alg {
ChecksumAlg::Blake3 => Ok(blake3_already_computed.to_string()),
ChecksumAlg::Md5 | ChecksumAlg::Sha256 => {
let path = assembled_path.to_path_buf();
tokio::task::spawn_blocking(move || -> Result<String, std::io::Error> {
use std::io::Read;
let mut file = std::fs::File::open(&path)?;
let mut buf = vec![0u8; 524_288];
match alg {
ChecksumAlg::Md5 => {
use md5::Digest as _;
let mut h = md5::Md5::new();
loop {
let n = file.read(&mut buf)?;
if n == 0 {
break;
}
h.update(&buf[..n]);
}
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
}
ChecksumAlg::Sha256 => {
use sha2::Digest as _;
let mut h = sha2::Sha256::new();
loop {
let n = file.read(&mut buf)?;
if n == 0 {
break;
}
h.update(&buf[..n]);
}
Ok(h.finalize().iter().map(|b| format!("{b:02x}")).collect())
}
// Blake3 handled above — this branch is unreachable but
// keeps the match exhaustive without an else-clause.
ChecksumAlg::Blake3 => unreachable!(),
}
})
.await
.map_err(|e| std::io::Error::other(format!("hash task join failed: {e}")))?
}
}
}
/// POST /api/uploads/:upload_id/complete - Finalize upload /// POST /api/uploads/:upload_id/complete - Finalize upload
/// ///
/// Assembles all chunks into the final file and creates the file record. /// Streams the uploaded chunk parts, in order, straight into the CDC
/// When `body.checksum` is supplied, the assembled file's hash is /// chunk store and creates the file record — no assembled temp file is
/// verified before the blob is promoted to storage — mismatch /// ever written. When `body.checksum` is supplied it is verified from
/// returns 400 and the assembled temp is removed (the session /// the same streaming pass (BLAKE3 comes from the store itself;
/// itself is kept so the client can re-issue complete after /// MD5/SHA-256 are computed by an in-flight tee) — mismatch returns 400
/// diagnosing). /// with the blob reference released, and the chunk parts stay on disk
/// so the client can re-issue complete after diagnosing.
pub(super) async fn complete_upload_impl( pub(super) async fn complete_upload_impl(
State(state): State<Arc<AppState>>, State(state): State<Arc<AppState>>,
auth_user: AuthUser, auth_user: AuthUser,
@@ -379,10 +321,11 @@ impl ChunkedUploadHandler {
) -> impl IntoResponse { ) -> impl IntoResponse {
let chunked_service = &state.core.chunked_upload_service; let chunked_service = &state.core.chunked_upload_service;
let upload_service = &state.applications.file_upload_service; let upload_service = &state.applications.file_upload_service;
let dedup = &state.core.dedup_service;
// ── Parse the optional algorithm BEFORE assembly so a bad // ── Parse the optional algorithm BEFORE completion so a bad
// `checksumalg` doesn't waste the (potentially expensive) // `checksumalg` doesn't waste any work on a request we'll
// hash work on a request we'll reject anyway. // reject anyway.
let alg = match body.checksumalg.as_deref() { let alg = match body.checksumalg.as_deref() {
Some(name) => match ChecksumAlg::parse(name) { Some(name) => match ChecksumAlg::parse(name) {
Some(a) => Some(a), Some(a) => Some(a),
@@ -397,37 +340,54 @@ impl ChunkedUploadHandler {
}; };
let expected_checksum = body.checksum.as_deref(); let expected_checksum = body.checksum.as_deref();
// Assemble chunks (hash-on-write: BLAKE3 computed during assembly) // Validate completion and get the chunk parts in assembly order.
let (assembled_path, filename, folder_id, content_type, total_size, hash) = let parts = match chunked_service
match chunked_service .complete_upload(&upload_id, auth_user.id)
.complete_upload(&upload_id, auth_user.id) .await
.await {
{ Ok(result) => result,
Ok(result) => result, Err(e) => {
Err(e) => { return AppError::from(e).into_response();
return AppError::from(e).into_response(); }
} };
};
// MD5/SHA-256 verification taps the stream while it is ingested;
// BLAKE3 needs no tee — the store's own content hash IS BLAKE3.
let alg = expected_checksum.map(|_| alg.unwrap_or(ChecksumAlg::Blake3));
let tee = match alg {
Some(ChecksumAlg::Md5) | Some(ChecksumAlg::Sha256) => {
Some(upload_ingest::checksum_tee(alg.unwrap()))
}
_ => None,
};
// ── Stream the parts into the CDC chunk store ───────────────
let ingested = match upload_ingest::ingest_stream_to_cas(
upload_ingest::stream_from_files(parts.chunk_paths),
dedup,
&parts.filename,
&parts.content_type,
usize::MAX,
tee.clone(),
)
.await
{
Ok(ingested) => ingested,
Err(e) => return e.into_response(),
};
// ── End-to-end integrity verification ─────────────────────── // ── End-to-end integrity verification ───────────────────────
// Only fires when the client supplied an `expected` checksum. if let (Some(expected), Some(alg)) = (expected_checksum, alg) {
// For BLAKE3 (the documented preferred choice) this is a string let computed = match alg {
// comparison against the hash assembly already produced. For ChecksumAlg::Blake3 => Some(ingested.hash.clone()),
// MD5/SHA-256 we re-hash the assembled file on the blocking pool. _ => tee.as_ref().and_then(upload_ingest::finalize_checksum_tee),
if let Some(expected) = expected_checksum { };
let alg = alg.unwrap_or(ChecksumAlg::Blake3); let Some(computed) = computed else {
let computed = match Self::compute_assembled_hash(&assembled_path, alg, &hash).await { upload_ingest::discard_ingested(dedup, &ingested).await;
Ok(c) => c, return AppError::internal_error("Checksum tee produced no digest").into_response();
Err(e) => {
let _ = tokio::fs::remove_file(&assembled_path).await;
return AppError::internal_error(format!(
"Failed to compute assembled checksum: {e}"
))
.into_response();
}
}; };
if !computed.eq_ignore_ascii_case(expected) { if !computed.eq_ignore_ascii_case(expected) {
let _ = tokio::fs::remove_file(&assembled_path).await; upload_ingest::discard_ingested(dedup, &ingested).await;
tracing::warn!( tracing::warn!(
target: "audit", target: "audit",
event = "chunked_upload.checksum_mismatch", event = "chunked_upload.checksum_mismatch",
@@ -449,36 +409,28 @@ impl ChunkedUploadHandler {
} }
} }
// ── MIME detection (magic bytes + extension fallback) ───── // Register the file row against the ingested blob.
let content_type = crate::common::mime_detect::refine_content_type_from_file( let size = ingested.size;
&assembled_path,
&filename,
&content_type,
)
.await;
// Upload from assembled file on disk — zero extra RAM copies, hash pre-computed
match upload_service match upload_service
.upload_file_from_path( .upload_file_streaming(
filename.clone(), parts.filename.clone(),
folder_id.clone(), parts.folder_id.clone(),
content_type, ingested.content_type.clone(),
&assembled_path, ingested.stored(),
Some(hash),
) )
.await .await
{ {
Ok(file) => { Ok(file) => {
// Cleanup session // Cleanup session (removes the chunk part files)
let _ = chunked_service let _ = chunked_service
.finalize_upload(&upload_id, auth_user.id) .finalize_upload(&upload_id, auth_user.id)
.await; .await;
tracing::info!( tracing::info!(
"✅ CHUNKED UPLOAD COMPLETE: {} (ID: {}, {} bytes)", "✅ CHUNKED UPLOAD COMPLETE: {} (ID: {}, {} bytes)",
filename, parts.filename,
file.id, file.id,
total_size size
); );
( (
@@ -486,14 +438,14 @@ impl ChunkedUploadHandler {
Json(CompleteUploadResponse { Json(CompleteUploadResponse {
file_id: file.id, file_id: file.id,
filename: file.name, filename: file.name,
size: total_size, size,
path: file.path, path: file.path,
}), }),
) )
.into_response() .into_response()
} }
Err(e) => { Err(e) => {
tracing::error!("Failed to create file from assembled upload: {:?}", e); tracing::error!("Failed to create file from chunked upload: {:?}", e);
AppError::internal_error(format!("Failed to create file: {}", e)).into_response() AppError::internal_error(format!("Failed to create file: {}", e)).into_response()
} }
} }
+49 -112
View File
@@ -5,12 +5,11 @@ use axum::{
response::IntoResponse, response::IntoResponse,
}; };
use serde::Serialize; use serde::Serialize;
use tokio::io::AsyncWriteExt;
use utoipa::ToSchema; use utoipa::ToSchema;
use crate::application::ports::dedup_ports::DedupResultDto;
use crate::common::di::AppState; use crate::common::di::AppState;
use crate::interfaces::middleware::auth::AuthUser; use crate::interfaces::middleware::auth::AuthUser;
use crate::interfaces::upload_ingest;
use std::sync::Arc; use std::sync::Arc;
/// Global application state for dependency injection /// Global application state for dependency injection
@@ -155,10 +154,10 @@ impl DedupHandler {
/// Upload content with automatic deduplication (streaming). /// Upload content with automatic deduplication (streaming).
/// ///
/// Spools the upload to a temp file while computing the BLAKE3 hash /// Streams the multipart field straight into the CDC chunk store —
/// incrementally (hash-on-write). Memory usage is constant (~512 KB) /// chunking, BLAKE3 hashing and dedup checks happen while the bytes
/// regardless of file size. Then delegates to `store_from_file` with /// arrive (no temp file, no re-read; peak RAM is bounded regardless
/// the pre-computed hash so the file is never re-read for hashing. /// of file size).
/// ///
/// POST /api/dedup/upload /// POST /api/dedup/upload
pub(super) async fn upload_with_dedup_impl( pub(super) async fn upload_with_dedup_impl(
@@ -177,64 +176,29 @@ impl DedupHandler {
.content_type() .content_type()
.unwrap_or("application/octet-stream") .unwrap_or("application/octet-stream")
.to_string(); .to_string();
let filename = field.file_name().unwrap_or("unnamed").to_string();
// ── Spool to temp file + BLAKE3 hash-on-write ──────── // ── Stream into the CDC chunk store ──────────────────
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp"); let source = upload_ingest::multipart_field_stream(field);
let temp_path = temp_dir.join(format!("dedup-{}", uuid::Uuid::new_v4())); let ingested = match upload_ingest::ingest_stream_to_cas(
source,
let mut total_size: u64 = 0; dedup,
let mut hasher = blake3::Hasher::new(); &filename,
let mut field = field; &content_type,
usize::MAX,
let spool_result: Result<(), String> = async { None,
let file = tokio::fs::File::create(&temp_path) )
.await .await
.map_err(|e| format!("Failed to create temp file: {}", e))?; {
Ok(ingested) => ingested,
// 512 KB buffer — reduces write syscalls Err(e) => {
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file); tracing::warn!("Dedup upload ingest failed: {}", e.message);
return e.into_response();
loop {
match field.chunk().await {
Ok(Some(chunk)) => {
total_size += chunk.len() as u64;
hasher.update(&chunk);
writer
.write_all(&chunk)
.await
.map_err(|e| format!("Failed to write chunk: {}", e))?;
}
Ok(None) => break,
Err(e) => {
return Err(format!(
"Connection lost during upload (received {} bytes): {}",
total_size, e
));
}
}
} }
};
writer if ingested.size == 0 {
.flush() upload_ingest::discard_ingested(dedup, &ingested).await;
.await
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
Ok(())
}
.await;
if let Err(msg) = spool_result {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::warn!("Dedup upload spool failed: {}", msg);
return Response::builder()
.status(StatusCode::BAD_REQUEST)
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(format!(r#"{{"error": "{}"}}"#, msg)))
.unwrap()
.into_response();
}
if total_size == 0 {
let _ = tokio::fs::remove_file(&temp_path).await;
return Response::builder() return Response::builder()
.status(StatusCode::BAD_REQUEST) .status(StatusCode::BAD_REQUEST)
.header(header::CONTENT_TYPE, "application/json") .header(header::CONTENT_TYPE, "application/json")
@@ -243,60 +207,33 @@ impl DedupHandler {
.into_response(); .into_response();
} }
let hash = hasher.finalize().to_hex().to_string(); let metadata = dedup.get_blob_metadata(&ingested.hash).await;
// ── Store with deduplication (pre-computed hash) ────── let response = DedupUploadResponse {
match dedup is_new: ingested.is_new_blob,
.store_from_file(&temp_path, Some(content_type), Some(hash)) hash: ingested.hash.clone(),
.await size: ingested.size,
{ bytes_saved: ingested.bytes_saved,
Ok(result) => { ref_count: metadata.map(|m| m.ref_count).unwrap_or(1),
let (is_new, bytes_saved) = match &result { };
DedupResultDto::NewBlob { .. } => (true, 0),
DedupResultDto::ExistingBlob { saved_bytes, .. } => {
(false, *saved_bytes)
}
};
let metadata = dedup.get_blob_metadata(result.hash()).await; tracing::info!(
"🔗 Dedup upload: hash={}, new={}, saved={}",
ingested.hash,
ingested.is_new_blob,
ingested.bytes_saved
);
let response = DedupUploadResponse { return Response::builder()
is_new, .status(if ingested.is_new_blob {
hash: result.hash().to_string(), StatusCode::CREATED
size: result.size(), } else {
bytes_saved, StatusCode::OK
ref_count: metadata.map(|m| m.ref_count).unwrap_or(1), })
}; .header(header::CONTENT_TYPE, "application/json")
.body(Body::from(serde_json::to_string(&response).unwrap()))
tracing::info!( .unwrap()
"🔗 Dedup upload: hash={}, new={}, saved={}", .into_response();
result.hash(),
is_new,
bytes_saved
);
return Response::builder()
.status(if is_new {
StatusCode::CREATED
} else {
StatusCode::OK
})
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(serde_json::to_string(&response).unwrap()))
.unwrap()
.into_response();
}
Err(e) => {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("Dedup upload failed: {}", e);
return Response::builder()
.status(StatusCode::INTERNAL_SERVER_ERROR)
.header(header::CONTENT_TYPE, "application/json")
.body(Body::from(r#"{"error": "Upload failed"}"#))
.unwrap()
.into_response();
}
}
} }
} }
+37 -101
View File
@@ -21,6 +21,7 @@ use crate::common::di::AppState;
use crate::interfaces::errors::AppError; use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::AuthUser; use crate::interfaces::middleware::auth::AuthUser;
use crate::interfaces::range_requests::not_modified_response; use crate::interfaces::range_requests::not_modified_response;
use crate::interfaces::upload_ingest;
use crate::{application::dtos::file_dto::FileDto, domain::services::authorization::Permission}; use crate::{application::dtos::file_dto::FileDto, domain::services::authorization::Permission};
use std::sync::Arc; use std::sync::Arc;
@@ -51,11 +52,12 @@ impl FileHandler {
// UPLOAD // UPLOAD
// ═══════════════════════════════════════════════════════════════════════ // ═══════════════════════════════════════════════════════════════════════
/// Streaming file upload — constant ~64 KB RAM regardless of file size. /// Streaming file upload — bounded RAM regardless of file size.
/// ///
/// **Hash-on-Write**: BLAKE3 is computed while spooling the multipart /// The multipart body is streamed straight into the CDC chunk store:
/// body to the temp file. This eliminates the second sequential read /// chunking, hashing and dedup checks happen while the bytes arrive.
/// that dedup_service would otherwise need, cutting total I/O in half. /// No spool file, no re-read — chunks the store already has are never
/// written to disk at all.
pub async fn upload_file( pub async fn upload_file(
State(state): State<GlobalState>, State(state): State<GlobalState>,
auth_user: AuthUser, auth_user: AuthUser,
@@ -71,7 +73,7 @@ impl FileHandler {
/// [`Self::upload_file_with_thumbnails`]. /// [`Self::upload_file_with_thumbnails`].
/// ///
/// Returns `(FileDto, blob_hash)` on success. The blob hash is the /// Returns `(FileDto, blob_hash)` on success. The blob hash is the
/// BLAKE3 digest computed during the hash-on-write spool and is /// BLAKE3 digest computed during the streaming ingest and is
/// propagated without an extra database round-trip so that callers /// propagated without an extra database round-trip so that callers
/// (e.g. thumbnail generation) can resolve the physical blob path /// (e.g. thumbnail generation) can resolve the physical blob path
/// immediately. /// immediately.
@@ -158,120 +160,55 @@ impl FileHandler {
} }
} }
// ── Spool multipart field to temp file + hash-on-write ── // ── Stream the field into the CDC chunk store ────────
// .dedup_temp is created once by DedupService::initialize() at startup // Chunking (FastCDC) + hashing (BLAKE3) + dedup checks +
let temp_dir = state.core.path_service.get_root_path().join(".dedup_temp"); // MIME sniffing all happen while the bytes arrive; chunks
let temp_path = temp_dir.join(format!("upload-{}", uuid::Uuid::new_v4())); // the store already has never touch the disk. Size is
// capped globally by DefaultBodyLimit.
let mut total_size: u64 = 0; let dedup = &state.core.dedup_service;
let mut hasher = blake3::Hasher::new(); let source = upload_ingest::multipart_field_stream(field);
let spool_result: Result<(), String> = async { let ingested = match upload_ingest::ingest_stream_to_cas(
let file = tokio::fs::File::create(&temp_path) source,
.await dedup,
.map_err(|e| format!("Failed to create temp file: {}", e))?;
// Pre-allocate if Content-Length is known (reduces fragmentation)
let hint = field
.headers()
.get(axum::http::header::CONTENT_LENGTH)
.and_then(|v| v.to_str().ok())
.and_then(|s| s.parse::<u64>().ok());
if let Some(len) = hint {
let _ = file.set_len(len).await; // best-effort
}
// 512 KB buffer — 8× fewer write syscalls than 64 KB
let mut writer = tokio::io::BufWriter::with_capacity(524_288, file);
let mut field = field;
// IMPORTANT: use explicit match instead of `while let Ok(Some(..))`.
// The old pattern silently swallowed Err (client disconnect)
// and accepted partially received data as a complete upload.
loop {
match field.chunk().await {
Ok(Some(chunk)) => {
total_size += chunk.len() as u64;
hasher.update(&chunk);
tokio::io::AsyncWriteExt::write_all(&mut writer, &chunk)
.await
.map_err(|e| format!("Failed to write chunk: {}", e))?;
}
Ok(None) => break, // End of field — upload complete
Err(e) => {
return Err(format!(
"Connection lost during upload (received {} bytes): {}",
total_size, e
));
}
}
}
tokio::io::AsyncWriteExt::flush(&mut writer)
.await
.map_err(|e| format!("Failed to flush temp file: {}", e))?;
Ok(())
}
.await;
if let Err(e) = spool_result {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("❌ UPLOAD SPOOL FAILED: {} - {}", filename, e);
return Err(Self::domain_error_response(
crate::common::errors::DomainError::internal_error("FileUpload", e),
));
}
// Empty file — use streaming path with the (empty) temp file
if total_size == 0 {
let hash = hasher.finalize().to_hex().to_string();
let dto = upload_service
.upload_file_streaming(
filename,
folder_id,
content_type,
&temp_path,
0,
Some(hash.clone()),
)
.await
.map_err(Self::domain_error_response)?;
return Ok((dto, hash));
}
// Finalize hash
let hash = hasher.finalize().to_hex().to_string();
// ── MIME detection (magic bytes + extension fallback) ─
let content_type = crate::common::mime_detect::refine_content_type_from_file(
&temp_path,
&filename, &filename,
&content_type, &content_type,
usize::MAX,
None,
) )
.await; .await
{
Ok(ingested) => ingested,
Err(e) => {
tracing::error!("❌ UPLOAD INGEST FAILED: {} - {}", filename, e.message);
return Err(e.into_response());
}
};
// ── Quota enforcement ──────────────────────────────── // ── Quota enforcement (exact size now known) ─────────
if let Some(storage_svc) = state.storage_usage_service.as_ref() if let Some(storage_svc) = state.storage_usage_service.as_ref()
&& let Err(err) = storage_svc && let Err(err) = storage_svc
.check_storage_quota(auth_user.id, total_size) .check_storage_quota(auth_user.id, ingested.size)
.await .await
{ {
let _ = tokio::fs::remove_file(&temp_path).await; upload_ingest::discard_ingested(dedup, &ingested).await;
tracing::warn!( tracing::warn!(
"⛔ UPLOAD REJECTED (quota): user={}, file={}, size={}", "⛔ UPLOAD REJECTED (quota): user={}, file={}, size={}",
auth_user.username, auth_user.username,
filename, filename,
total_size ingested.size
); );
return Err(Self::quota_error_response(err)); return Err(Self::quota_error_response(err));
} }
// ── Streaming upload (temp file → blob store, hash pre-computed) ─ // ── Register the file row against the ingested blob ──
let hash = ingested.hash.clone();
let size = ingested.size;
match upload_service match upload_service
.upload_file_streaming( .upload_file_streaming(
filename.clone(), filename.clone(),
folder_id, folder_id,
content_type, ingested.content_type.clone(),
&temp_path, ingested.stored(),
total_size,
Some(hash.clone()),
) )
.await .await
{ {
@@ -279,13 +216,12 @@ impl FileHandler {
tracing::info!( tracing::info!(
"✅ STREAMING UPLOAD: {} ({} bytes, ID: {})", "✅ STREAMING UPLOAD: {} ({} bytes, ID: {})",
filename, filename,
total_size, size,
file.id file.id
); );
return Ok((file, hash)); return Ok((file, hash));
} }
Err(err) => { Err(err) => {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("❌ UPLOAD FAILED: {} - {}", filename, err); tracing::error!("❌ UPLOAD FAILED: {} - {}", filename, err);
return Err(Self::domain_error_response(err)); return Err(Self::domain_error_response(err));
} }
+20 -29
View File
@@ -898,10 +898,11 @@ async fn handle_head(
/** /**
* Handles PUT requests to create or update files. * Handles PUT requests to create or update files.
* *
* **Streaming implementation**: the request body is spooled to a temp file * **Streaming implementation**: the request body is streamed straight into
* with incremental BLAKE3 hashing. Peak RAM usage is ~256 KB regardless * the CDC chunk store (FastCDC + BLAKE3 while the bytes arrive — no spool
* of file size. The temp file is then atomically moved into blob storage * file, no re-read; peak RAM is bounded regardless of file size), then the
* via `update_file_streaming`. * file row is atomically swapped onto the ingested blob via
* `update_file_streaming`.
* *
* @param state The application state containing service dependencies * @param state The application state containing service dependencies
* @param path The requested resource path * @param path The requested resource path
@@ -913,7 +914,7 @@ async fn handle_put(
req: Request<Body>, req: Request<Body>,
path: String, path: String,
) -> Result<Response<Body>, AppError> { ) -> Result<Response<Body>, AppError> {
use crate::interfaces::upload_spool::spool_body_to_temp; use crate::interfaces::upload_ingest;
let user = extract_user(&req)?; let user = extract_user(&req)?;
@@ -972,32 +973,31 @@ async fn handle_put(
.unwrap_or("application/octet-stream") .unwrap_or("application/octet-stream")
.to_string(); .to_string();
// ── Streaming spool: body → temp file + incremental hash ── // ── Streaming ingest: body → CDC chunk store ──────────────
// Shared with the NextCloud-compat PUT handler; peak heap ~one frame // Shared with the NextCloud-compat PUT handler; chunking + hashing +
// regardless of file size. Honors `upload_temp_dir` to keep the spool // dedup checks run while the body arrives — no spool file, no re-read.
// off tmpfs/RAM. let filename = crate::common::mime_detect::filename_from_path(&path).to_string();
let spooled = spool_body_to_temp( let ingested = upload_ingest::ingest_body_to_cas(
req.into_body(), req.into_body(),
&state.core.dedup_service,
&filename,
&content_type,
max_upload, max_upload,
state.core.config.storage.upload_temp_dir.clone(),
) )
.await?; .await?;
let temp_path = spooled.temp.path().to_path_buf();
let total_bytes = spooled.size as usize;
let hash = spooled.hash;
// ── Quota enforcement ──────────────────────────────────── // ── Quota enforcement ────────────────────────────────────
if let Some(storage_svc) = state.storage_usage_service.as_ref() if let Some(storage_svc) = state.storage_usage_service.as_ref()
&& let Err(err) = storage_svc && let Err(err) = storage_svc
.check_storage_quota(user.id, total_bytes as u64) .check_storage_quota(user.id, ingested.size)
.await .await
{ {
let _ = tokio::fs::remove_file(&temp_path).await; upload_ingest::discard_ingested(&state.core.dedup_service, &ingested).await;
tracing::warn!( tracing::warn!(
"⛔ WEBDAV PUT REJECTED (quota): user={}, file={}, size={}", "⛔ WEBDAV PUT REJECTED (quota): user={}, file={}, size={}",
user.id, user.id,
path, path,
total_bytes ingested.size
); );
return Err(AppError::new( return Err(AppError::new(
StatusCode::INSUFFICIENT_STORAGE, StatusCode::INSUFFICIENT_STORAGE,
@@ -1006,21 +1006,12 @@ async fn handle_put(
)); ));
} }
// ── Atomic store: temp file → dedup blob + DB metadata update ── // ── Atomic store: swap the file row onto the ingested blob ──
let content_type = ingested.content_type.clone();
let result = file_upload_service let result = file_upload_service
.update_file_streaming( .update_file_streaming(&path, ingested.stored(), &content_type, None)
&path,
&temp_path,
total_bytes as u64,
&content_type,
Some(hash),
None,
)
.await; .await;
// Clean up temp file (may already be moved by dedup, ignore error)
let _ = tokio::fs::remove_file(&temp_path).await;
match result { match result {
Ok(_file_dto) => Ok(Response::builder() Ok(_file_dto) => Ok(Response::builder()
.status(StatusCode::NO_CONTENT) .status(StatusCode::NO_CONTENT)
+17 -64
View File
@@ -165,10 +165,6 @@ async fn put_file(
State(state): State<WopiState>, State(state): State<WopiState>,
req: Request<Body>, req: Request<Body>,
) -> Response { ) -> Response {
use http_body_util::BodyStream;
use tokio::io::AsyncWriteExt;
use tokio_stream::StreamExt;
let claims = match state let claims = match state
.token_service .token_service
.validate_token(&token_query.access_token) .validate_token(&token_query.access_token)
@@ -218,75 +214,32 @@ async fn put_file(
Err(_) => return StatusCode::NOT_FOUND.into_response(), Err(_) => return StatusCode::NOT_FOUND.into_response(),
}; };
// ── Streaming spool: body → temp file + incremental BLAKE3 ── // ── Streaming ingest: body → CDC chunk store (no temp file) ──
let temp_file = match tempfile::NamedTempFile::new() {
Ok(f) => f,
Err(e) => {
tracing::error!("WOPI PutFile: failed to create temp file: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
};
let temp_path = temp_file.path().to_path_buf();
let mut file_out = match tokio::fs::File::create(&temp_path).await {
Ok(f) => f,
Err(e) => {
tracing::error!("WOPI PutFile: failed to open temp file: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
};
let content_type = file.mime_type.clone(); let content_type = file.mime_type.clone();
let mut hasher = blake3::Hasher::new(); let ingested = match crate::interfaces::upload_ingest::ingest_body_to_cas(
let mut total_bytes: u64 = 0; req.into_body(),
let mut stream = BodyStream::new(req.into_body()); &state.app_state.core.dedup_service,
&file.name,
while let Some(frame_result) = stream.next().await { &content_type,
let frame = match frame_result { usize::MAX,
Ok(f) => f, )
Err(e) => { .await
let _ = tokio::fs::remove_file(&temp_path).await; {
tracing::error!("WOPI PutFile: body read error: {}", e); Ok(ingested) => ingested,
return StatusCode::INTERNAL_SERVER_ERROR.into_response(); Err(e) => {
} tracing::error!("WOPI PutFile: ingest failed: {}", e.message);
}; return StatusCode::INTERNAL_SERVER_ERROR.into_response();
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len() as u64;
hasher.update(chunk);
if let Err(e) = file_out.write_all(chunk).await {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("WOPI PutFile: temp write error: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
} }
} };
if let Err(e) = file_out.flush().await {
let _ = tokio::fs::remove_file(&temp_path).await;
tracing::error!("WOPI PutFile: flush error: {}", e);
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
}
drop(file_out);
let hash = hasher.finalize().to_hex().to_string(); // ── Atomic store: swap the file row onto the ingested blob ──
// ── Atomic store: temp file → dedup blob + DB metadata update ──
let result = state let result = state
.app_state .app_state
.applications .applications
.file_upload_service .file_upload_service
.update_file_streaming( .update_file_streaming(&file.path, ingested.stored(), &content_type, None)
&file.path,
&temp_path,
total_bytes,
&content_type,
Some(hash),
None,
)
.await; .await;
// Clean up temp file (may already be moved by dedup, ignore error)
let _ = tokio::fs::remove_file(&temp_path).await;
match result { match result {
Ok(_file_dto) => StatusCode::OK.into_response(), Ok(_file_dto) => StatusCode::OK.into_response(),
Err(e) => { Err(e) => {
+1 -1
View File
@@ -3,7 +3,7 @@ pub mod errors;
pub mod middleware; pub mod middleware;
pub mod nextcloud; pub mod nextcloud;
pub mod range_requests; pub mod range_requests;
pub mod upload_spool; pub mod upload_ingest;
pub mod web; pub mod web;
pub use api::create_api_routes; pub use api::create_api_routes;
+38 -39
View File
@@ -7,10 +7,12 @@ use std::sync::Arc;
use crate::application::ports::file_ports::{FileRetrievalUseCase, FileUploadUseCase}; use crate::application::ports::file_ports::{FileRetrievalUseCase, FileUploadUseCase};
use crate::common::di::AppState; use crate::common::di::AppState;
use crate::common::mime_detect::{filename_from_path, refine_content_type_from_file}; use crate::common::mime_detect::filename_from_path;
use crate::interfaces::errors::AppError; use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::{AuthUser, CurrentUser}; use crate::interfaces::middleware::auth::{AuthUser, CurrentUser};
use crate::interfaces::upload_spool::stream_body_to_path; use crate::interfaces::upload_ingest::{
discard_ingested, ingest_stream_to_cas, stream_body_to_path, stream_from_files,
};
/// Dispatch Nextcloud chunked upload WebDAV requests. /// Dispatch Nextcloud chunked upload WebDAV requests.
/// ///
@@ -239,18 +241,17 @@ async fn handle_assemble(
let dest_subpath = extract_files_subpath(&destination, &user.username) let dest_subpath = extract_files_subpath(&destination, &user.username)
.ok_or_else(|| AppError::bad_request("Invalid Destination URL"))?; .ok_or_else(|| AppError::bad_request("Invalid Destination URL"))?;
// Assemble chunks into a temp file with hash-on-write (BLAKE3 computed // Stream the chunk parts, in order, straight into the CDC chunk store —
// during the same read/write loop that copies chunks into the // no assembled temp file is ever written. Chunking (FastCDC), BLAKE3
// assembled file). The hash is passed downstream as `pre_computed_hash` // hashing, dedup checks and MIME sniffing (magic bytes off the first
// so the dedup layer never re-reads the assembled file just to compute // part) all happen in that single read pass. The parts stay on disk
// it — saves one full file-sized read pass per upload. // until the session cleanup below, so a failed completion is retryable.
let (temp_path, size, blake3_hash) = nc let chunk_paths = nc
.chunked_uploads .chunked_uploads
.assemble(&user.username, upload_id) .ordered_chunk_paths(&user.username, upload_id)
.await .await
.map_err(|e| AppError::internal_error(format!("Failed to assemble chunks: {}", e)))?; .map_err(|e| AppError::internal_error(format!("Failed to list chunks: {}", e)))?;
// Write assembled file to storage via the upload service.
let upload_service = &state.applications.file_upload_service; let upload_service = &state.applications.file_upload_service;
let file_service = &state.applications.file_retrieval_service; let file_service = &state.applications.file_retrieval_service;
let folder_service = &state.applications.folder_service; let folder_service = &state.applications.folder_service;
@@ -261,35 +262,31 @@ async fn handle_assemble(
dest_subpath.trim_matches('/') dest_subpath.trim_matches('/')
); );
// Detect content type via magic bytes + extension fallback. let filename = filename_from_path(&dest_subpath).to_string();
let filename = filename_from_path(&dest_subpath); let ingested = ingest_stream_to_cas(
let content_type = stream_from_files(chunk_paths),
refine_content_type_from_file(&temp_path, filename, "application/octet-stream").await; &state.core.dedup_service,
&filename,
"application/octet-stream",
usize::MAX,
None,
)
.await?;
let content_type = ingested.content_type.clone();
// Check if file exists (update vs create). // Check if file exists (update vs create).
let existing = file_service.get_file_by_path(&internal_path).await; let existing = file_service.get_file_by_path(&internal_path).await;
let etag: Option<String> = if existing.is_ok() { let etag: Option<String> = if existing.is_ok() {
let dto = upload_service let dto = upload_service
.update_file_streaming( .update_file_streaming(&internal_path, ingested.stored(), &content_type, oc_mtime)
&internal_path,
&temp_path,
size,
&content_type,
Some(blake3_hash.clone()),
oc_mtime,
)
.await .await
.map_err(|e| AppError::internal_error(format!("Failed to update file: {}", e)))?; .map_err(|e| AppError::internal_error(format!("Failed to update file: {}", e)))?;
Some(dto.etag) Some(dto.etag)
} else { } else {
// New-file branch: resolve the parent folder by path and pass the // New-file branch: resolve the parent folder by path and register
// assembled file's path directly to `upload_file_from_path` so the // the file row against the already-ingested blob.
// bytes never get read back into RAM. Previously this branch did
// `tokio::fs::read(&temp_path)` — an extra full file-sized read
// pass AND a peak-RAM allocation equal to the upload size, which
// defeated the streaming model on large NC uploads.
let (parent_sub, filename) = match dest_subpath.rsplit_once('/') { let (parent_sub, filename) = match dest_subpath.rsplit_once('/') {
Some((p, n)) => (p, n), Some((p, n)) => (p, n),
None => ("", dest_subpath.as_str()), None => ("", dest_subpath.as_str()),
@@ -302,18 +299,23 @@ async fn handle_assemble(
let parent_internal = parent_internal.trim_end_matches('/'); let parent_internal = parent_internal.trim_end_matches('/');
use crate::application::ports::folder_ports::FolderUseCase; use crate::application::ports::folder_ports::FolderUseCase;
let parent_folder = folder_service let parent_folder = match folder_service.get_folder_by_path(parent_internal).await {
.get_folder_by_path(parent_internal) Ok(folder) => folder,
.await Err(e) => {
.map_err(|e| AppError::internal_error(format!("Parent folder lookup failed: {}", e)))?; discard_ingested(&state.core.dedup_service, &ingested).await;
return Err(AppError::internal_error(format!(
"Parent folder lookup failed: {}",
e
)));
}
};
let dto = upload_service let dto = upload_service
.upload_file_from_path( .upload_file_streaming(
filename.to_string(), filename.to_string(),
Some(parent_folder.id), Some(parent_folder.id),
content_type.to_string(), content_type.to_string(),
&temp_path, ingested.stored(),
Some(blake3_hash),
) )
.await .await
.map_err(|e| AppError::internal_error(format!("Failed to create file: {}", e)))?; .map_err(|e| AppError::internal_error(format!("Failed to create file: {}", e)))?;
@@ -321,9 +323,6 @@ async fn handle_assemble(
Some(dto.etag) Some(dto.etag)
}; };
// Clean up temp file (session cleanup below removes the directory anyway).
let _ = tokio::fs::remove_file(&temp_path).await;
// Cleanup session. // Cleanup session.
let _ = nc.chunked_uploads.cleanup(&user.username, upload_id).await; let _ = nc.chunked_uploads.cleanup(&user.username, upload_id).await;
+19 -30
View File
@@ -22,12 +22,12 @@ use crate::application::ports::file_ports::{
use crate::application::ports::folder_ports::FolderUseCase; use crate::application::ports::folder_ports::FolderUseCase;
use crate::application::ports::trash_ports::TrashUseCase; use crate::application::ports::trash_ports::TrashUseCase;
use crate::common::di::AppState; use crate::common::di::AppState;
use crate::common::mime_detect::{filename_from_path, refine_content_type_from_file}; use crate::common::mime_detect::filename_from_path;
use crate::interfaces::api::handlers::webdav_handler::PROPFIND_BATCH_SIZE; use crate::interfaces::api::handlers::webdav_handler::PROPFIND_BATCH_SIZE;
use crate::interfaces::errors::AppError; use crate::interfaces::errors::AppError;
use crate::interfaces::middleware::auth::{AuthUser, CurrentUser}; use crate::interfaces::middleware::auth::{AuthUser, CurrentUser};
use crate::interfaces::range_requests::{not_modified_response, range_response}; use crate::interfaces::range_requests::{not_modified_response, range_response};
use crate::interfaces::upload_spool::spool_body_to_temp; use crate::interfaces::upload_ingest::ingest_body_to_cas;
/// Extension trait to map XML write errors to `String` concisely. /// Extension trait to map XML write errors to `String` concisely.
trait XmlResultExt<T> { trait XmlResultExt<T> {
@@ -575,46 +575,35 @@ async fn handle_put(
// at 95 % loses everything. // at 95 % loses everything.
let max_upload = state.core.config.storage.direct_put_max_bytes; let max_upload = state.core.config.storage.direct_put_max_bytes;
// Stream the body to a temp file + incremental hash — never buffer the // Stream the body straight into the CDC chunk store — never buffer the
// full upload in RAM. The old `body::to_bytes` path loaded the entire // full upload in RAM and never spool it to disk. Chunking, hashing,
// file (e.g. an 800 MB ISO) into anonymous memory before any dedup logic, // dedup checks and MIME sniffing (magic bytes off the first frames)
// OOMKilling the process even on dedup hits. Shared with the native // all run while the body arrives; chunks the store already has are
// WebDAV PUT handler; peak heap ~one frame regardless of file size. // never written at all. Shared with the native WebDAV PUT handler.
let spooled = spool_body_to_temp( //
req.into_body(),
max_upload,
state.core.config.storage.upload_temp_dir.clone(),
)
.await?;
// Detect real MIME type from the first bytes on disk (no full read).
// `filename` is owned so we don't hold a borrow of the `subpath` param // `filename` is owned so we don't hold a borrow of the `subpath` param
// across the await (which would make the handler future non-Send). // across the await (which would make the handler future non-Send).
let filename = filename_from_path(subpath).to_string(); let filename = filename_from_path(subpath).to_string();
let content_type = let ingested = ingest_body_to_cas(
refine_content_type_from_file(spooled.temp.path(), &filename, &claimed_type).await; req.into_body(),
&state.core.dedup_service,
&filename,
&claimed_type,
max_upload,
)
.await?;
let content_type = ingested.content_type.clone();
// Distinguish create (201) vs update (204) for the response status. // Distinguish create (201) vs update (204) for the response status.
let existed = file_service.get_file_by_path(&internal_path).await.is_ok(); let existed = file_service.get_file_by_path(&internal_path).await.is_ok();
// Single streaming path — handles both update and create internally, // Single streaming path — handles both update and create internally,
// passing the precomputed hash so the dedup fast path can short-circuit // swapping the file row onto the already-ingested blob.
// without re-reading the file.
let stored = upload_service let stored = upload_service
.update_file_streaming( .update_file_streaming(&internal_path, ingested.stored(), &content_type, oc_mtime)
&internal_path,
spooled.temp.path(),
spooled.size,
&content_type,
Some(spooled.hash),
oc_mtime,
)
.await .await
.map_err(|e| AppError::internal_error(format!("Failed to store file: {}", e)))?; .map_err(|e| AppError::internal_error(format!("Failed to store file: {}", e)))?;
// dedup may have already moved the temp on a new-blob store; ignore error.
let _ = tokio::fs::remove_file(spooled.temp.path()).await;
let status = if existed { let status = if existed {
StatusCode::NO_CONTENT StatusCode::NO_CONTENT
} else { } else {
+588
View File
@@ -0,0 +1,588 @@
//! Shared streaming upload ingestion: request body → CDC chunk store.
//!
//! Used by every upload surface (REST multipart, native WebDAV PUT,
//! NextCloud PUT, chunked-upload assembly, WOPI PutFile) so none of them
//! buffers the full body in RAM **or spools it to a temp file**. The bytes
//! flow straight into [`DedupService::store_from_stream`], which chunks
//! (FastCDC), hashes (BLAKE3) and dedup-checks them while they arrive —
//! each uploaded byte touches the disk at most once, and not at all when
//! the store already has its chunk.
//!
//! MIME refinement happens in-flight: when the claimed Content-Type is
//! generic, the first bytes are peeked off the stream for magic-byte
//! detection before being forwarded unchanged.
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::sync::Mutex as StdMutex;
use std::sync::atomic::{AtomicBool, Ordering};
use axum::body::Body;
use bytes::Bytes;
use futures::stream::{self, Stream, StreamExt, TryStreamExt};
use http_body_util::BodyStream;
// The `Digest` trait (re-exported by both `md5` and `sha2` from the
// `digest` crate) gives `Md5` and `Sha256` their `new` / `update` /
// `finalize` methods. Importing once via `sha2` covers both —
// otherwise every call site would need fully-qualified
// `<md5::Md5 as md5::Digest>::…` syntax.
use sha2::Digest as _;
use tokio::io::AsyncWriteExt;
use tokio_util::io::ReaderStream;
use crate::application::ports::chunked_upload_ports::ChecksumAlg;
use crate::application::ports::file_ports::StoredBlob;
use crate::common::mime_detect::{MAGIC_BYTES_LEN, is_generic_mime, refine_content_type};
use crate::infrastructure::services::dedup_service::DedupService;
use crate::interfaces::errors::AppError;
/// Content stored in the chunk store by one upload ingest.
///
/// The ingest holds ONE blob reference; pass [`IngestedBlob::stored`] to the
/// upload service (which takes ownership of the reference) or hand it back
/// via [`discard_ingested`] when the upload is rejected after the fact.
pub struct IngestedBlob {
/// BLAKE3 of the full content (the blob/manifest key).
pub hash: String,
/// Total bytes ingested.
pub size: u64,
/// Refined content type (claimed type or magic-byte detection).
pub content_type: String,
/// `false` when the exact content already existed (dedup hit).
pub is_new_blob: bool,
/// Bytes that did not need to be transferred to storage (dedup hit).
pub bytes_saved: u64,
}
impl IngestedBlob {
/// The blob reference to hand to the upload service.
pub fn stored(&self) -> StoredBlob {
StoredBlob {
hash: self.hash.clone(),
size: self.size,
is_new_blob: self.is_new_blob,
}
}
}
/// Hand back the blob reference taken by a successful ingest when the upload
/// is rejected after the fact (quota exceeded, checksum mismatch, …).
pub async fn discard_ingested(dedup: &DedupService, blob: &IngestedBlob) {
if let Err(e) = dedup.remove_reference(&blob.hash).await {
tracing::warn!(
"Failed to release blob reference of rejected upload {}: {e}",
&blob.hash[..blob.hash.len().min(12)]
);
}
}
/// Shared mutable tee for computing a client-requested checksum during the
/// ingest pass (REST chunked uploads) — no post-store re-read needed.
pub type ChecksumTee = Arc<StdMutex<Option<IncrementalHasher>>>;
/// Create a checksum tee for [`ingest_stream_to_cas`].
pub fn checksum_tee(alg: ChecksumAlg) -> ChecksumTee {
Arc::new(StdMutex::new(Some(IncrementalHasher::new(alg))))
}
/// Finalize a checksum tee into its lowercase hex digest.
pub fn finalize_checksum_tee(tee: &ChecksumTee) -> Option<String> {
tee.lock()
.ok()
.and_then(|mut h| h.take())
.map(IncrementalHasher::finalize_hex)
}
/// Out-of-band state observed by the stream adapters while the dedup engine
/// consumes the stream — lets the caller map an opaque engine error back to
/// the precise HTTP failure (413 vs 400).
struct IngestFlags {
too_large: AtomicBool,
source_error: StdMutex<Option<String>>,
}
/// Stream a request body (or any byte stream) into the CDC chunk store.
///
/// Single pass: size-cap enforcement, optional checksum tee, MIME sniffing
/// (first [`MAGIC_BYTES_LEN`] bytes, only when `claimed_type` is generic)
/// and the CDC chunk/hash/store pipeline all run while the bytes arrive.
/// Peak heap is bounded by the dedup engine (~9 MiB) regardless of size.
///
/// On error nothing stays referenced — the engine compensates internally.
pub async fn ingest_stream_to_cas<S, E>(
source: S,
dedup: &Arc<DedupService>,
filename: &str,
claimed_type: &str,
max_bytes: usize,
checksum: Option<ChecksumTee>,
) -> Result<IngestedBlob, AppError>
where
S: Stream<Item = Result<Bytes, E>> + Send,
E: std::fmt::Display,
{
let flags = Arc::new(IngestFlags {
too_large: AtomicBool::new(false),
source_error: StdMutex::new(None),
});
// ── Adapter: cap + checksum tee + error capture ──────────────
let adapter_flags = flags.clone();
let mut total: usize = 0;
let counted = source.map(move |item| match item {
Ok(bytes) => {
total += bytes.len();
if total > max_bytes {
adapter_flags.too_large.store(true, Ordering::Relaxed);
return Err(std::io::Error::other("upload exceeds size cap"));
}
if let Some(tee) = &checksum
&& let Ok(mut hasher) = tee.lock()
&& let Some(hasher) = hasher.as_mut()
{
hasher.update(&bytes);
}
Ok(bytes)
}
Err(e) => {
let message = e.to_string();
if let Ok(mut slot) = adapter_flags.source_error.lock() {
*slot = Some(message.clone());
}
Err(std::io::Error::other(message))
}
});
// `fuse` is load-bearing: when the source is shorter than the MIME peek
// (< MAGIC_BYTES_LEN), the peek loop drains it to None and the `chain`
// below polls it once more — non-fused sources (e.g. `stream::unfold`,
// as used for multipart fields) panic on a post-None poll.
let mut counted = Box::pin(counted.fuse());
// ── In-flight MIME sniff (only when the claimed type is generic) ──
let mut head: Vec<Result<Bytes, std::io::Error>> = Vec::new();
let content_type = if is_generic_mime(claimed_type) {
let mut head_len = 0usize;
while head_len < MAGIC_BYTES_LEN {
match counted.next().await {
Some(Ok(bytes)) => {
head_len += bytes.len();
head.push(Ok(bytes));
}
Some(Err(e)) => {
head.push(Err(e));
break;
}
None => break,
}
}
let mut magic = Vec::with_capacity(head_len.min(MAGIC_BYTES_LEN));
for item in head.iter().flatten() {
let take = (MAGIC_BYTES_LEN - magic.len()).min(item.len());
magic.extend_from_slice(&item[..take]);
if magic.len() >= MAGIC_BYTES_LEN {
break;
}
}
refine_content_type(&magic, filename, claimed_type)
} else {
claimed_type.to_string()
};
// ── Store: peeked head + remainder, one continuous stream ────
let full_stream = stream::iter(head).chain(counted);
let result = dedup
.store_from_stream(full_stream, Some(content_type.clone()))
.await;
match result {
Ok(stored) => {
let is_new_blob = !stored.was_deduplicated();
let bytes_saved = match &stored {
crate::application::ports::dedup_ports::DedupResultDto::ExistingBlob {
saved_bytes,
..
} => *saved_bytes,
_ => 0,
};
Ok(IngestedBlob {
hash: stored.hash().to_string(),
size: stored.size(),
content_type,
is_new_blob,
bytes_saved,
})
}
Err(e) => {
if flags.too_large.load(Ordering::Relaxed) {
return Err(AppError::payload_too_large(format!(
"Upload body exceeds the direct-PUT cap ({max_bytes} bytes). \
Use the chunked-upload protocol (REST: `/api/uploads/...`, \
NextCloud: `/remote.php/dav/uploads/...`) for files larger than this. \
Chunked uploads are resumable on transient failure."
)));
}
let source_error = flags.source_error.lock().ok().and_then(|s| s.clone());
if let Some(message) = source_error {
return Err(AppError::bad_request(format!(
"Failed to read request body: {message}"
)));
}
Err(AppError::from(e))
}
}
}
/// [`ingest_stream_to_cas`] for an HTTP request body.
pub async fn ingest_body_to_cas(
body: Body,
dedup: &Arc<DedupService>,
filename: &str,
claimed_type: &str,
max_bytes: usize,
) -> Result<IngestedBlob, AppError> {
let source = BodyStream::new(body).filter_map(|item| async move {
match item {
Ok(frame) => frame.into_data().ok().map(Ok),
Err(e) => Some(Err(e)),
}
});
ingest_stream_to_cas(source, dedup, filename, claimed_type, max_bytes, None).await
}
/// Adapt a multipart field into a byte stream for [`ingest_stream_to_cas`].
///
/// Terminates after the first error — multipart fields are not resumable.
pub fn multipart_field_stream(
field: axum::extract::multipart::Field<'_>,
) -> impl Stream<Item = Result<Bytes, axum::extract::multipart::MultipartError>> + Send + '_ {
stream::unfold((field, false), |(mut field, done)| async move {
if done {
return None;
}
match field.chunk().await {
Ok(Some(bytes)) => Some((Ok(bytes), (field, false))),
Ok(None) => None,
Err(e) => Some((Err(e), (field, true))),
}
})
}
/// Concatenate already-uploaded chunk part files into one byte stream, in
/// the given order — feeds chunked-upload assembly into the CDC store
/// without ever materializing an assembled file on disk.
pub fn stream_from_files(
paths: Vec<PathBuf>,
) -> impl Stream<Item = Result<Bytes, std::io::Error>> + Send {
stream::iter(paths.into_iter().map(Ok::<_, std::io::Error>))
.and_then(|path| async move {
tokio::fs::File::open(path)
.await
.map(|file| ReaderStream::with_capacity(file, 64 * 1024))
})
.try_flatten()
}
/// Result of a streamed write to a caller-supplied path.
pub struct StreamedToPath {
/// Total bytes written.
pub bytes_written: u64,
/// Lowercase hex digest, populated only when `checksum_alg=Some(_)`
/// was passed. The algorithm is identified by [`StreamedToPath::alg`].
pub checksum_hex: Option<String>,
/// Algorithm used to compute `checksum_hex`. Echoed back so the
/// caller can include it in audit logs or response headers.
pub alg: Option<ChecksumAlg>,
}
/// Stream an HTTP request body directly to a known destination file,
/// enforcing `max_bytes` as a hard size limit.
///
/// Used by the chunked-upload PUT handlers — each chunk has a
/// deterministic on-disk path (`NextcloudChunkedUploadService::safe_chunk_path`
/// for the NC surface, `ChunkedUploadService::prepare_chunk` for the
/// REST surface), so there's no spool/move dance. Peak heap is ~one
/// HTTP frame regardless of chunk size or `max_bytes`.
///
/// `checksum_alg` is the optional client-requested integrity check
/// (default `md5` per the legacy `Content-MD5` contract; `blake3`
/// available for forward-compat). When `Some`, the hash is computed
/// incrementally during streaming — no extra disk read for verification.
///
/// On size overflow the partial file is removed before the function
/// returns, so a client retry against the same chunk name starts from
/// a clean slate. On any other I/O error the partial file is also
/// removed and the error surfaces — callers can assume the path is
/// either fully written or absent.
pub async fn stream_body_to_path(
body: Body,
path: &Path,
max_bytes: usize,
checksum_alg: Option<ChecksumAlg>,
) -> Result<StreamedToPath, AppError> {
let mut file = tokio::fs::File::create(path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);
let mut hasher = checksum_alg.map(IncrementalHasher::new);
while let Some(frame_result) = stream.next().await {
let frame = match frame_result {
Ok(f) => f,
Err(e) => {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::bad_request(format!(
"Failed to read request body: {e}"
)));
}
};
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_bytes {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::payload_too_large(format!(
"Chunk exceeds maximum size of {max_bytes} bytes"
)));
}
if let Some(h) = hasher.as_mut() {
h.update(chunk);
}
if let Err(e) = file.write_all(chunk).await {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::internal_error(format!(
"Failed to write chunk: {e}"
)));
}
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush chunk file: {e}")))?;
drop(file);
Ok(StreamedToPath {
bytes_written: total_bytes as u64,
checksum_hex: hasher.map(IncrementalHasher::finalize_hex),
alg: checksum_alg,
})
}
/// Algorithm-agnostic incremental hasher used by [`stream_body_to_path`]
/// and the [`ChecksumTee`] of chunked-upload completion.
/// Per-frame `update` is sub-millisecond for all three algorithms at the
/// 64 KB frame sizes axum's body stream produces, so we don't need
/// `spawn_blocking` (which the old buffered path used because it hashed
/// the full multi-MB chunk in one shot).
pub enum IncrementalHasher {
Md5(md5::Md5),
Sha256(sha2::Sha256),
// Boxing — blake3::Hasher is ~1.7 KB on the stack while md5::Md5
// (~100 bytes) and sha2::Sha256 (~100 bytes) are tiny; boxing the
// outlier keeps the enum size proportional to the common case
// rather than the worst case.
Blake3(Box<blake3::Hasher>),
}
impl IncrementalHasher {
fn new(alg: ChecksumAlg) -> Self {
match alg {
ChecksumAlg::Md5 => Self::Md5(md5::Md5::new()),
ChecksumAlg::Sha256 => Self::Sha256(sha2::Sha256::new()),
ChecksumAlg::Blake3 => Self::Blake3(Box::new(blake3::Hasher::new())),
}
}
fn update(&mut self, bytes: &[u8]) {
match self {
Self::Md5(h) => h.update(bytes),
Self::Sha256(h) => h.update(bytes),
Self::Blake3(h) => {
h.update(bytes);
}
}
}
fn finalize_hex(self) -> String {
match self {
Self::Md5(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Sha256(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Blake3(h) => h.finalize().to_hex().to_string(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use bytes::Bytes;
#[tokio::test]
async fn stream_body_to_path_caps_oversized() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
// 5 MiB body, 4 MiB cap → must reject.
let body = Body::from(Bytes::from(vec![0u8; 5 * 1024 * 1024]));
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
assert!(
result.is_err(),
"expected PayloadTooLarge, got Ok(bytes_written={})",
result.ok().map(|r| r.bytes_written).unwrap_or(0)
);
// Partial file must be removed on rejection.
assert!(
!path.exists(),
"rejected chunk file should be removed, but {} still exists",
path.display()
);
}
#[tokio::test]
async fn stream_body_to_path_accepts_under_cap() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 1024 * 1024])); // 1 MiB
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
let outcome = result.expect("should succeed");
assert_eq!(outcome.bytes_written, 1024 * 1024);
assert!(outcome.checksum_hex.is_none(), "no alg requested → no hash");
assert!(path.exists());
}
#[tokio::test]
async fn stream_body_to_path_caps_at_exact_boundary() {
// Edge case: body exactly equal to cap should succeed; cap+1 must fail.
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 100]));
let outcome = stream_body_to_path(body, &path, 100, None)
.await
.expect("100 bytes at 100-byte cap should succeed");
assert_eq!(outcome.bytes_written, 100);
let path2 = temp_dir.path().join("chunk2");
let body = Body::from(Bytes::from(vec![1u8; 101]));
assert!(
stream_body_to_path(body, &path2, 100, None).await.is_err(),
"101 bytes at 100-byte cap must reject"
);
assert!(!path2.exists());
}
#[tokio::test]
async fn ingest_rejects_oversized_before_touching_storage() {
// 1 KiB body against a 100-byte cap: the adapter must abort the
// stream before any flush, so the stub dedup service (which cannot
// reach PG) is never asked to settle a batch.
let dedup = Arc::new(DedupService::new_stub());
let source = stream::iter(vec![Ok::<_, std::io::Error>(Bytes::from(vec![0u8; 1024]))]);
let result = ingest_stream_to_cas(
source,
&dedup,
"file.bin",
"application/octet-stream",
100,
None,
)
.await;
let err = result.err().expect("oversized body must be rejected");
assert_eq!(err.status_code, axum::http::StatusCode::PAYLOAD_TOO_LARGE);
}
#[tokio::test]
async fn ingest_surfaces_source_errors_as_bad_request() {
let dedup = Arc::new(DedupService::new_stub());
let source = stream::iter(vec![
Ok::<_, std::io::Error>(Bytes::from_static(b"partial")),
Err(std::io::Error::other("connection reset by peer")),
]);
let result =
ingest_stream_to_cas(source, &dedup, "file.bin", "text/plain", usize::MAX, None).await;
let err = result.err().expect("source error must surface");
assert_eq!(err.status_code, axum::http::StatusCode::BAD_REQUEST);
assert!(
err.message.contains("connection reset by peer"),
"original cause must be preserved: {}",
err.message
);
}
/// Regression: a source shorter than the MIME peek (< MAGIC_BYTES_LEN)
/// is drained to None during sniffing and then polled once more by the
/// `chain` that re-attaches the peeked head. Non-fused sources — like
/// the `stream::unfold` used for multipart fields — panic on that
/// post-None poll ("Unfold must not be polled after it returned
/// `Poll::Ready(None)`") unless the ingest fuses the stream first.
/// The stub dedup service can't reach PG, so an orderly `Err` (not a
/// panic) proves the stream layer survived.
#[tokio::test]
async fn ingest_short_stream_with_generic_mime_does_not_repoll_source() {
let dedup = Arc::new(DedupService::new_stub());
let source = stream::unfold(false, |done| async move {
if done {
None
} else {
Some((Ok::<_, std::io::Error>(Bytes::from_static(b"tiny")), true))
}
});
let result = ingest_stream_to_cas(
source,
&dedup,
"tiny.bin",
"application/octet-stream",
usize::MAX,
None,
)
.await;
assert!(
result.is_err(),
"stub DB must reject the store — but only AFTER the stream \
layer survived the post-peek poll"
);
}
#[tokio::test]
async fn stream_from_files_concatenates_in_order() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let a = temp_dir.path().join("a");
let b = temp_dir.path().join("b");
tokio::fs::write(&a, b"Hello, ").await.unwrap();
tokio::fs::write(&b, b"World!").await.unwrap();
let mut out = Vec::new();
let s = stream_from_files(vec![a, b]);
futures::pin_mut!(s);
while let Some(chunk) = s.next().await {
out.extend_from_slice(&chunk.expect("read"));
}
assert_eq!(out, b"Hello, World!");
}
#[tokio::test]
async fn checksum_tee_roundtrip() {
let tee = checksum_tee(ChecksumAlg::Md5);
if let Ok(mut h) = tee.lock()
&& let Some(h) = h.as_mut()
{
h.update(b"hello world");
}
let hex = finalize_checksum_tee(&tee).expect("digest");
assert_eq!(hex, "5eb63bbbe01eeed093cb22bb8f5acdc3");
assert!(
finalize_checksum_tee(&tee).is_none(),
"second finalize returns None"
);
}
}
-289
View File
@@ -1,289 +0,0 @@
//! Shared streaming upload spool: request body → temp file + incremental hash.
//!
//! Used by both the native WebDAV PUT handler and the NextCloud-compat PUT
//! handler so neither buffers the full request body in memory. Peak heap is
//! ~one HTTP frame regardless of file size; the body is written to a temp
//! file (off tmpfs when [`StorageConfig::upload_temp_dir`] is configured) and
//! BLAKE3-hashed on the fly so the dedup layer can short-circuit on a hit.
use std::path::{Path, PathBuf};
use axum::body::Body;
use http_body_util::BodyStream;
// The `Digest` trait (re-exported by both `md5` and `sha2` from the
// `digest` crate) gives `Md5` and `Sha256` their `new` / `update` /
// `finalize` methods. Importing once via `sha2` covers both —
// otherwise every call site would need fully-qualified
// `<md5::Md5 as md5::Digest>::…` syntax.
use sha2::Digest as _;
use tempfile::NamedTempFile;
use tokio::io::AsyncWriteExt;
use tokio_stream::StreamExt;
use crate::application::ports::chunked_upload_ports::ChecksumAlg;
use crate::common::temp::new_spool_temp_file;
use crate::interfaces::errors::AppError;
/// Outcome of spooling a request body to disk.
pub struct SpooledBody {
/// The temp file holding the body. Kept alive by the caller (dropping it
/// removes the file unless the dedup layer already consumed/moved it).
pub temp: NamedTempFile,
/// Hex-encoded BLAKE3 of the full body — matches `DedupService::hash_file`,
/// so passing it as `pre_computed_hash` enables the dedup fast path.
pub hash: String,
/// Total bytes written.
pub size: u64,
}
/// Stream an HTTP request body to a temp file, computing its BLAKE3 hash
/// incrementally and enforcing `max_upload` as a hard size limit.
///
/// Peak heap is ~one frame — the body is never fully buffered in RAM.
///
/// `temp_dir` is taken by value (not `&Path`) so the returned future captures
/// no borrowed lifetime — required for the handler future to stay `Send`.
pub async fn spool_body_to_temp(
body: Body,
max_upload: usize,
temp_dir: Option<PathBuf>,
) -> Result<SpooledBody, AppError> {
let temp = new_spool_temp_file(temp_dir.as_deref())
.map_err(|e| AppError::internal_error(format!("Failed to create temp file: {e}")))?;
let temp_path = temp.path().to_path_buf();
let mut file = tokio::fs::File::create(&temp_path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open temp file: {e}")))?;
let mut hasher = blake3::Hasher::new();
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);
while let Some(frame_result) = stream.next().await {
let frame = frame_result
.map_err(|e| AppError::bad_request(format!("Failed to read request body: {e}")))?;
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_upload {
// Abort early — stop reading, delete temp file.
drop(file);
let _ = tokio::fs::remove_file(&temp_path).await;
return Err(AppError::payload_too_large(format!(
"Upload body exceeds the direct-PUT cap ({max_upload} bytes). \
Use the chunked-upload protocol (REST: `/api/uploads/...`, \
NextCloud: `/remote.php/dav/uploads/...`) for files larger than this. \
Chunked uploads are resumable on transient failure."
)));
}
hasher.update(chunk);
file.write_all(chunk).await.map_err(|e| {
AppError::internal_error(format!("Failed to write to temp file: {e}"))
})?;
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush temp file: {e}")))?;
drop(file);
let hash = hasher.finalize().to_hex().to_string();
Ok(SpooledBody {
temp,
hash,
size: total_bytes as u64,
})
}
/// Result of a streamed write to a caller-supplied path.
pub struct StreamedToPath {
/// Total bytes written.
pub bytes_written: u64,
/// Lowercase hex digest, populated only when `checksum_alg=Some(_)`
/// was passed. The algorithm is identified by [`StreamedToPath::alg`].
pub checksum_hex: Option<String>,
/// Algorithm used to compute `checksum_hex`. Echoed back so the
/// caller can include it in audit logs or response headers.
pub alg: Option<ChecksumAlg>,
}
/// Stream an HTTP request body directly to a known destination file,
/// enforcing `max_bytes` as a hard size limit.
///
/// Used by the chunked-upload PUT handlers — each chunk has a
/// deterministic on-disk path (`NextcloudChunkedUploadService::safe_chunk_path`
/// for the NC surface, `ChunkedUploadService::prepare_chunk` for the
/// REST surface), so there's no spool/move dance. Peak heap is ~one
/// HTTP frame regardless of chunk size or `max_bytes`.
///
/// `checksum_alg` is the optional client-requested integrity check
/// (default `md5` per the legacy `Content-MD5` contract; `blake3`
/// available for forward-compat). When `Some`, the hash is computed
/// incrementally during streaming — no extra disk read for verification.
///
/// On size overflow the partial file is removed before the function
/// returns, so a client retry against the same chunk name starts from
/// a clean slate. On any other I/O error the partial file is also
/// removed and the error surfaces — callers can assume the path is
/// either fully written or absent.
pub async fn stream_body_to_path(
body: Body,
path: &Path,
max_bytes: usize,
checksum_alg: Option<ChecksumAlg>,
) -> Result<StreamedToPath, AppError> {
let mut file = tokio::fs::File::create(path)
.await
.map_err(|e| AppError::internal_error(format!("Failed to open chunk file: {e}")))?;
let mut total_bytes: usize = 0;
let mut stream = BodyStream::new(body);
let mut hasher = checksum_alg.map(IncrementalHasher::new);
while let Some(frame_result) = stream.next().await {
let frame = match frame_result {
Ok(f) => f,
Err(e) => {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::bad_request(format!(
"Failed to read request body: {e}"
)));
}
};
if let Some(chunk) = frame.data_ref() {
total_bytes += chunk.len();
if total_bytes > max_bytes {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::payload_too_large(format!(
"Chunk exceeds maximum size of {max_bytes} bytes"
)));
}
if let Some(h) = hasher.as_mut() {
h.update(chunk);
}
if let Err(e) = file.write_all(chunk).await {
drop(file);
let _ = tokio::fs::remove_file(path).await;
return Err(AppError::internal_error(format!(
"Failed to write chunk: {e}"
)));
}
}
}
file.flush()
.await
.map_err(|e| AppError::internal_error(format!("Failed to flush chunk file: {e}")))?;
drop(file);
Ok(StreamedToPath {
bytes_written: total_bytes as u64,
checksum_hex: hasher.map(IncrementalHasher::finalize_hex),
alg: checksum_alg,
})
}
/// Algorithm-agnostic incremental hasher used by [`stream_body_to_path`].
/// Per-frame `update` is sub-millisecond for all three algorithms at the
/// 64 KB frame sizes axum's body stream produces, so we don't need
/// `spawn_blocking` (which the old buffered path used because it hashed
/// the full multi-MB chunk in one shot).
enum IncrementalHasher {
Md5(md5::Md5),
Sha256(sha2::Sha256),
// Boxing — blake3::Hasher is ~1.7 KB on the stack while md5::Md5
// (~100 bytes) and sha2::Sha256 (~100 bytes) are tiny; boxing the
// outlier keeps the enum size proportional to the common case
// rather than the worst case.
Blake3(Box<blake3::Hasher>),
}
impl IncrementalHasher {
fn new(alg: ChecksumAlg) -> Self {
match alg {
ChecksumAlg::Md5 => Self::Md5(md5::Md5::new()),
ChecksumAlg::Sha256 => Self::Sha256(sha2::Sha256::new()),
ChecksumAlg::Blake3 => Self::Blake3(Box::new(blake3::Hasher::new())),
}
}
fn update(&mut self, bytes: &[u8]) {
match self {
Self::Md5(h) => h.update(bytes),
Self::Sha256(h) => h.update(bytes),
Self::Blake3(h) => {
h.update(bytes);
}
}
}
fn finalize_hex(self) -> String {
match self {
Self::Md5(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Sha256(h) => h.finalize().iter().map(|b| format!("{b:02x}")).collect(),
Self::Blake3(h) => h.finalize().to_hex().to_string(),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use bytes::Bytes;
#[tokio::test]
async fn stream_body_to_path_caps_oversized() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
// 5 MiB body, 4 MiB cap → must reject.
let body = Body::from(Bytes::from(vec![0u8; 5 * 1024 * 1024]));
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
assert!(
result.is_err(),
"expected PayloadTooLarge, got Ok(bytes_written={})",
result.ok().map(|r| r.bytes_written).unwrap_or(0)
);
// Partial file must be removed on rejection.
assert!(
!path.exists(),
"rejected chunk file should be removed, but {} still exists",
path.display()
);
}
#[tokio::test]
async fn stream_body_to_path_accepts_under_cap() {
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 1024 * 1024])); // 1 MiB
let result = stream_body_to_path(body, &path, 4 * 1024 * 1024, None).await;
let outcome = result.expect("should succeed");
assert_eq!(outcome.bytes_written, 1024 * 1024);
assert!(outcome.checksum_hex.is_none(), "no alg requested → no hash");
assert!(path.exists());
}
#[tokio::test]
async fn stream_body_to_path_caps_at_exact_boundary() {
// Edge case: body exactly equal to cap should succeed; cap+1 must fail.
let temp_dir = tempfile::tempdir().expect("tempdir");
let path = temp_dir.path().join("chunk");
let body = Body::from(Bytes::from(vec![1u8; 100]));
let outcome = stream_body_to_path(body, &path, 100, None)
.await
.expect("100 bytes at 100-byte cap should succeed");
assert_eq!(outcome.bytes_written, 100);
let path2 = temp_dir.path().join("chunk2");
let body = Body::from(Bytes::from(vec![1u8; 101]));
assert!(
stream_body_to_path(body, &path2, 100, None).await.is_err(),
"101 bytes at 100-byte cap must reject"
);
assert!(!path2.exists());
}
}