07 - Chunked Uploads

OxiCloud implements a TUS-like chunked upload protocol for large files (≥10 MB). Files are split into chunks (default 5 MB) that can be uploaded in parallel (up to 6 concurrent), with progress tracking, optional MD5 checksums, and automatic session expiration.

Architecture

Layer	Component	File
Application Port	ChunkedUploadPort trait + DTOs	`src/application/ports/chunked_upload_ports.rs`
Infrastructure	ChunkedUploadService	`src/infrastructure/services/chunked_upload_service.rs`
Interfaces	ChunkedUploadHandler	`src/interfaces/api/handlers/chunked_upload_handler.rs`

Constants

Constant	Value	Description
`CHUNKED_UPLOAD_THRESHOLD`	10 MB	Files above this should use chunked upload
`DEFAULT_CHUNK_SIZE`	5 MB	Default chunk size (minimum 1 MB)
`MAX_PARALLEL_CHUNKS`	6	Maximum concurrent chunk uploads
`SESSION_EXPIRATION`	24 hours	Sessions expire after this duration

REST API

All routes under /api/uploads, authentication required.

Method	Path	Handler	Description
`POST`	`/api/uploads`	`create_upload`	Create upload session
`PATCH`	`/api/uploads/{upload_id}`	`upload_chunk`	Upload a single chunk
`HEAD`	`/api/uploads/{upload_id}`	`get_upload_status`	Query upload progress
`POST`	`/api/uploads/{upload_id}/complete`	`complete_upload`	Assemble chunks → create file
`DELETE`	`/api/uploads/{upload_id}`	`cancel_upload`	Cancel and cleanup

Protocol Flow

1. POST /api/uploads
   Body: { "filename": "video.mp4", "total_size": 104857600, "content_type": "video/mp4" }
   Response: { "upload_id": "abc-123", "chunk_size": 5242880, "total_chunks": 20, "expires_at": 1707868800 }

2. PATCH /api/uploads/abc-123?chunk_index=0   ──┐
   PATCH /api/uploads/abc-123?chunk_index=1   ──┼── Up to 6 in parallel
   PATCH /api/uploads/abc-123?chunk_index=2   ──┘
   Body: raw chunk bytes
   Response: { "chunk_index": 0, "bytes_received": 52428800, "progress": 50.0, "is_complete": false }

3. HEAD /api/uploads/abc-123
   Response headers: Upload-Offset, Upload-Length, Upload-Progress, Upload-Chunks-Total, Upload-Chunks-Complete

4. POST /api/uploads/abc-123/complete
   Response: { "file_id": "def-456", "filename": "video.mp4", "size": 104857600, "path": "/videos" }
   Status: 201 Created

Request/Response DTOs

Create Upload Request

pub struct CreateUploadRequest {
    pub filename: String,
    pub folder_id: Option<String>,
    pub content_type: Option<String>,  // default: "application/octet-stream"
    pub total_size: u64,
    pub chunk_size: Option<usize>,     // default: 5 MB, minimum: 1 MB
}

Chunk Upload Query Parameters

pub struct ChunkUploadParams {
    pub chunk_index: usize,
    pub checksum: Option<String>,      // MD5 hash (also via Content-MD5 header)
}

Upload Status Response

pub struct UploadStatusResponseDto {
    pub upload_id: String,
    pub filename: String,
    pub total_size: u64,
    pub bytes_received: u64,
    pub progress: f64,                 // 0.0 - 100.0
    pub total_chunks: usize,
    pub completed_chunks: usize,
    pub pending_chunks: Vec<usize>,
    pub is_complete: bool,
}

Complete Upload Response

pub struct CompleteUploadResponse {
    pub file_id: String,
    pub filename: String,
    pub size: u64,
    pub path: String,
}

Custom Response Headers

On upload_chunk (PATCH)

Upload-Offset: total bytes received so far
Upload-Progress: percentage complete (0-100)
Upload-Complete: true (only when all chunks are uploaded)

On get_upload_status (HEAD)

Upload-Offset: bytes received
Upload-Length: total expected size
Upload-Progress: percentage
Upload-Chunks-Total: total chunk count
Upload-Chunks-Complete: completed chunk count

Internal Storage

<temp_dir>/
  <upload_id>/
    chunk_000000          ← individual chunk files
    chunk_000001
    chunk_000002
    ...
    assembled             ← final assembled file (after complete)

Completion Flow

complete_upload() assembles all chunks in order into a single assembled file
Reads the assembled file content
Delegates to FileUploadService::upload_file() to create the permanent file record
Calls finalize_upload() to clean up the session and temp directory
Returns 201 Created with file metadata

Session Management

In-memory sessions: HashMap<String, UploadSession> protected by RwLock
Chunk status tracking: each chunk has a status: Pending → Uploading → Complete (or Failed)
Expiration: sessions expire after 24 hours of inactivity
Cleanup: background task runs hourly to remove expired sessions and orphaned temp directories

Client Usage Example

# 1. Create upload session
RESPONSE=$(curl -s -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"filename":"large-file.zip","total_size":52428800}' \
  "https://oxicloud.example.com/api/uploads")

UPLOAD_ID=$(echo $RESPONSE | jq -r '.upload_id')
CHUNK_SIZE=$(echo $RESPONSE | jq -r '.chunk_size')
TOTAL_CHUNKS=$(echo $RESPONSE | jq -r '.total_chunks')

# 2. Upload chunks in parallel
for i in $(seq 0 $((TOTAL_CHUNKS - 1))); do
  dd if=large-file.zip bs=$CHUNK_SIZE skip=$i count=1 2>/dev/null | \
  curl -s -X PATCH -H "Authorization: Bearer $TOKEN" \
    --data-binary @- \
    "https://oxicloud.example.com/api/uploads/$UPLOAD_ID?chunk_index=$i" &
done
wait

# 3. Complete upload
curl -X POST -H "Authorization: Bearer $TOKEN" \
  "https://oxicloud.example.com/api/uploads/$UPLOAD_ID/complete"

# 4. Check progress (optional)
curl -I -H "Authorization: Bearer $TOKEN" \
  "https://oxicloud.example.com/api/uploads/$UPLOAD_ID"

5.8 KiB Raw Blame History