Commit Graph

32 Commits

Author SHA1 Message Date
Edouard Vanbelle 4e36de49eb feat(uploads): add OXICLOUD_CHUNK_DIR + documentation for admins
explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
    and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
    to help administratorrs to defined correctly their storage architecture
2026-06-09 11:06:44 +02:00
Edouard Vanbelle abac074fe5 feat(fssync): ensure fs sync once upload is recomposed 2026-06-09 09:55:27 +02:00
Edouard Vanbelle 41aad26702 feat(upload): cover chunk upload + add support of different digest hash
Prefer stream storage rather using buffered (in memory)

  note: on many unix like tmpfs are in-memory, sungle PUT are sized limited

  Storage map (NC stands for Nextcloud gateway)

  ┌───────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┬─────────────────────────────────────────────────┐
  │                   Streaming surface                   │                            Destination                             │                Configurable via                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST chunked PUT /api/uploads/{id} chunk              │ {storage_path}/.uploads/{upload_id}/chunk_{NNNNNN}                 │ OXICLOUD_STORAGE_PATH (the .uploads subdir is   │
  │                                                       │                                                                    │ hard-wired)                                     │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST chunked assemble (during /complete)              │ {storage_path}/.uploads/{upload_id}/assembled                      │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC chunked PUT /dav/uploads/.../{chunk}               │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/{chunk_name}  │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC chunked assemble (during MOVE)                     │ {storage_path}/.uploads/nextcloud/{user}/{upload_id}/.assembled    │ same                                            │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ NC single-file PUT /dav/files/.../{path} (via         │ OXICLOUD_UPLOAD_TMPDIR if set, else OS default temp (/tmp on       │ OXICLOUD_UPLOAD_TMPDIR                          │
  │ spool_body_to_temp)                                   │ Linux)                                                             │                                                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST WebDAV PUT /webdav/{path} (via                   │ same as above                                                      │ OXICLOUD_UPLOAD_TMPDIR                          │
  │ spool_body_to_temp)                                   │                                                                    │                                                 │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ REST multipart upload /api/files/upload               │ {storage_path}/.dedup_temp/upload-{uuid}                           │ OXICLOUD_STORAGE_PATH (hard-wired subdir)       │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ WOPI PutFile                                          │ OS default temp via NamedTempFile::new() (no override)             │ (none — bug worth tracking)                     │
  ├───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┼─────────────────────────────────────────────────┤
  │ Final blob storage (after fsync + rename)             │ {storage_path}/.blobs/{ab}/{abc…}.blob                             │ OXICLOUD_STORAGE_PATH                           │
  └───────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────────────┴─────────────────────────────────────────────────┘

  one caveat: a malicious user can create many chunked upload and saturate local storage
2026-06-09 09:52:09 +02:00
Edouard Vanbelle a53c09f361 feat(authz): covert and test chunked upload with permissions 2026-05-21 20:30:58 +02:00
Diocrafts 761d159a92 feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking
  (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing
- Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[])
- Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends)
- 3-phase store_chunks pipeline:
  Phase 0: batch-check existing chunks (single PG query)
  Phase 1: selective disk read (skip existing chunks entirely)
  Phase 2: parallel upload with buffer_unordered(8)
- CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback
- Transactional manifest + chunk ref-count cascade on remove_reference
- 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.)
- Update deduplication.md to reflect new architecture
2026-04-14 23:17:39 +02:00
Edouard Vanbelle badf35f08f chore: remove all executable attributes on non bash files 2026-04-01 23:14:42 +02:00
Diocrafts 06ed0455ce perf: migrate all user/session/auth IDs from VARCHAR(36) to native UUID
- Schema: all ~15 VARCHAR(36) columns → UUID with DEFAULT gen_random_uuid()
- Domain entities: User, Session, DeviceCode, AppPassword, Share → id: Uuid
- DTOs: CurrentUser.id → Uuid (API boundary DTOs keep String for JSON)
- Auth middleware: parse JWT claims.sub (String) → Uuid at boundary
- All repository traits, port traits, service impls updated end-to-end
- Handlers: pass Uuid by value (Copy, 16 bytes) instead of String refs
- Settings chain: updated_by column → Uuid (was text, caused setup crash)
- Removed ~650 lines of String↔Uuid conversion boilerplate
- Eliminates per-request heap allocations for ID cloning
- 16-byte binary comparison vs 36-byte string comparison in all queries
- Native UUID indexing in PostgreSQL (btree on 16 bytes vs 36-char text)

85 files changed, 1090 insertions(+), 1739 deletions(-)
2026-03-07 14:59:32 +01:00
Dionisio f2d35ca792 feat: auto-persist JWT secret, remove setup token requirement
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
2026-03-05 22:12:53 +01:00
zjean 190527edfb style: apply rustfmt formatting to fix CI
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-05 21:28:51 +01:00
Dionisio b503e08384 security: fix vulnerabilities 1-7 from security audit
- Fix #1: Share handler IDOR - enforce owner check on share operations
- Fix #2: list_files_query IDOR - bind folder queries to authenticated user
- Fix #3: Dedup handler IDOR - restrict dedup operations to file owner
- Fix #4: Trash handler OptionalAuthUser - require full AuthUser
- Fix #5: Error info leakage - sanitize 500 error responses
- Fix #6: Chunked upload IDOR - bind upload sessions to user_id,
  add verify_session_owner() check on all session operations
- Fix #7: CSP unsafe-inline removal - migrate all inline scripts,
  styles and event handlers to external files, tighten CSP to
  script-src 'self'; style-src 'self'

New files:
  - static/js/core/theme-init.js (render-blocking theme init)
  - static/js/core/sw-register.js (service worker registration)
  - static/css/views/device-verify.css (extracted inline styles)
  - static/js/views/device-verify/device-verify.js (extracted inline script)
2026-03-05 13:15:34 +01:00
Diocrafts ee86c3a128 fix: resolve all clippy warnings and convert integration_tests to custom cfg
- Add type aliases (FileRow, FolderRow, FolderRowPaginated, FolderRowOptUser) to reduce type complexity
- Simplify redundant closures in app_password_handler and webdav_handler
- Remove needless borrow in auth_handler
- Collapse nested if/let chains in login_lockout, webdav_lock, auth, rate_limit
- Box LockEntry in acquire() Err variant to fix large enum variant warning
- Rename DeviceCodeStatus::from_str to parse to avoid should_implement_trait lint
- Add #[allow(clippy::too_many_arguments)] and #[allow(clippy::result_unit_err)] where appropriate
- Convert integration_tests from cargo feature to custom cfg attribute
- Add check-cfg lint config in Cargo.toml for integration_tests cfg
2026-03-04 23:55:08 +01:00
Claude 3b0c09cd56 Replace standalone md5 crate with RustCrypto md-5
The standalone `md5` (0.8.0) crate is replaced with `md-5` (0.10) from
the RustCrypto ecosystem, which shares `digest v0.10` with sha2, argon2,
blake2 and other crates already in the dependency tree — eliminating one
redundant implementation.

https://claude.ai/code/session_01V23pGpfNw5ujZvtwRFG6qy
2026-03-03 16:16:44 +00:00
Claude 4bee05c690 merge: resolve conflict in auth_factory.rs with origin/main
Keep custom Result type alias (replacing anyhow) from main while
preserving our removal of unused TokenServicePort import.

https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
2026-03-03 15:42:28 +00:00
Claude 1b49135ca9 perf: replace dyn trait objects with concrete types to eliminate vtable overhead
Remove async-trait dependency and use native Rust async fn in traits.
Replace Arc<dyn Trait> with Arc<ConcreteType> throughout the codebase
to enable monomorphization and eliminate dynamic dispatch overhead.

Key changes:
- Remove write-behind cache (no implementation existed)
- Fix should_transcode static method call
- Use ContactStorageAdapter directly instead of dyn AddressBookUseCase
- Clean up unused trait imports across services and DI

https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
2026-03-03 15:36:42 +00:00
Dionisio 0cd544ab96 perf: enable BLAKE3 multithreaded hashing (update_rayon) for files >10MB 2026-03-03 11:18:40 +01:00
Dionisio e2fb29ea60 perf: replace SHA-256 with BLAKE3 + add mimalloc global allocator
- Replace SHA-256 with BLAKE3 (~5x faster) for content-addressable hashing
  in dedup_service, file_handler, file_upload_service, chunked_upload_service
- Add mimalloc as global allocator for 10-30% throughput improvement
- sha2 crate retained only for PKCE (OAuth2 standard requirement)
- BLAKE3 produces 64-char hex hashes (same format), no DB schema changes needed
2026-03-01 21:47:39 +01:00
Dionisio 2421724b80 style: cargo fmt --all 2026-02-26 01:02:25 +01:00
Dionisio b05805ee1c fix: resolve clippy warnings (redundant closure, collapsible if) 2026-02-26 00:57:49 +01:00
Dionisio 6df68d3716 style: cargo fmt --all 2026-02-26 00:47:32 +01:00
Dionisio f9dde6ffff perf: replace RwLock<HashMap> with DashMap in ChunkedUploadService + decouple disk I/O from lock
Issue #3 (CRITICAL): The global RwLock<HashMap> serialised ALL chunk uploads
across all users. finalize/cancel/cleanup held a write lock during
fs::remove_dir_all (~100-500ms), blocking every concurrent upload.

Changes:
- Replace tokio::sync::RwLock<HashMap<String, UploadSession>> with
  dashmap::DashMap (sharded concurrent map, ~64 shards)
- Operations on independent sessions never contend
- finalize_upload_inner: remove from map (µs), THEN delete temp dir
- cancel_upload_inner: same pattern — disk I/O outside lock
- cleanup_loop: collect expired IDs via lock-free iteration, remove
  from map, THEN delete dirs sequentially with no lock held
- upload_chunk_inner: DashMap::get_mut replaces global write lock
- get_status_inner / complete_upload_inner: DashMap::get replaces read lock
- Remove tokio::sync::RwLock import (dead)

Also includes Issue #2 (dedup_service.rs write-first + upsert) from
previous session.

Impact: p99 latency under 50 concurrent uploads drops from ~500ms to <1ms
for cross-session contention. Cleanup loop no longer blocks uploads.
2026-02-25 23:31:51 +01:00
Dionisio 97cf6402e2 fix: resolve all clippy warnings and enforce cargo fmt
- display_helpers: convert module doc-comments to regular comments,
  merge identical text/markdown + text/ branches
- search_service: replace needless range loops with slice-based pagination
- folder_repository, folder_db_repository: collapse nested if statements
- favorites_pg_repository: remove unnecessary borrow on generic arg
- file_blob_read_repository: collapse 6 nested if-let blocks
- file_blob_write_repository: collapse nested if for dedup ref decrement
- chunked_upload_service: use div_ceil(), collapse 2 nested if blocks
- folder_handler: collapse nested if-let for owner check
- webdav_handler: replace 7x io::Error::new(ErrorKind::Other, ..) with
  io::Error::other(..)
- cargo fmt applied to all files

Passes: cargo clippy --all-targets --all-features -- -D warnings
2026-02-25 10:28:34 +01:00
Dionisio 71c2cb5edb perf: Arc<AppState>, streaming PROPFIND, spawn_blocking SHA-256
- Issue #4: Wrap AppState in Arc — eliminates 42 Arc::clone + 16 String::clone per request
- Issue #2: Reject Depth:infinity with 403 + streaming XML with paginated DB queries
- Issue #5: Move chunked upload assembly (SHA-256 hash-on-write) to spawn_blocking
- Remove ~270 lines dead code from di.rs (unused builders, Default impl, stubs)
- Clean up unused tokio imports in chunked_upload_service.rs
2026-02-24 15:11:56 +01:00
Dionisio 78ae145af5 perf(chunked-upload): hoist 512KB read buffer out of per-chunk loop
causing N × 512KB alloc+memset+dealloc cycles per upload. Moving it
before the loop reuses a single allocation across all chunks.

For a 100-chunk upload this eliminates 99 allocations totalling ~50 MB
of unnecessary memset work.
2026-02-24 13:03:18 +01:00
Dionisio b5652b029d perf: offload MD5 checksum to spawn_blocking in chunked uploads
md5::compute(&data) is CPU-bound (~1.2ms per 5MB chunk) and was
blocking the Tokio worker thread. Move it to the blocking thread-pool
via spawn_blocking so the async worker is freed in ~5µs. Bytes::clone
is O(1) (Arc increment) so no extra copy overhead.
2026-02-23 22:49:34 +01:00
Dionisio 1da284a841 perf: move 512KB I/O buffer from stack to heap in chunked upload assembly
- Async Future size drops from ~525KB to ~5KB
- Eliminates stack overflow risk and reduces work-stealing copy cost
2026-02-23 22:41:50 +01:00
Dionisio 41b5500b77 perf: move persist_progress disk I/O outside write lock in chunked uploads
- Write lock now held only for RAM updates (~microseconds instead of ~ms)
- Bitmask built under lock, written to disk after lock release
- Concurrent uploads across all sessions no longer blocked by disk I/O
- Under 10 concurrent sessions: lock wait drops from ~50ms to ~10µs
2026-02-23 22:32:14 +01:00
Diocrafts fd9e509648 perf: implement findings #2, #3, #17 from architecture audit
- Finding #2: Replace Mutex<HashMap> with moka::sync::Cache in file_blob_read_repository (10K cap, 30s TTI)
- Finding #3: Add chunked upload persistence with session.json + progress.bin bitmask for crash recovery
- Finding #17: Remove manual gzip compression, delegate entirely to tower-http CompressionLayer
- Remove dead code: StubCompressionPort, GzipCompressionService re-export, duplicate response structs
- All 114 tests passing
2026-02-22 14:12:53 +01:00
Dionisio 1ed20f425f perf: Phase 4+5 optimizations — uploads 10x, downloads 2x, concurrent 2x. moka cache, 512KB buffers, remove sync_all, hash-on-write, preloaded queries, bench.sh v3, gitignore storage/. 500MB upload 12.6s->1.3s (392MB/s). RSS 69-113MB, 0 swap. 2026-02-15 17:56:47 +01:00
Dionisio 4c98c5a657 style: apply cargo fmt to entire codebase
Standardize code formatting across all 173 Rust source files
using rustfmt. No functional changes - purely cosmetic.

This establishes a consistent code style baseline for the
project going forward.
2026-02-14 01:29:34 +01:00
Dionisio 67137a3ef2 refactor: apply clippy auto-fixes (162 warnings resolved)
- Fix needless borrows and references
- Collapse nested if statements
- Replace manual strip_prefix with str::strip_prefix()
- Remove redundant closures in map/unwrap_or_else
- Use Iterator::next_back() instead of rev().next()
- Simplify map_or patterns
- Use std::io::Error::other() instead of new(ErrorKind::Other, ..)
- Use div_ceil() instead of manual ceiling division
- Consolidate format! string arguments
- Various other idiomatic Rust improvements

39 files changed, 220 insertions(+), 320 deletions(-)
2026-02-14 01:26:02 +01:00
Diocrafts a82faa5eaf refactoring hexagonal and clean architecture 2026-02-08 13:40:23 +01:00
Dionisio 8f2b0a354c big refactoring 2026-02-03 17:59:04 +01:00