Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
The Nextcloud Android client compares ETags before and after upload to
verify its write landed. OxiCloud was returning the stable file UUID as
the ETag, which never changed on content updates, causing false
SYNC_CONFLICT errors on every upload.
Five fixes applied:
1. Thread blob_hash (SHA-256) through File entity, FileDto, all read/write
queries, and all WebDAV/PROPFIND responses as the ETag — changes on
every content update, no DB migration needed.
2. Honor X-OC-Mtime header: parse the client-supplied mtime and use it
for updated_at via COALESCE(to_timestamp($n), NOW()).
3. Disable phantom checksum capability (preferredUploadType/supportedTypes)
that the server never actually implemented, stopping retry loops.
4. Add nc:creation_time and nc:upload_time to PROPFIND responses.
5. Return oc-etag header in chunked upload MOVE (assemble) responses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- JWT secret auto-generates and persists to <STORAGE_PATH>/.jwt_secret
- Remove setup token: first admin setup is open until system initialized
- Fix schema.sql: move CREATE EXTENSION pg_trgm/ltree to top
- Update login UI and auth.js to remove setup token fields
Update share_service test impl to match upstream trait changes
(requester_id params, verify_shared_link_password returns ShareDto).
Fix map_or, collapsible_if, dead_code, too_many_arguments warnings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nextcloud app uploads sent application/octet-stream as Content-Type,
causing images to not be recognized. Now both WebDAV PUT and chunked
upload paths call refine_content_type() which detects via magic bytes,
then extension, then falls back to the client header. Also fixes
update_file() which previously hardcoded application/octet-stream.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add type aliases (FileRow, FolderRow, FolderRowPaginated, FolderRowOptUser) to reduce type complexity
- Simplify redundant closures in app_password_handler and webdav_handler
- Remove needless borrow in auth_handler
- Collapse nested if/let chains in login_lockout, webdav_lock, auth, rate_limit
- Box LockEntry in acquire() Err variant to fix large enum variant warning
- Rename DeviceCodeStatus::from_str to parse to avoid should_implement_trait lint
- Add #[allow(clippy::too_many_arguments)] and #[allow(clippy::result_unit_err)] where appropriate
- Convert integration_tests from cargo feature to custom cfg attribute
- Add check-cfg lint config in Cargo.toml for integration_tests cfg
Replace String with Arc<str> for fields that contain repeated static
values (mime_type, icon_class, icon_special_class, category) in
FileDto, FolderDto, and OptimizedFileContent.
These fields are computed from ~40 static lookup tables and cloned
on every request. With Arc<str>, clone becomes O(1) atomic increment
instead of O(n) heap allocation — saving thousands of allocations/s
under load.
Fields kept as String: id, name, path, folder_id, owner_id
(unique per item, rarely cloned).
Zero API impact — serde serializes Arc<str> identically to String.
https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
Remove async-trait dependency and use native Rust async fn in traits.
Replace Arc<dyn Trait> with Arc<ConcreteType> throughout the codebase
to enable monomorphization and eliminate dynamic dispatch overhead.
Key changes:
- Remove write-behind cache (no implementation existed)
- Fix should_transcode static method call
- Use ContactStorageAdapter directly instead of dyn AddressBookUseCase
- Clean up unused trait imports across services and DI
https://claude.ai/code/session_01EbAFEfyJNLRmJHmmYDX3Tt
Replace list_files_in_subtree (fetch_all → Vec) with stream_files_in_subtree
that returns a Pin<Box<dyn Stream<Item = Result<File/FileDto>>>> backed by a
PostgreSQL cursor via sqlx::fetch().
Changes:
- FileReadPort::stream_files_in_subtree() returns streaming cursor (no default)
- FileRetrievalUseCase::stream_files_in_subtree() maps File→FileDto on the fly
- FileBlobReadRepository: async_stream::try_stream! + sqlx::fetch() cursor
- batch_operations: consume stream into HashMap incrementally
- zip_service: consume stream into HashMap incrementally
- All stubs/mocks updated (return empty stream)
Eliminates:
- Double allocation: Vec<(9-tuple)> + Vec<File> materialized simultaneously
- Unbounded RAM proportional to subtree size (was ~500 bytes × N files)
- Latency: callers blocked until last row fetched from PG
RAM is now O(folders) for the HashMap, not O(files).
Issue #4 (HIGH): save_file(Vec<u8>) and update_file_content(Vec<u8>)
accepted up to 10 MB of contiguous memory per request. While the main
upload paths already used streaming, the WebDAV compat methods
(create_file, update_file) and the empty-file handler still used the
buffered path, creating a .to_vec() copy.
Changes:
- FileWritePort trait: remove save_file(Vec<u8>) and
update_file_content(Vec<u8>) — only streaming variants remain
- FileUploadUseCase trait: remove upload_file(Vec<u8>)
- file_upload_service.rs: create_file() and update_file() now spool
&[u8] to NamedTempFile + Sha256::digest, then delegate to streaming
path (save_file_from_temp / update_file_streaming)
- file_handler.rs: empty file uploads use upload_file_streaming with
- FileBlobWriteRepository: remove save_file and update_file_content impls
- StubFileWritePort, StubFileUploadUseCase, MockFileRepository: remove
corresponding dead method impls
Impact: impossible to accidentally use a buffered upload path. All
content goes through streaming with ~256 KB peak RAM. -166 LOC.
Replace BFS traversal that issued 2 SQL queries per folder (list_files +
list_folders) with 2 total queries using PostgreSQL ltree <@ operator:
1. list_subtree_folders: single GiST-indexed scan for all folders
2. list_files_in_subtree: single GiST-indexed join for all files
Files are grouped by folder_id in a HashMap, then iterated in directory
order (folders pre-sorted by path from SQL).
Changes across 4 architecture layers:
- Domain: FolderRepository::list_subtree_folders (default impl)
- Application ports: FolderUseCase, FileRetrievalUseCase, FileReadPort
- Application services: FolderService, FileRetrievalService passthroughs
- Infrastructure: PG implementations + ZipService rewrite
Query count: O(N) → O(1). Latency for 100-folder tree: ~200 round-trips → 3.
Standardize code formatting across all 173 Rust source files
using rustfmt. No functional changes - purely cosmetic.
This establishes a consistent code style baseline for the
project going forward.
Claude