Request-path thumbnail generation (REST get_thumbnail and NC preview)
read the full source blob into RAM and decoded it with no concurrency
bound — a first-view gallery of K images stacked K full-size buffers
and K parallel decodes on the blocking pool. The background hook had
the inverse ordering problem: it read the blob eagerly and only then
queued on the decode semaphore, so N concurrent uploads held N
originals in memory while waiting.
All three paths now acquire the decode semaphore first and read the
blob under the permit, capping peak RAM at permits x image size:
- new ThumbnailService::get_thumbnail_from_blob defers the blob read
into the moka init closure (cache and disk hits never touch the
blob); both handlers use it and no longer pre-read.
- get_thumbnail_from_bytes (direct-bytes variant) now also takes a
permit before decoding; shared generate_and_persist core keeps the
two entrypoints duplicate-free.
- generate_all_sizes_background_from_blob (renamed from _from_bytes)
checks blob existence and disk-dedup state, then acquires the
permit, then reads; the redundant pre-read spawn wrapper in
ThumbnailRefreshHook is gone.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Two DAV-surface fixes that replicate patterns the codebase already had:
NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.
WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Closes#323.
LoginLockoutService cached failed-attempt counters keyed only on
the username, so any caller that could reach the auth endpoint and
guess (or enumerate) a username could lock that account out for the
entire lockout window — the rate limiter happily lets each IP make
its share of bad-password attempts before clamping, which is enough
to trip the per-account threshold in seconds. The reporter
demonstrated a complete DOS by spoofing X-Forwarded-For with
OXICLOUD_TRUST_PROXY_HEADERS=true.
Fix: change the lockout cache key from `username` to `username|ip`.
A flood from one IP locks that IP out of that account, but a
legitimate user coming from a different IP is unaffected.
Changes:
- LoginLockoutService::{check, record_failure, record_success} take
client_ip as a second argument; cache key is built via Self::key
(`format!("{username}|{ip}")`).
- middleware/rate_limit.rs: factor out extract_client_ip_from_parts
(HeaderMap + Option<&SocketAddr>) so handlers that don't take a
full Request<B> can still derive the same client identifier
extract_client_ip uses. extract_client_ip now delegates to it.
- auth_handler.rs login: derive client_ip from headers (the only
signal available without ConnectInfo) and pass it through to all
three lockout calls.
- nextcloud/basic_auth_middleware.rs: do the same with the full
Request via extract_client_ip.
Tests:
- Updated existing 4 unit tests to thread an IP arg.
- New does_not_lock_out_other_ips_for_same_account: lock from IP1,
assert IP2 still allowed (the #323 regression).
- New success_resets_only_the_acting_ip: a successful login from
IP2 must NOT clear an attacker's lockout from IP1.
Verification:
- `cargo build` ✅
- `cargo test login_lockout` → 6 passed (4 existing thread an IP
arg without behaviour change, 2 new pin the per-IP scoping).
Signed-off-by: SAY-5 <say.apm35@gmail.com>
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
purpose: avoid current scheme:
1. write to disk
2. reopen file to read data and digest it
now: digest is done while writing data to disk
all other implementation than Nextcloud are corrrect
add OXICLOUD_CHUNK_MAX_BYTES which correspond to the max upload chunk allowed
(differs from OXICLOUD_MAX_UPLOAD_SIZE which is the max total size of a file)
hurl test validate the change
Streams the request body straight to the chunk file with peak heap of
~one HTTP frame, regardless of chunk size or the configured cap. The
`storage.chunk_max_bytes` config (env `OXICLOUD_CHUNK_MAX_BYTES`,
default 100 MB) bounds a single PUT — separate from `max_upload_size`
which governs whole-file uploads. Without this separation, a client
could submit a chunk up to the whole-file cap (10 GB default) and
monopolise server memory.
- add /ocs/v1.php/cloud/user route (same as /ocs/v2.php/cloud/user )
- add also /remote.php/dav/avatars/{user}/{size}.png route (used by Nextcloud sync)
note: webp are transcoded in png, no cache but using pragma HTTP header
PROPFIND on upload chunk is necessary for application to resume the upload
it is used at least by Nextcloud Android app
issue has been raised via #415
unable to test, so I am using a playbook in test/webdav to simulate a
PROPIND on remote.php/dav/uploads/{user}/{session}
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:
- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
Now streams the body to a temp file with incremental BLAKE3 and goes
through update_file_streaming (shared spool helper with the native
WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.
- DedupService::store_chunks materialized every new chunk's data in a Vec
before uploading. Now reads each new chunk by positioned I/O
(read_exact_at, off the runtime via spawn_blocking) just before its
upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.
- The upload spool used the OS temp dir, often tmpfs/RAM in containers
where its page-cache counts against the cgroup memory limit. Add
OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.
Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.
Refs #404
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
username: now optional, if defined 2..64 chars
password: now optional (no mode __NO_PASSWORD...__)
oidc: now optional
important: if need Nextcloud, username must be defined
* make more coherent lifecycles
* remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
* reduce risk of orphean objects
* ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
* more details on docs/architecture/file-and-blob-lifecycle.md :
```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {
fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
fn on_file_updated(file_id, blob_hash, content_type);
fn on_file_copied(file_id, blob_hash, content_type, source_id)
fn on_file_deleted(file_id);
}
// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {
fn on_blob_created(blob_hash, content_type);
fn on_blob_deleted(blob_hash);
}
```
Bug 1 & 2 (webdav_handler.rs handle_put() update branch):
- After a successful file update via WebDAV PUT, if the content type is a supported image:
a. delete_thumbnails(file_id) — evicts the stale moka cache entry
b. Spawns a background task to read the new blob bytes and call generate_all_sizes_background_from_bytes
Bug 3 & 4 (dedup_service.rs):
- Added thumbnail_service: Option<Arc<ThumbnailService>> field with a with_thumbnail_service() builder
- In remove_legacy_reference(): calls delete_blob_thumbnails(hash) when ref_count hits 0
- In remove_manifest_reference(): calls delete_blob_thumbnails(file_hash) when manifest's last ref is dropped
- Wired in di.rs — the thumbnail service is created before dedup service so the ordering works cleanly
Thumbnails are now keyed by blob_hash on disk so identical files share
a single set of thumbnails (icon/preview/large). For 4000 duplicate
files with the same content, this reduces thumbnail storage from 12,000
files to just 3.
Changes:
- get_thumbnail_path() keys by blob_hash instead of file_id
- get_thumbnail(), get_cached_thumbnail(), generate_all_sizes_background()
accept blob_hash parameter for disk dedup
- generate_all_sizes_background() fast path: if blob-hash thumbnails
already exist on disk, skip image processing entirely and just
populate moka cache for the new file_id
- delete_thumbnails() only invalidates moka cache (shared disk
thumbnails must not be deleted when one file is removed)
- delete_blob_thumbnails() added for GC; garbage_collect() now cleans
up orphaned thumbnail files alongside blob files
- External thumbnails (video frames) stored as ext-{file_id}.jpg
since they are client-generated and not dedup-able
- ThumbnailPort trait updated with blob_hash parameters
- All handler call sites updated (file_handler, preview_handler)
- Tests updated for new signatures
The Nextcloud Android client compares ETags before and after upload to
verify its write landed. OxiCloud was returning the stable file UUID as
the ETag, which never changed on content updates, causing false
SYNC_CONFLICT errors on every upload.
Five fixes applied:
1. Thread blob_hash (SHA-256) through File entity, FileDto, all read/write
queries, and all WebDAV/PROPFIND responses as the ETag — changes on
every content update, no DB migration needed.
2. Honor X-OC-Mtime header: parse the client-supplied mtime and use it
for updated_at via COALESCE(to_timestamp($n), NOW()).
3. Disable phantom checksum capability (preferredUploadType/supportedTypes)
that the server never actually implemented, stopping retry loops.
4. Add nc:creation_time and nc:upload_time to PROPFIND responses.
5. Return oc-etag header in chunked upload MOVE (assemble) responses.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix all clippy lints (collapsible if, clone on Copy, needless borrow,
redundant bindings, unused params) and apply rustfmt across the codebase.
Update test mocks to match Uuid-based trait signatures.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- rate_limit: cache OXICLOUD_TRUST_PROXY_HEADERS in OnceLock<bool> to avoid
syscall on every request (~500ns → ~1ns)
- auth middleware: insert Arc<CurrentUser> instead of bare CurrentUser;
all 5 extractors now clone Arc (~1ns) instead of 4 Strings (~60-100ns)
- search_service: pre-compute query.to_lowercase() once before loops,
eliminating N redundant heap allocations per search
Add column-gap to list header/items, prevent rubber-band selection from
swallowing clicks on list headers and file items, preserve rubber-band
selection after mouseup, and extract type aliases to resolve clippy
type_complexity warnings.
- Add /remote.php/dav discovery endpoint for Android app server detection
- Add /index.php/204 connectivity check endpoint (returns 204 No Content)
- Redirect login flow to nc:// deep link for mobile credential delivery
- Support GET/HEAD on folders (NC clients use as existence checks)
- Recursive MKCOL to create missing parent directories
- Fix single-file PROPFIND returning empty multistatus response
- Strip instance suffix from preview fileId (e.g. "00000326ocnca")
- Add recommendations stub endpoint
- dedup_service: fall back to copy+delete when rename() fails with
EXDEV (os error 18), which occurs when temp and blob dirs are on
different filesystems
- NC webdav_handler: return 405 instead of 500 when MKCOL targets an
existing folder (RFC 4918 §9.3.1)
- Allow async_fn_in_trait lint crate-wide (internal project, 413 warnings)
- Add integration_tests feature to Cargo.toml to fix unexpected cfg warnings
- Collapse nested if statements into single conditions (13 locations)
- Add type_complexity allows on pg repository functions (12 locations)
- Fix dead code warnings in test modules with allow attributes
- Fix E0599 by gating new_stub() for integration_tests feature
- Add result_unit_err and result_large_err allows where appropriate
- Apply rustfmt formatting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Nextcloud app uploads sent application/octet-stream as Content-Type,
causing images to not be recognized. Now both WebDAV PUT and chunked
upload paths call refine_content_type() which detects via magic bytes,
then extension, then falls back to the client header. Also fixes
update_file() which previously hardcoded application/octet-stream.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement a complete Nextcloud client compatibility layer so that
Nextcloud desktop/mobile sync clients can connect to OxiCloud.
Key additions:
- Login Flow v2 (device auth) with OIDC bridge support
- WebDAV handler compatible with Nextcloud clients (PROPFIND, GET,
PUT, DELETE, MKCOL, MOVE, COPY, HEAD, PROPPATCH)
- OCS API endpoints (user info, capabilities, notifications stubs,
sharees, unified search)
- Basic Auth middleware with app password verification, account
lockout integration, and blake3-keyed auth cache
- App password management: create, list, revoke via both native
API (JWT-authenticated profile page) and Nextcloud OCS endpoints
- Nextcloud file ID mapping (oc:fileid) with persistent DB storage
- Chunked upload support (Nextcloud v2 chunking protocol)
- Trashbin WebDAV interface
- Avatar (SVG placeholder) and preview (redirect) handlers
- User profile page with app password management UI
- URL user validation on all DAV routes (403 on mismatch)
- Database schema for app_passwords and nextcloud_object_ids tables
All services are behind a `nextcloud.enabled` config flag and
cleanly separated under src/interfaces/nextcloud/.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DioCrafts