Resolves the four-file conflict with PR #444, which landed a parallel
implementation of the same batch-fsync design (unsynced chunk writes plus
one sync_blobs durability barrier). Both sides converged on identical
trait signatures; main's version is kept for all overlapping code since
it is already merged and carries unit tests:
- blob_storage_ports.rs: doc-comment-only conflict, main's docs kept
- dedup_service.rs: main's pipeline kept. The naive textual merge would
have retained BOTH durability-barrier/INSERT blocks, double-counting
ref_count for every new chunk; the duplicated block from this branch
is dropped
- encrypted_blob_backend.rs, local_blob_backend.rs: main's
implementations kept (fsync_paths_parallel + tests); this branch's
equivalent helpers (fsync_dir, fsync_blob_file, SYNC_BLOBS_CONCURRENCY)
are referenced nowhere else and are dropped
Everything unique to this branch (streaming PROPFIND, range requests,
batch ID resolution, CalDAV/CardDAV indexed lookups, transcode and
breadcrumb changes) merges cleanly and is unchanged.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Creating a grant for a group dispatched the notification emails one at
a time inside the HTTP request — 30 members × ~500 ms of SMTP ≈ 15 s
holding the POST /api/grants response (the code carried a TODO
acknowledging it).
Dispatches are independent (coalescing and rate-limiting key on the
(granter, recipient) pair, distinct per member), so run them through
`buffered(6)`: ~6× less wall time for group fan-outs while capping
parallel SMTP sessions, with outcome order still matching member order.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The CDC chunk path paid `sync_all` + parent-dir fsync + one ref-count
upsert round-trip PER ~256 KB chunk: a new 1 GB file ≈ 8 000 fsyncs +
4 000 sequential INSERTs on the upload critical path (seconds on SSD,
tens of seconds on HDD).
- New `put_blob_from_bytes_unsynced` + `sync_blobs` on
BlobStorageBackend. Defaults delegate to the durable variants, so
remote backends (S3/Azure — durable on PUT) are untouched.
LocalBlobBackend writes chunks without fsync and `sync_blobs` then
fsyncs all files concurrently (kernel coalesces the writeback) plus
one fsync per DISTINCT shard directory — at most 256 — instead of one
per chunk. EncryptedBlobBackend delegates both so encryption over the
local backend keeps the optimization.
- store_chunks: uploads use the unsynced variant; one `sync_blobs`
barrier runs before anything references the chunks, and the per-chunk
ref-count upserts collapse into a single
`INSERT ... SELECT unnest(...) ON CONFLICT` statement.
Durability contract is unchanged: every chunk is on stable storage
before the manifest row that references it commits. A crash mid-upload
leaves chunk files without DB rows — the same orphan class the
per-chunk scheme already produced, just a wider window.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.
Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
GET /api/auth/me ran a synchronous O(N) SUM(size) over all the user's
files plus an unconditional UPDATE of auth.users on every call — one of
the most frequently hit endpoints — adding per-request latency, DB write
load, dead tuples and WAL even when nothing changed.
- /api/auth/me now serves the cached storage_used_bytes column instead of
recomputing it inline.
- New StorageUsageService::start_reconciliation_job runs a periodic sweep
on the maintenance pool that keeps the cached value current for every
mutation (uploads, deletes, trash), so freshness no longer depends on
hitting /me. Interval via OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default
600s, floored at 30s; first sweep deferred one interval to avoid boot load).
- update_storage_usage only writes when the value actually changes
(IS DISTINCT FROM), so the sweep produces no dead tuple / WAL on no-ops.
- New covering partial index idx_files_user_size_active makes the usage
SUM an index-only scan instead of a heap scan over all the user's files.
Also collapse the same pre-existing clippy collapsible_else_if in
carddav_handler that blocks the -D warnings gate on this base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Large uploads (e.g. ~800 MB ISOs) could OOMKill the process, even on
dedup hits, due to three separate full-file-in-memory paths:
- NextCloud PUT (/remote.php/dav) buffered the entire body in RAM via
body::to_bytes before any dedup logic, then re-wrote and re-hashed it.
Now streams the body to a temp file with incremental BLAKE3 and goes
through update_file_streaming (shared spool helper with the native
WebDAV PUT handler); peak heap is ~one HTTP frame regardless of size.
- DedupService::store_chunks materialized every new chunk's data in a Vec
before uploading. Now reads each new chunk by positioned I/O
(read_exact_at, off the runtime via spawn_blocking) just before its
upload; peak heap bounded to ~CHUNK_UPLOAD_CONCURRENCY x CDC_MAX_CHUNK.
- The upload spool used the OS temp dir, often tmpfs/RAM in containers
where its page-cache counts against the cgroup memory limit. Add
OXICLOUD_UPLOAD_TMPDIR to point the spool at real disk.
Also collapse a pre-existing clippy collapsible_else_if in carddav_handler.
Refs #404
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
- add resentd method in share menu item (work for both internal and external users)
- user can disable email notification via his properties
- add env variable from admin to disable notifications
- OIDC JIT define the locale only at user creation, user can so change his preference later
- invited users will inherit inviter's locale
- email will use prefered_locale
- login to a new browser will use prefered_locale
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
magic-link as now 2 modes:
- invitation: long TTL (24), no challenge
- passwordless login: short TTL (10min), cookie challenge to ensure that
user goes back to same browser (no man in the middle capturing email)
Important: anti-enumeration is active only if SMTP is defined, welcome email can be used
otherwise it is a classic registration with ok or conflic if account alrady exists
OXICLOUD_MAGIC_LINK_OPEN_TO_PASSWORD_USERS (default false)
For security I recommand to keep it false
OIDC cannot be bypassed because OIDC may have MFA in place
Backend
- RegisterDto — username and password both become Option<String> with #[serde(default)] so JSON can omit them entirely.
- AuthApplicationService::register — username uniqueness check skipped when None (multiple NULLs OK under the UNIQUE index); password hashing skipped when None; User::new called with the actual Options instead of forcing Some(...).
- auth_handler::register — branches on dto.password.is_none(). With password → existing 201 + UserDto. Without → triggers MagicLinkInviteService::send_login_link(&email) best-effort, then returns 200 + {"message": "Check your email…"}. The
OIDC-mode-disables-password-registration gate now only fires for the password path (email-only signup is still allowed even in OIDC-only mode, because it doesn't store a password).
- magic_link_handler::redirect_target — new 3-way decision tree:
- Resource target (folder invitation) → /#/files/folder/{id} (existing)
- NULL resource + is_external = false → /#/files (the welcome path for new internal users — they have a home folder)
- NULL resource + is_external = true → /#/sharedwithme (the existing external-user landing)
Tests
- New tests/api/registration.hurl with 9 requests covering: classic (with-password) register → 201 + UserDto, email-only register → 200 + uniform message + welcome magic-link captured, redemption → 302 to /#/files + cookies set, profile read → username
absent + is_external: false, resend magic-link works (eligible while passwordless), cleanup deletes both new users.
- Wired into tests/api/run.sh right after auth_login.hurl.
Plan additions
- auth-simplification.md gained PR 22 at the bottom of the PR sequence — device-bound magic-link redemption via challenge cookie + asymmetric TTLs (login: 10 min, invitation: 24 h). Full design recap, schema migration, config knobs
(OXICLOUD_MAGIC_LINK_LOGIN_TTL_MINUTES / _INVITE_TTL_HOURS), and Hurl coverage outline are in the plan. Slots in before PR 21's docs so the architecture page describes the final state from the start.
Checks — cargo fmt, cargo clippy --all-features --all-targets -- -D warnings, cargo test --lib (297 passed), biome, stylelint, tsc, full Hurl suite (16 files) all green.
username: now optional, if defined 2..64 chars
password: now optional (no mode __NO_PASSWORD...__)
oidc: now optional
important: if need Nextcloud, username must be defined
- add possibility to grant an external user.
- route /api/users/{id} added (rate limited for security)
- security: start route limitation for external users
ex: they must not browse /api/users/{id} nor addressbook
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
reflect OIDC
schema migration, User entity additions, and three defense-in-depth gaps closed, with all 280 unit tests and 13 Hurl files green
infrastructure (lettre + EmailSender port).
- Migration migrations/20260612000003_users_username_email_login.sql — adds nullable given_name/family_name columns to auth.users.
- User entity (src/domain/entities/user.rs) — has_login_credential() placeholder-check encapsulation, set_username revalidating setter, given/family-name fields + getters/setters, validate_username widened 32→254 and now accepts email shape.
from_data_full extended with two new params; all 7 callsites in user_pg_repository.rs updated.
- Schema-side legacy guards (src/application/services/auth_application_service.rs) — bumped the duplicated 32-char check in setup_create_admin and admin_create_user to 254 to match.
- Gap #1 (subject_group_service.rs) — add_member now rejects external candidates with an audit-logged AccessDenied. Service gained an Arc<UserPgRepository> field, wired through DI. New integration test test_external_user_cannot_be_added_as_member.
- Gap #2 (user_repository.rs + auth_ports.rs + user_pg_repository.rs) — list_users/search_users gained an include_external: bool param defaulting effectively to false everywhere internal-user-facing. auth_application_service exposes a new
list_users_including_external for the admin surface.
- Gap #3 (pg_acl_engine.rs) — expand_user now SELECTs is_external and skips INTERNAL_GROUP_ID for externals; defaults to is_external=true on missing user to fail closed.
prepare identity service for external users, support of:
- magic_link (url challenge via email)
- self issued oidc (eventually social login)
- open cloud mesh
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
- SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
- DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
- Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
this implements first version (manageable only by admin right now)
routes:
GET /api/groups
List subject groups (paginated). Admin-only.
POST /api/groups
Create a new ReBAC subject group. Admin-only. The name must match the RFC 5321 local-part shape and be globally unique (case-insensitive).
GET /api/groups/search
Search non-virtual groups by name substring. Authenticated only (no admin role required) — backs the share-dialog recipient autocomplete.
GET /api/groups/{id}
Fetch a single group's details. Admin-only.
DELETE /api/groups/{id}
Delete a group. Cascades to `subject_group_members` (FK) and to `access_grants` rows referencing this group as a subject. Admin-only.
PATCH /api/groups/{id}
Update a group's metadata. Admin-only. v1 only persists name renames.
GET /api/groups/{id}/effective-members
List every user transitively reached through this group (members of members of members, etc.). Used by admin / audit tooling. Admin-only.
GET /api/groups/{id}/members
List the *direct* members of a group (one level only). Admin-only.
POST /api/groups/{id}/members
Add a member to a group. Exactly one of `user_id` / `group_id` must be provided. Adding a group-member runs a write-time cycle check and a nesting-depth check (max 8). Admin-only.
DELETE /api/groups/{id}/members/group/{gid}
Remove a nested group-member from a group. Admin-only.
DELETE /api/groups/{id}/members/user/{uid}
Remove a user-member from a group. Admin-only.
fix hurl
groups
round
groups
Claude