Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:
- GET /api/files/{id}/manifest returns the file's chunk recipe
({file_hash, total_size, chunks}). Owner-scoped like the rest of the
delta surface (Read permission through the authz engine first, then
the chunk layer's possession standard; shared files use the regular
download endpoints). A manifest is immutable for a given file_hash,
so it is served with ETag = file_hash and If-None-Match answers 304 —
polling sync clients pay one header round-trip per unchanged file.
Legacy pre-CDC blobs are presented as a single-chunk manifest of
themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
[u32 BE length][bytes] frames in request order — the same wire format
the upload direction uses. Entitlement is the same possession rule as
negotiate/commit (chunks reachable through the caller's own files);
anything else returns 404 {not_available} — deliberately
indistinguishable from "never existed" — with a
delta_download.rejected audit event. Batches are bounded by the
chunk_max_bytes budget; Content-Length is exact (sizes come from the
dedup index) and peak RAM is one backend read frame.
Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).
Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 2 — the client side of "upload only what changed", closing the
delta-sync plan.
WASM (wasm/oxicloud-hash): DeltaChunker adds incremental FastCDC with
the server's exact crate and parameters (64K/256K/1M) next to the BLAKE3
hasher. The incremental split is provably identical to a single pass:
every chunk except the last ends on a content/max-size condition whose
decision window was fully buffered, so only the tail is provisional and
re-examined as slices arrive. A mirror test — the client twin of the
server's stream≡slice test — chunks 4 MiB of xorshift noise with
adversarial slice sizes (7 B … 8 MiB) and requires boundary-for-boundary
equality with one FastCDC pass. Vendored artifacts rebuilt (55 KB wasm).
Worker (static/js/workers/deltaWorker.js): the full protocol off the
main thread with OVERLAPPED stages — 8 MiB file slices feed the chunker
while earlier batches (256 hashes) negotiate and their missing chunks
upload through a 2-deep PUT pool (≤8 MiB framed bodies, bytes re-sliced
from the File at send time, never hoarded). Commit handles 409
still_missing by uploading exactly the named hashes and retrying.
Orchestrator (features/files/deltaUpload.js): threshold (8 MiB),
worker lifecycle + size-scaled timeout, progress relay to the upload
bell, conclusive-outcome mapping (201/200, 507 quota, 409 name
conflict) and silent fallback to the byte upload for everything else.
Wired into uploadFiles and uploadFolderEntries, which now surface one
batch summary of the bytes dedup saved. This subsumes the whole-file
instant-upload module — a fully-known file negotiates to nothing
missing and the commit short-circuits on possession — so
instantUpload.js and hashWorker.js are removed (the /api/dedup/check
and /api/files/by-hash endpoints remain for API clients).
Verified end-to-end against PostgreSQL 16 — the cross-boundary proof
the whole design hangs on, in both directions: a 24 MB file byte-
uploaded (server-side CDC) then edited and delta-negotiated with
WASM-computed chunks reported missing 1/74 (boundaries bit-identical),
synced with 344 KB on the wire vs 24 MB (98.6% saved) and downloaded
byte-identical; inversely, a file created via delta then byte-uploaded
as identical content produced a server-side manifest DEDUP HIT with the
same content_hash. Insertion at the head of the file (the adversarial
CDC case) still negotiated missing 1/74. Chunk+hash throughput ≈275 MB/s
in V8 with SIMD128.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
to help administratorrs to defined correctly their storage architecture
- add coalesced protection to avoid mail bombing if an invited goes many grant in a short period
- add resentd method in share menu item (work for both internal and external users)
- user can disable email notification via his properties
- add env variable from admin to disable notifications
- remove the hardcoded list of locales in favor of a discovry on start time
- server will stop on badly formatted locale .json
- add server.* entries for serer side translation
server side translation will be used for templating and email
note: no json in some embded html (like in /magic), amount of work was similar
1. Identity model — email is the identity, three optional credential slots (username, password_hash, oidc_subject), the @-in-username ban that makes namespaces provably disjoint.
2. Login dispatcher — @-in-input decides the path. Single DB lookup.
3. Login paths — four ways in (username+password, email+password, email+magic-link, OIDC redirect) with their pre-conditions.
4. Magic-link eligibility ladder — the three-branch table with OIDC unconditional reject, password flag-gated, no-credential allowed.
5. Registration paths — four ways (/api/auth/register with full creds, email-only, grant-by-email invitation, OIDC JIT).
6. Anti-enumeration — per-endpoint table showing what each surface returns and why; explicit note that instance-wide policy stays visible.
7. Security trade-offs — mailbox-as-bypass (lenient vs strict), no native MFA, magic-link as bearer token, enumeration via timing.
8. Rate limits — the five caps from PR 12 + classic auth, with env knobs.
9. Audit events — table of 12 event names with reason keys; every entry verified grep-able in src/.
10. Migration path — what existing instances see when PR 16-20 land (forward-only, non-destructive).
11. Future direction — login_strategy — the seven-row matrix sketch as the explicit-policy north star.
12. Out of scope — TOTP/WebAuthn, external→internal promotion, session_kind, differentiated session TTL, OCM, email-verified gates (PR 23), device-bound tokens (PR 22), anti-enum timing parity, per-user opt-out, login_strategy implementation.
13. Related documents — cross-refs to magic-link-auth, ReBAC, share-integration, env-vars.
- SMTP has a mock to enable end to end test and validate the whole path
(via OXICLOUD_SMTP_MOCK)
- add email normalisation ( including punicode)
- api to share to external user
prepare identity service for external users, support of:
- magic_link (url challenge via email)
- self issued oidc (eventually social login)
- open cloud mesh
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
- SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
- DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
- Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
this implements first version (manageable only by admin right now)
routes:
GET /api/groups
List subject groups (paginated). Admin-only.
POST /api/groups
Create a new ReBAC subject group. Admin-only. The name must match the RFC 5321 local-part shape and be globally unique (case-insensitive).
GET /api/groups/search
Search non-virtual groups by name substring. Authenticated only (no admin role required) — backs the share-dialog recipient autocomplete.
GET /api/groups/{id}
Fetch a single group's details. Admin-only.
DELETE /api/groups/{id}
Delete a group. Cascades to `subject_group_members` (FK) and to `access_grants` rows referencing this group as a subject. Admin-only.
PATCH /api/groups/{id}
Update a group's metadata. Admin-only. v1 only persists name renames.
GET /api/groups/{id}/effective-members
List every user transitively reached through this group (members of members of members, etc.). Used by admin / audit tooling. Admin-only.
GET /api/groups/{id}/members
List the *direct* members of a group (one level only). Admin-only.
POST /api/groups/{id}/members
Add a member to a group. Exactly one of `user_id` / `group_id` must be provided. Adding a group-member runs a write-time cycle check and a nesting-depth check (max 8). Admin-only.
DELETE /api/groups/{id}/members/group/{gid}
Remove a nested group-member from a group. Admin-only.
DELETE /api/groups/{id}/members/user/{uid}
Remove a user-member from a group. Admin-only.
fix hurl
groups
round
groups
initially server was accepting multiple instances on same port (Linux and MacOS only)
this can create issues during development (if a dev forget another running instance...)
Add OXICLOUD_REUSE_PORT variable to activate it, so only admins knowing this feature can activate itt
(permits multiple instance + let the OS schedduler to decide which process will handle a request)
if not enabled, other instance will receive exit with a:
`Error: Os { code: 48, kind: AddrInUse, message: "Address already in use" }`
- Add src/application/dtos/cursor.rs with three shared types:
· PageCursor trait — default base64url+JSON encode/decode; one bare
impl line per cursor struct
· CursorQuery struct — standard limit/cursor/sort_by query params with
limit_clamped() and decode_cursor<C>() helpers; compose via flatten
· CursorListResponse<T> — standard {items, next_cursor?} envelope with
from_oversized() and with_cursor() builders
- Migrate GrantCursor to impl PageCursor (remove duplicate encode/decode)
- Update GET /api/grants/incoming/resources:
· SharedWithMeQuery now embeds CursorQuery via #[serde(flatten)]
· Replace file/folder nullable pair with ResourceContentDto (untagged
enum) under a single always-present 'resource' field
· SharedWithMeDto is now a type alias for CursorListResponse<SharedWithMeItemDto>
· Handler uses q.paging.limit_clamped() and decode_cursor<GrantCursor>()
- Add docs/architecture/resource-listing.md — authoritative contract for
all listing endpoints (cursor design, SQL keyset WHERE, sort_by naming,
Rust + JS skeletons, compliance table, migration guide)
- Register doc in VitePress sidebar and architecture index
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* make more coherent lifecycles
* remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
* reduce risk of orphean objects
* ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
* more details on docs/architecture/file-and-blob-lifecycle.md :
```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {
fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
fn on_file_updated(file_id, blob_hash, content_type);
fn on_file_copied(file_id, blob_hash, content_type, source_id)
fn on_file_deleted(file_id);
}
// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {
fn on_blob_created(blob_hash, content_type);
fn on_blob_deleted(blob_hash);
}
```
Claude