After every upload, maybe_update_storage_usage spawned a full
`SUM(size) OVER all the user's non-trashed files` to refresh
auth.users.storage_used_bytes — O(N) in the user's file count per upload,
i.e. O(N²) for a bulk upload. (The covering index makes it index-only but
still scans N rows.)
Replace it with an O(1) incremental `storage_used_bytes += size`, keyed by the
file's owner_id (dropping the brittle "My Folder - <user>" path-parsing hack).
Deletes/trash never decremented this value — they already rely on the periodic
reconciliation sweep — so the model is unchanged: the sweep remains the
correctness backstop for every mutation, and the counter is clamped at 0.
Both stay fire-and-forget on a background task, off the upload's latency path.
Benchmarked (per-call, vs the user's existing file count N):
N=1k: full-SUM 202us vs incremental 123us
N=10k: full-SUM 1185us vs incremental 113us (10x)
N=50k: full-SUM 5397us vs incremental 114us (47x — incremental is flat O(1))
Bulk upload of 10k files (insert + usage update each):
full-SUM (O(N²)) 10.37s -> incremental (O(N)) 4.89s (>2x, diverges with scale)
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
save_file_with_blob_impl did three sequential DB round-trips per upload:
resolve_user_id (SELECT folders.user_id), the INSERT, then lookup_folder_path
(SELECT folders.path) — the first and third re-reading the same folders row.
Replace them with a single statement: a `parent` CTE reads the folder once and
the INSERT derives user_id from it and returns the path via the CTE, in one
round-trip. An empty CTE (folder vanished between ingest and insert) inserts
zero rows and now surfaces as a clean NotFound instead of a generic owner
error. Deadlock retry, blob-ref compensation, and the 23505 (duplicate name)
mapping are preserved; owner resolution + insert are now atomic (no TOCTOU).
What it does and doesn't buy (benchmarked, honest):
- Server CPU: UNCHANGED. A server-side 50k loop is identical (13.6s vs 13.6s)
— the two extra folder reads are cached point lookups, negligible against
the INSERT + per-statement triggers + 13 indexes.
- Client-observed latency: 2 fewer client<->DB round-trips per upload. At the
measured ~189us/round-trip on localhost that's ~0.38ms/upload; on a
networked DB (RTT 0.5-1ms) ~1-2ms/upload.
- Connection pool: the metadata phase holds a pooled connection for 1
round-trip instead of 3, freeing it ~3x sooner under upload concurrency.
So this is a latency + connection-utilization win (and an atomicity/cleanup),
not a server-CPU win. register_file_deferred keeps its own path.
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
FileContentCache (moka, 512 MiB) was keyed by the file UUID, so content that
the CDC store already deduplicates to ONE blob on disk was cached once PER
FILE in RAM: N files sharing a blob held N copies, all counting against the
512 MiB cap. With effective dedup the cache filled with duplicates and
thrashed.
Key it by the blob hash instead (already on FileDto::content_hash):
- The in-RAM cache now benefits from dedup like the disk does — each distinct
blob is cached once and shared across every file/user that references it,
so a download by user A warms the cache for user B's identical content.
- Content is immutable by hash, so entries never go stale; the existing
invalidate(file_id) calls become harmless no-ops (a UUID never matches a
hash key) and can be removed in a later cleanup.
- ETag is now the immutable content hash (strong validator).
- Guarded: a hash-less stub DTO disables caching for that request rather than
colliding every such file on the empty key.
Response Content-Type still comes from the DTO, not the cache, so keying does
not affect the served MIME (verified).
Benchmark (real moka, exact 512 MiB/weight config, 400 files x 4 MiB = 1600 MiB
working set, 4000 uniform-random accesses):
dedup 1x : file_id 35.8% hit / 2568 reads vs hash 35.6% / 2577 (no dedup -> no change; control)
dedup 5x : file_id 35.8% hit / 2570 reads vs hash 98.0% / 80 (32x fewer disk reads, RAM 512->320 MiB)
dedup 20x: file_id 35.1% hit / 2596 reads vs hash 99.5% / 20 (130x fewer disk reads, RAM 512->80 MiB)
The win scales with the dedup ratio; with no dedup it is a no-op.
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
read_blob_stream / read_blob_range_stream reassembled a CDC file by fetching
its chunks with `buffered(1)` — strictly sequential, so the next chunk's
backend fetch (a file `open` locally; a full request round-trip on S3/Azure)
only started after the current chunk was fully drained.
A benchmark of the exact pipeline (stream::iter(chunks).map(get).buffered(K)
.try_flatten()) showed a blind `buffered(4)` is the WRONG fix: on a local
disk it is neutral on a warm page cache and ~37% SLOWER cold, because
concurrent opens turn one sequential read into several competing random-I/O
streams over content-addressed (scattered) chunk files. The win is entirely
on remote backends, where per-chunk request latency dominates and overlapping
fetches hide it (≈ linear in K).
So the read-ahead depth is now a backend hint, not a constant:
- BlobStorageBackend::read_prefetch() default 1 (sequential; safe for local).
- S3 / Azure override to 8 (overlap GETs to hide TTFB).
- cached / encrypted / retry / migration delegate to the backend that serves
the bytes.
- Both CDC read paths use `self.backend.read_prefetch().max(1)`.
Net: local backend unchanged (no regression); remote reassembly ~4-8x faster.
Ordered `buffered` (not buffer_unordered) keeps chunks in sequence.
Bench (per-chunk fetch-latency model): buffered(1)->(4)/(8) = x3.9 / x7.8
@1ms, x4.0 / x8.1 @5ms, x4.0 / x8.0 @20ms. Local warm: 230ms@1 vs 227ms@4
(noise); local cold: 425ms@1 vs 585ms@4 (why local stays at 1).
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
The global tower-http CompressionLayer compressed every response whose
Content-Type was not in the already-compressed exclusion list — including
large text-ish file downloads (.csv/.log/.sql/.json/source). That (a) burned
CPU re-encoding multi-GB bodies on the request path with no cached result,
and (b) made tower-http strip Content-Length and Accept-Ranges, breaking
byte-range seek and download resume.
Add a NotForDownloads predicate that skips compression for any response
carrying Content-Disposition (every download surface: REST file, share,
folder/zip, batch-zip, inline previews). API JSON and static assets never
set Content-Disposition, so they stay compressed.
Verified with the real tower-http layer + this exact predicate (64 MiB
text/plain download):
- Full download regains Content-Length + Accept-Ranges (were stripped);
/api/data stays brotli-compressed (fix is surgical).
- CPU: 2.7-3.6x less per download sequential; 6-7x less under 8-way concurrency.
- TTFB: 44ms->1ms (gzip), 110ms->1ms (brotli).
- Delivered content throughput: 2.4-2.8x higher.
Tradeoff: genuinely-compressible downloads now send more bytes on the wire;
reclaim via compress-at-rest if it ever matters.
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
Pre-existing in the delta-upload service: Rust 1.93's stricter large_enum_variant lint flags the FileDto-carrying Done variant (the sibling variant is tiny). The value is short-lived (one per commit), so silence the lint rather than box FileDto and complicate every call site. Keeps the CI clippy -D warnings gate green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — Photos timeline now groups by real capture date instead of upload time. New MediaMetadataService (FileLifecycleHook) extracts EXIF DateTimeOriginal from images and container creation_time from videos (mov/mp4/mkv) via nom-exif, timezone-correct (OffsetTimeOriginal), persisting captured_at so the existing media_sort_date trigger takes over. Adds POST /admin/photos/metadata/reextract to backfill existing media. Falls back to upload date when no embedded date exists.
Frontend — premium grid cards: combined metadata line (relative date · size, owner avatar when shared), custom selection checkbox with a clear checked state, uniform full-width 4:3 thumbnail tiles independent of filename length, centered file-type icons, and a hit-test fix so checkbox/star/kebab clicks reach the controls (the decorative thumbnail no longer captures pointer events). Notification messages internationalised across all 16 locales. Broader polish: design tokens, a11y/focus-visible states, brand + PWA assets.
Chore — bump semver-compatible dependencies (cargo upgrade); add nom-exif 3.6.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both sides added a parameter to create_application_services and a
setup step before it: this branch's storage-usage/quota service (for
the instant-upload path) and main's Tantivy content index (for
SearchService). The resolution keeps both — the signature takes both
arguments and the build runs storage usage as step 3c and the content
index as 3d.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:
- GET /api/files/{id}/manifest returns the file's chunk recipe
({file_hash, total_size, chunks}). Owner-scoped like the rest of the
delta surface (Read permission through the authz engine first, then
the chunk layer's possession standard; shared files use the regular
download endpoints). A manifest is immutable for a given file_hash,
so it is served with ETag = file_hash and If-None-Match answers 304 —
polling sync clients pay one header round-trip per unchanged file.
Legacy pre-CDC blobs are presented as a single-chunk manifest of
themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
[u32 BE length][bytes] frames in request order — the same wire format
the upload direction uses. Entitlement is the same possession rule as
negotiate/commit (chunks reachable through the caller's own files);
anything else returns 404 {not_available} — deliberately
indistinguishable from "never existed" — with a
delta_download.rejected audit event. Batches are bounded by the
chunk_max_bytes budget; Content-Length is exact (sizes come from the
dedup index) and peak RAM is one backend read frame.
Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).
Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.
Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
(storage.search_index_dirty) - every write surface (REST, WebDAV,
NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
batched single-writer Tantivy commits; queue rows are deleted only
after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
reseeds it from Postgres, which remains the single source of truth
SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.
The frontend renders the snippet under the file name in list view.
New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.
https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 0 of the delta-sync plan. Re-uploading a file the user already has
(another device, a restore, a duplicate) used to transfer every byte just
for the server to discard them as a dedup hit. The frontend now computes
the file's BLAKE3 locally and, on a hit, registers the file with a single
~150-byte metadata call.
Server — POST /api/files/by-hash:
- All checks live in the application service per the AuthZ rule:
Create permission on the target folder via the authorization engine,
hash ownership via the existing user-scoped query (a non-owned hash
returns 404 — same shape as "no such blob" — and emits an
instant_upload.rejected audit event), quota on the logical size.
- On success: one ref_count bump + the existing save_file_with_blob row
registration (compensation included); is_new_blob=false so lifecycle
hooks skip thumbnail regeneration. ~10 ms warm.
- The storage-usage service is now built before the application services
and injected, instead of only living on AppState.
Client — WASM BLAKE3 + worker:
- wasm/oxicloud-hash: the exact same blake3 crate the server uses,
compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match
server content addresses bit for bit. Built by scripts/build-wasm.sh;
the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no
npm dependencies, no wasm toolchain needed for regular builds.
- static/js/workers/hashWorker.js streams the File in 8 MiB slices off
the main thread (constant RAM at any file size).
- features/files/instantUpload.js orchestrates: threshold (8 MiB — below
it the round-trips cost more than the bytes), user-scoped
/api/dedup/check, by-hash registration, and silent fallback to the
normal byte upload on any miss, race or unsupported environment.
Wired into both uploadFiles and uploadFolderEntries.
- biome.json vendors exclusion fixed to cover nested directories
(previous vendors were .mjs and never matched the *.js include).
Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals
the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms
warm with a 151-byte request (vs 20,971,873 bytes for the byte upload);
the copy downloads byte-identical and the manifest ref_count goes 1→2;
a second user probing the same hash gets exists:false and 404 plus the
audit line; duplicate name → 409, malformed hash → 400; worker and wasm
are served with correct MIME (application/wasm).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Two follow-ups to the streaming-upload work:
The dedup garbage_collect() pass only ran when a user manually emptied
their trash. Zero-reference rows — chunks orphaned by aborted streaming
uploads (registered at ref_count 0 by the ingest rollback) and blobs
dereferenced by trash expiry itself — could linger indefinitely on
instances where nobody empties trash. The periodic TrashCleanupService
sweep now ends every run with garbage_collect() (maintenance pool,
batched), bounding orphan lifetime to the cleanup interval.
Folder-ZIP creation was strictly sequential: open blob stream, deflate,
close, repeat — every per-file blob-store round-trip (PG lookup + backend
open; a full HTTP round-trip on S3/Azure) added to the wall clock. It now
runs as a 2-stage pipeline: a prefetch task streams the planned files'
content ahead of the writer through a bounded channel (~4 MiB), so the
next file's read latency overlaps the current file's compression. ZIP
entries are still written strictly in order, peak RAM stays flat, and a
writer error hangs up the channel so the prefetcher stops on its own.
Verified end-to-end against PostgreSQL 16: an upload aborted at ~14 MB
left exactly 30 ref_count=0 chunk rows which the GC then reclaimed
(8.5 MB, rows + physical files); a chunked upload completed with a wrong
MD5 returned 400 with the tee-computed digest and the SAME session then
completed successfully with the right checksum (parts persist — the old
assembly deleted them, so the documented retry never actually worked);
a 4-file folder ZIP downloaded and extracted byte-identical.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
JWT validation cache now stores Arc<TokenClaims> and validate_token
returns Arc<TokenClaims>. On a cache hit — the 99% path for every
authenticated request — the moka lookup was deep-cloning the whole
claims struct (5 Strings: sub, jti, username, email, role) on every
call. It is now a refcount bump. Read-only callers (admin middleware)
go through Deref and allocate nothing; the auth middleware clones only
the three fields it moves into CurrentUser (was 5 clones, now 3), and
the admin paths clone only role (was 5, now 1). A new test asserts the
hit path returns a pointer-equal Arc.
TokenServicePort::validate_token is the single trait method touched;
its only implementor is JwtTokenService and the only production callers
are the auth and admin middleware (the WOPI handler uses a separate
WopiTokenService).
Dockerfile: rust:1.94.1-alpine3.23 -> rust:1.96-alpine3.24 and
alpine:3.23.3 -> alpine:3.24.0 for the runtime stage.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Files uploaded before chunk_manifests landed (20260414000000) are stored
as ONE whole-file blob with no manifest. Every legacy fallback in
DedupService exists to serve them, and the cost concentrates on Range
reads: with encryption enabled, seeking inside a legacy video decrypts
the ENTIRE blob (AES-GCM is all-or-nothing), where a CDC file decrypts
only the overlapping <=1 MiB chunks.
This adds a one-time, idempotent background migration (spawned from the
composition root after dedup init, maintenance pool) that converts each
legacy blob into a regular CDC file, indistinguishable from a native
upload:
1. Spool the blob through the normal read path (decrypts when
encryption is on) to a per-attempt-unique temp file, verifying
BLAKE3 == hash; sizes come from the verified spool, never from the
legacy storage.blobs.size column (the manifest's total_size drives
Range arithmetic).
2. CDC-chunk + store chunks via the existing store_chunks (one
manifest reference per distinct chunk).
3. One short accounting TX with the blob row locked: manifest INSERT
with ref_count = N current file references, blob ref_count -= N,
row deleted only at exactly 0 - so single-chunk files (chunk hash
== file hash) keep the physical blob, which IS the chunk; only
bookkeeping moves, no bytes are rewritten.
4. Physical whole-file blob deleted only when its row dropped.
Races lean on the row lock: a concurrent identical upload landing a
legacy reference after commit keeps the blob row alive and that file
readable via the fallback (bounded space leak, never data loss); a
crash between chunk store and the TX over-counts one file's chunk refs
(also a bounded leak). Corrupt blobs (content != hash) are logged,
counted, excluded from the sweep and left untouched, with a hard cap
before aborting.
Per-hash failures never block the sweep; manifests are the resumability
marker, so a restart continues where it left off. The legacy read/write
fallbacks stay in place as the safety net while a deployment converges;
they can be deleted once fleets report "legacy re-chunk: nothing to do".
Opt-out via OXICLOUD_LEGACY_RECHUNK=false (documented in example.env)
for metered remote backends where the one-time re-read should be
scheduled deliberately.
Covered by five integration tests against real PostgreSQL (multi-chunk
accounting + Range across a chunk boundary, single-chunk physical-blob
preservation, corrupt-blob isolation, empty blob, and the full
encrypted-backend roundtrip); they run concurrently, which also
exercises the cross-sweep race handling.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Two hot-path fixes for DAV sync clients, which poll continuously:
1. App-password Basic Auth cache TTL 30s → 300s. Every cache miss costs
a full Argon2id verification (~50-100ms CPU) plus two DB queries; at
30s a continuously-syncing client re-paid that every 30s. 5 min cuts
it ~10x. Safe because revoke() already invalidates the user's cached
entries immediately; expiry/deactivation are only re-checked on a
miss, so they now have a <=5 min grace window (comparable to a JWT
access-token lifetime) — documented on the constant.
2. WebDAV lock store: replace the tokio::spawn + sleep scheduled per
acquire/refresh with Moka's per-entry Expiry policy. Office clients
refresh locks constantly, leaving thousands of orphaned sleeping
timers pinned in the runtime that were never cancelled. by_path now
carries the exact per-lock TTL via LockExpiry (no background tasks);
by_token keeps a 24h backstop and resolves through by_path, so a
lingering reverse-index entry can never resurrect an expired lock.
Adds unit tests for the expiry policy and acquire/refresh/release
(the module had none).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.
- Add find_events_by_ical_uids / get_contacts_by_uids through every
layer (domain repo trait → PG repo → storage port/adapter → use case
→ handler) using one indexed `= ANY($2)` query, mirroring the
existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
uid_from_multiget_href helper (percent-decoding, case-insensitive
extension strip, collection hrefs rejected). Exact UID matching also
removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
GET /api/address-books/{id}/contacts: optional ListQuery fields
thread through ContactUseCase::list_contacts to a paginated repo
query. Omitted params keep returning the full book (frontend and
DAV listing/sync paths unchanged).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
The read repository's hash_cache assumed blob hashes were immutable
per file_id, but update_file_content_from_temp remaps the SAME row to
a new hash via swap_blob_hash. After a WebDAV/NC PUT overwrite,
streaming downloads (>=10 MB tier, Range requests, video playback)
kept resolving the OLD blob for the 30 s TTI window — and because
every read refreshes the TTI, a polling client could pin the stale
mapping indefinitely, eventually turning into 500s once the old
blob's refcount hit zero and GC removed it.
The write repository now holds a shared handle to the same moka cache
(clones share storage) and invalidates the entry right after every
content swap and hard delete commits — covering every present and
future caller of the write port rather than one service path. Stale
doc comments claiming the mapping was immutable (and SHA-256) fixed.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Request-path thumbnail generation (REST get_thumbnail and NC preview)
read the full source blob into RAM and decoded it with no concurrency
bound — a first-view gallery of K images stacked K full-size buffers
and K parallel decodes on the blocking pool. The background hook had
the inverse ordering problem: it read the blob eagerly and only then
queued on the decode semaphore, so N concurrent uploads held N
originals in memory while waiting.
All three paths now acquire the decode semaphore first and read the
blob under the permit, capping peak RAM at permits x image size:
- new ThumbnailService::get_thumbnail_from_blob defers the blob read
into the moka init closure (cache and disk hits never touch the
blob); both handlers use it and no longer pre-read.
- get_thumbnail_from_bytes (direct-bytes variant) now also takes a
permit before decoding; shared generate_and_persist core keeps the
two entrypoints duplicate-free.
- generate_all_sizes_background_from_blob (renamed from _from_bytes)
checks blob existence and disk-dedup state, then acquires the
permit, then reads; the redundant pre-read spawn wrapper in
ThumbnailRefreshHook is gone.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Replace the per-row tree_modified_at bump triggers with AFTER ... FOR
EACH STATEMENT triggers using transition tables: each DML statement now
pays one bump covering the distinct ancestor chains of all affected
rows (locked in id order so concurrent bumps over overlapping chains
cannot deadlock), instead of one chain UPDATE per affected row.
Value-based change detection replaces UPDATE OF column lists (PG
forbids those with transition tables), so EXIF media_sort_date syncs
and no-op updates no longer bump at all, and file moves now invalidate
the source chain as well as the destination.
Also fix a latent bug surfaced while testing this: the descendant
path/lpath cascade (trg_folders_cascade_path) was declared AFTER
UPDATE OF path, lpath, but rename/move statements SET name/parent_id,
and BEFORE-trigger rewrites do not count for UPDATE OF — the cascade
never fired, leaving every descendant folder with a stale path/lpath
after any rename or move. The migration canonically repairs existing
trees and re-creates the cascade on the columns the app actually
writes.
delete_expired_bulk now deletes in LIMIT-ed batches (1000 files / 100
folders per round, each its own implicit transaction) ordered by
trashed_at and served by new partial indexes over trashed rows, so
retention purges no longer hold one unbounded transaction.
Verified against a local PG16 cluster: all migrations apply from
scratch, the new one re-runs idempotently, and a 16-case behavioral
battery passes (chain bumps for insert/update/delete/move/trash, EXIF
invisibility, no-op invisibility, cascade repair, FK-cascade depth
guards, batched purge shape and partial-index plan).
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
The five FileLifecycleHook call sites handed dto.etag (format
"{blob_hash[..16]}-{modified_at}") to the blob_hash parameter. Blob
lookups keyed by that value can never resolve, so background thumbnail
generation and audio metadata extraction silently failed on every
upload/copy/update, pushing all thumbnail work onto the request path.
Pass dto.content_hash — the raw full BLAKE3 hash already carried by
FileDto — instead.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Same whitespace-only reformat as branch claude/jolly-johnson-yso7z7:
PR #326 landed three files that fail cargo fmt --check and its CI run
skipped the Rustfmt job, breaking the check for every later Rust PR.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Resolves the four-file conflict with PR #444, which landed a parallel
implementation of the same batch-fsync design (unsynced chunk writes plus
one sync_blobs durability barrier). Both sides converged on identical
trait signatures; main's version is kept for all overlapping code since
it is already merged and carries unit tests:
- blob_storage_ports.rs: doc-comment-only conflict, main's docs kept
- dedup_service.rs: main's pipeline kept. The naive textual merge would
have retained BOTH durability-barrier/INSERT blocks, double-counting
ref_count for every new chunk; the duplicated block from this branch
is dropped
- encrypted_blob_backend.rs, local_blob_backend.rs: main's
implementations kept (fsync_paths_parallel + tests); this branch's
equivalent helpers (fsync_dir, fsync_blob_file, SYNC_BLOBS_CONCURRENCY)
are referenced nowhere else and are dropped
Everything unique to this branch (streaming PROPFIND, range requests,
batch ID resolution, CalDAV/CardDAV indexed lookups, transcode and
breadcrumb changes) merges cleanly and is unchanged.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Creating a grant for a group dispatched the notification emails one at
a time inside the HTTP request — 30 members × ~500 ms of SMTP ≈ 15 s
holding the POST /api/grants response (the code carried a TODO
acknowledging it).
Dispatches are independent (coalescing and rate-limiting key on the
(granter, recipient) pair, distinct per member), so run them through
`buffered(6)`: ~6× less wall time for group fan-outs while capping
parallel SMTP sessions, with outcome order still matching member order.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The CDC chunk path paid `sync_all` + parent-dir fsync + one ref-count
upsert round-trip PER ~256 KB chunk: a new 1 GB file ≈ 8 000 fsyncs +
4 000 sequential INSERTs on the upload critical path (seconds on SSD,
tens of seconds on HDD).
- New `put_blob_from_bytes_unsynced` + `sync_blobs` on
BlobStorageBackend. Defaults delegate to the durable variants, so
remote backends (S3/Azure — durable on PUT) are untouched.
LocalBlobBackend writes chunks without fsync and `sync_blobs` then
fsyncs all files concurrently (kernel coalesces the writeback) plus
one fsync per DISTINCT shard directory — at most 256 — instead of one
per chunk. EncryptedBlobBackend delegates both so encryption over the
local backend keeps the optimization.
- store_chunks: uploads use the unsynced variant; one `sync_blobs`
barrier runs before anything references the chunks, and the per-chunk
ref-count upserts collapse into a single
`INSERT ... SELECT unnest(...) ON CONFLICT` statement.
Durability contract is unchanged: every chunk is on stable storage
before the manifest row that references it commits. A crash mid-upload
leaves chunk files without DB rows — the same orphan class the
per-chunk scheme already produced, just a wider window.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.
Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every GET of a JPEG ≤5 MB with `Accept: image/webp` ran a full decode +
lossless-WebP encode on the rayon pool (hundreds of ms of CPU), found
the result LARGER than the original — the `image` crate's WebP encoder
is lossless-only, which practically never beats an already-lossy photo —
and threw the work away. Nothing recorded the verdict, so the next GET
of the same file repeated it, forever. A 20-photo gallery saturated the
transcode pool on every visit.
- `can_transcode` no longer accepts `image/jpeg` — photos skip the
pipeline entirely. PNG/GIF stay (lossless WebP genuinely shrinks them).
- Negative verdicts for the remaining formats are now remembered: an
empty-Bytes sentinel in the moka cache (expires with the 10-min TTL)
plus a zero-byte `.skip` marker on disk that survives restarts.
`invalidate()` removes the marker alongside the cached transcode.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Two DAV-surface fixes that replicate patterns the codebase already had:
NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.
WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Closes#323.
LoginLockoutService cached failed-attempt counters keyed only on
the username, so any caller that could reach the auth endpoint and
guess (or enumerate) a username could lock that account out for the
entire lockout window — the rate limiter happily lets each IP make
its share of bad-password attempts before clamping, which is enough
to trip the per-account threshold in seconds. The reporter
demonstrated a complete DOS by spoofing X-Forwarded-For with
OXICLOUD_TRUST_PROXY_HEADERS=true.
Fix: change the lockout cache key from `username` to `username|ip`.
A flood from one IP locks that IP out of that account, but a
legitimate user coming from a different IP is unaffected.
Changes:
- LoginLockoutService::{check, record_failure, record_success} take
client_ip as a second argument; cache key is built via Self::key
(`format!("{username}|{ip}")`).
- middleware/rate_limit.rs: factor out extract_client_ip_from_parts
(HeaderMap + Option<&SocketAddr>) so handlers that don't take a
full Request<B> can still derive the same client identifier
extract_client_ip uses. extract_client_ip now delegates to it.
- auth_handler.rs login: derive client_ip from headers (the only
signal available without ConnectInfo) and pass it through to all
three lockout calls.
- nextcloud/basic_auth_middleware.rs: do the same with the full
Request via extract_client_ip.
Tests:
- Updated existing 4 unit tests to thread an IP arg.
- New does_not_lock_out_other_ips_for_same_account: lock from IP1,
assert IP2 still allowed (the #323 regression).
- New success_resets_only_the_acting_ip: a successful login from
IP2 must NOT clear an attacker's lockout from IP1.
Verification:
- `cargo build` ✅
- `cargo test login_lockout` → 6 passed (4 existing thread an IP
arg without behaviour change, 2 new pin the per-IP scoping).
Signed-off-by: SAY-5 <say.apm35@gmail.com>
The main search endpoint took `limit: params.limit.unwrap_or(100)` with no
ceiling, and that value flows straight into the SQL LIMIT of
search_files_paginated / search_files_in_subtree. A client passing
?limit=<huge> would make Postgres return that many rows into memory and into
the result cache. The suggestions endpoint already clamps (.min(20)); search
did not.
Cap at MAX_SEARCH_LIMIT (500). total_count still reflects the full match set
(COUNT(*) OVER()), so deeper results stay reachable via offset.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
store_chunks bumped storage.blobs.ref_count once per chunk *occurrence* (it
looped over the full chunk list, duplicates included), but
remove_manifest_reference decrements once per *distinct* chunk
(WHERE hash = ANY(chunk_hashes) matches each row a single time). For any file
that repeats a chunk -- zero-filled regions in disk/VM images, repeated
document structures, concatenated archives -- storing added +N while deleting
removed -1, so the blob's ref_count never returned to 0 and the chunk was
never garbage-collected: a permanent storage leak.
Count per distinct chunk on the store side too, matching deletion. This also
makes it faster:
- existing chunks: one batched `UPDATE ... WHERE hash = ANY($1)` instead of
one UPDATE per occurrence;
- a brand-new chunk repeated within a file is read, uploaded and INSERTed
once instead of once per occurrence.
The manifest still stores the full per-occurrence chunk sequence (needed to
reassemble the file). Forward fix: blobs already over-counted by the old path
stay over-counted (a reconcile/verify pass could recompute them), but the
bias is upward (leak), so no data is ever deleted early.
CDC tests pass (12); fmt + clippy clean.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
The /api router added its own predicate-less CompressionLayer (routes.rs),
nested inside the global predicate-aware one in main.rs. As the inner layer it
compressed responses first, so the global predicate that skips already-
compressed media was bypassed for every /api response: video/audio/image/zip
downloads got Brotli-compressed (CPU + first-byte latency for ~0 bytes saved)
and lost their Content-Length (forced to chunked -> no client progress bar).
- Remove the redundant /api CompressionLayer; /api now flows through the
single global layer in main.rs.
- Make that predicate smarter: compress by default so nothing shrinkable is
missed, and skip ONLY already-compressed types. It no longer blanket-excludes
image/*, so image/svg+xml (text, ~70% shrink) now compresses; raster formats
are listed individually. Added the previously-missed already-compressed
types: Office (docx/xlsx/pptx), ODF, epub, jar, apk, 7z/rar/bzip2/zstd/xz,
woff/woff2 fonts, icons.
Net: media downloads keep Content-Length and skip pointless compression, while
text/JSON/JS/CSS/SVG/XML/ttf/otf/wasm still compress. fmt + clippy clean.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
StoragePath::join deep-cloned the whole Vec<String> (every segment String)
just to append one element. Take self by value and push in place. All callers
pass owned values except PathService::create_file_path, which holds a borrow
and now clones explicitly — the same copy the old &self join already made.
File::with_name / with_folder / with_size took &self and rebuilt the struct,
cloning every carried-over field (id, mime_type, folder_id, blob_hash, ...).
Consume self and mutate only the fields that change. Behaviour is identical;
the fallible builders now drop the input on Err, which is fine for these
rename/move/resize transforms (all current callers replace the file).
Impact is small in practice — with_folder/with_size have no callers and
with_name is test-only, while the one hot join caller (create_file_path)
must copy segments regardless — but the consuming form is the idiomatic one.
Verified: cargo fmt + clippy --all-features --all-targets -D warnings clean;
domain tests (path_service::, entities::file::) pass — 30 + 6.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
webdav encode_uri_path runs on every PROPFIND href and did
.map(...).collect::<Vec<_>>().join("/"), allocating a String per segment plus
a joined Vec. Write each utf8_percent_encode Display adapter straight into a
single preallocated String. Behavior is identical (split on '/', encode each
segment, join with '/'), including leading/trailing-slash edge cases.
subject_group list / list_with_counts each issued a second SELECT COUNT(*)
round-trip for the total. Fold it into the page query via COUNT(*) OVER() —
the pattern folder_db_repository already uses — halving the round-trips.
total_count is read from the first row and is 0 on an empty page, matching
folder_db_repository's documented convention.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
list_incoming_grants and list_grants_on_resource ran fetch_all with no
LIMIT, so a pathological number of grants on one resource (or targeting one
subject) would be pulled fully into memory. These back the grant-management
endpoints ("Manage sharing", "Shared with me"), not the hot require() path.
A blind LIMIT is unsafe here: apply_role reads the full grant set to compute
an add/remove diff, so a silently truncated list would be acted on as if
complete (stale grants never revoked). Instead fetch MAX_GRANT_ROWS + 1 and
reject with an audit line (authz.grant_list_rejected / reason=over_row_cap)
when the cap is exceeded, bounding worst-case RAM without ever returning a
partial set. The check is shared via PgAclEngine::guard_grant_row_cap. Cap is
10_000 — orders of magnitude above any realistic single-resource/subject
grant count.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
CachedBlobBackend held its single tokio::Mutex<LruCache> across filesystem
syscalls, serializing every concurrent cache operation behind one lock:
- get_blob_stream / get_blob_range_stream: held across File::open()/seek()
- delete_blob: held across remove_file()
- initialize: held across the full cache-dir walk
- eviction (insert + fetch paths): held across remove_file() loops
Now the lock only guards the in-memory LRU. Presence checks bump recency
and release the guard before touching the filesystem (a vanished file falls
through to the existing fetch-and-cache path, covering the race), and
eviction selects victims under the lock then unlinks them after releasing
it. The duplicated eviction loop is extracted into
CachedRef::collect_evictions.
db: set test_before_acquire(false). With warm min_connections and a bounded
max_lifetime, the liveness ping sqlx issues on every acquire() costs more
than the rare dead connection it catches; stale sockets surface as a query
error and the pool recycles them either way.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
a prefix path as been add to avoid collision if OXICLOUD_UPLOAD_TMPDIR=OXICLOUD_CHUNK_DIR
upgrade to this version will convert previous upload chunked into the prefixed version at server start
ensure files does not exeed OXICLOUD_MAX_UPLOAD_SIZE, prefer to deny from header rather consuming bandwidth
add OXICLOUD_DIRECT_PUT_MAX_BYTES for direct PUT (non chunked), admins can fine tune their prefered values
explain OXICLOUD_CHUNK_DIR and OXICLOUD_UPLOAD_TMPDIR
and also the OXICLOUD_CHUNK_MAX_BYTES & OXICLOUD_UPLOAD_TMPDIR
to help administratorrs to defined correctly their storage architecture
Edouard Vanbelle