Phase 0 of the delta-sync plan. Re-uploading a file the user already has
(another device, a restore, a duplicate) used to transfer every byte just
for the server to discard them as a dedup hit. The frontend now computes
the file's BLAKE3 locally and, on a hit, registers the file with a single
~150-byte metadata call.
Server — POST /api/files/by-hash:
- All checks live in the application service per the AuthZ rule:
Create permission on the target folder via the authorization engine,
hash ownership via the existing user-scoped query (a non-owned hash
returns 404 — same shape as "no such blob" — and emits an
instant_upload.rejected audit event), quota on the logical size.
- On success: one ref_count bump + the existing save_file_with_blob row
registration (compensation included); is_new_blob=false so lifecycle
hooks skip thumbnail regeneration. ~10 ms warm.
- The storage-usage service is now built before the application services
and injected, instead of only living on AppState.
Client — WASM BLAKE3 + worker:
- wasm/oxicloud-hash: the exact same blake3 crate the server uses,
compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match
server content addresses bit for bit. Built by scripts/build-wasm.sh;
the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no
npm dependencies, no wasm toolchain needed for regular builds.
- static/js/workers/hashWorker.js streams the File in 8 MiB slices off
the main thread (constant RAM at any file size).
- features/files/instantUpload.js orchestrates: threshold (8 MiB — below
it the round-trips cost more than the bytes), user-scoped
/api/dedup/check, by-hash registration, and silent fallback to the
normal byte upload on any miss, race or unsupported environment.
Wired into both uploadFiles and uploadFolderEntries.
- biome.json vendors exclusion fixed to cover nested directories
(previous vendors were .mjs and never matched the *.js include).
Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals
the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms
warm with a 151-byte request (vs 20,971,873 bytes for the byte upload);
the copy downloads byte-identical and the manifest ref_count goes 1→2;
a second user probing the same hash gets exists:false and 404 plus the
audit line; duplicate name → 409, malformed hash → 400; worker and wasm
are served with correct MIME (application/wasm).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Two follow-ups to the streaming-upload work:
The dedup garbage_collect() pass only ran when a user manually emptied
their trash. Zero-reference rows — chunks orphaned by aborted streaming
uploads (registered at ref_count 0 by the ingest rollback) and blobs
dereferenced by trash expiry itself — could linger indefinitely on
instances where nobody empties trash. The periodic TrashCleanupService
sweep now ends every run with garbage_collect() (maintenance pool,
batched), bounding orphan lifetime to the cleanup interval.
Folder-ZIP creation was strictly sequential: open blob stream, deflate,
close, repeat — every per-file blob-store round-trip (PG lookup + backend
open; a full HTTP round-trip on S3/Azure) added to the wall clock. It now
runs as a 2-stage pipeline: a prefetch task streams the planned files'
content ahead of the writer through a bounded channel (~4 MiB), so the
next file's read latency overlaps the current file's compression. ZIP
entries are still written strictly in order, peak RAM stays flat, and a
writer error hangs up the channel so the prefetcher stops on its own.
Verified end-to-end against PostgreSQL 16: an upload aborted at ~14 MB
left exactly 30 ref_count=0 chunk rows which the GC then reclaimed
(8.5 MB, rows + physical files); a chunked upload completed with a wrong
MD5 returned 400 with the tee-computed digest and the SAME session then
completed successfully with the right checksum (parts persist — the old
assembly deleted them, so the documented retry never actually worked);
a 4-file folder ZIP downloaded and extracted byte-identical.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
JWT validation cache now stores Arc<TokenClaims> and validate_token
returns Arc<TokenClaims>. On a cache hit — the 99% path for every
authenticated request — the moka lookup was deep-cloning the whole
claims struct (5 Strings: sub, jti, username, email, role) on every
call. It is now a refcount bump. Read-only callers (admin middleware)
go through Deref and allocate nothing; the auth middleware clones only
the three fields it moves into CurrentUser (was 5 clones, now 3), and
the admin paths clone only role (was 5, now 1). A new test asserts the
hit path returns a pointer-equal Arc.
TokenServicePort::validate_token is the single trait method touched;
its only implementor is JwtTokenService and the only production callers
are the auth and admin middleware (the WOPI handler uses a separate
WopiTokenService).
Dockerfile: rust:1.94.1-alpine3.23 -> rust:1.96-alpine3.24 and
alpine:3.23.3 -> alpine:3.24.0 for the runtime stage.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Files uploaded before chunk_manifests landed (20260414000000) are stored
as ONE whole-file blob with no manifest. Every legacy fallback in
DedupService exists to serve them, and the cost concentrates on Range
reads: with encryption enabled, seeking inside a legacy video decrypts
the ENTIRE blob (AES-GCM is all-or-nothing), where a CDC file decrypts
only the overlapping <=1 MiB chunks.
This adds a one-time, idempotent background migration (spawned from the
composition root after dedup init, maintenance pool) that converts each
legacy blob into a regular CDC file, indistinguishable from a native
upload:
1. Spool the blob through the normal read path (decrypts when
encryption is on) to a per-attempt-unique temp file, verifying
BLAKE3 == hash; sizes come from the verified spool, never from the
legacy storage.blobs.size column (the manifest's total_size drives
Range arithmetic).
2. CDC-chunk + store chunks via the existing store_chunks (one
manifest reference per distinct chunk).
3. One short accounting TX with the blob row locked: manifest INSERT
with ref_count = N current file references, blob ref_count -= N,
row deleted only at exactly 0 - so single-chunk files (chunk hash
== file hash) keep the physical blob, which IS the chunk; only
bookkeeping moves, no bytes are rewritten.
4. Physical whole-file blob deleted only when its row dropped.
Races lean on the row lock: a concurrent identical upload landing a
legacy reference after commit keeps the blob row alive and that file
readable via the fallback (bounded space leak, never data loss); a
crash between chunk store and the TX over-counts one file's chunk refs
(also a bounded leak). Corrupt blobs (content != hash) are logged,
counted, excluded from the sweep and left untouched, with a hard cap
before aborting.
Per-hash failures never block the sweep; manifests are the resumability
marker, so a restart continues where it left off. The legacy read/write
fallbacks stay in place as the safety net while a deployment converges;
they can be deleted once fleets report "legacy re-chunk: nothing to do".
Opt-out via OXICLOUD_LEGACY_RECHUNK=false (documented in example.env)
for metered remote backends where the one-time re-read should be
scheduled deliberately.
Covered by five integration tests against real PostgreSQL (multi-chunk
accounting + Range across a chunk boundary, single-chunk physical-blob
preservation, corrupt-blob isolation, empty blob, and the full
encrypted-backend roundtrip); they run concurrently, which also
exercises the cross-sweep race handling.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Three targeted frontend fixes:
1. pdf.js smart preload (thumbnail.js, resourceIcon.js). The first PDF
thumbnail of a session stalled 1-2s on the lazy import of the ~1.3 MB
pdf.js stack. buildResourceIcon() now fires thumbnail.preloadPdf()
the moment a PDF row enters the DOM, warming both the module
(~300 KB, via the now promise-memoized getPdfjsLib, shared with real
users) and the worker script (~1 MB, via a cache-priming fetch —
pdf.js only requests it on first getDocument). Only folders that
actually contain PDFs pay the download; idempotent after first call,
resets on failure so transient offline retries.
2. Admin migration poll cleanup (admin.js). The 2s setInterval kept
hitting the API and updating hidden DOM after leaving the Storage
tab, and polled a failing endpoint forever after session expiry
(!resp.ok returned without clearing). New stopMigrationPolling()
helper, invoked on tab switch away from Storage, on non-running
status, and on failed polls; tab re-entry re-arms via loadStorage().
3. resourceList.js render hoisting. Per-row i18n.t() type-cell lookups
and the fully item-invariant _renderCustomActions() HTML were
recomputed for every row; they now resolve once per batch via
_buildItemLabels() (per-category labels memoized, rebuilt each
batch so locale switches keep working). _findLaneByKey() swaps the
container-wide attribute querySelector for an O(1) _lanes Map kept
in sync at the only lane create/wipe sites.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Two hot-path fixes for DAV sync clients, which poll continuously:
1. App-password Basic Auth cache TTL 30s → 300s. Every cache miss costs
a full Argon2id verification (~50-100ms CPU) plus two DB queries; at
30s a continuously-syncing client re-paid that every 30s. 5 min cuts
it ~10x. Safe because revoke() already invalidates the user's cached
entries immediately; expiry/deactivation are only re-checked on a
miss, so they now have a <=5 min grace window (comparable to a JWT
access-token lifetime) — documented on the constant.
2. WebDAV lock store: replace the tokio::spawn + sleep scheduled per
acquire/refresh with Moka's per-entry Expiry policy. Office clients
refresh locks constantly, leaving thousands of orphaned sleeping
timers pinned in the runtime that were never cancelled. by_path now
carries the exact per-lock TTL via LockExpiry (no background tasks);
by_token keeps a 24h backstop and resolves through by_path, so a
lingering reverse-index entry can never resurrect an expired lock.
Adds unit tests for the expiry policy and acquire/refresh/release
(the module had none).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.
- Add find_events_by_ical_uids / get_contacts_by_uids through every
layer (domain repo trait → PG repo → storage port/adapter → use case
→ handler) using one indexed `= ANY($2)` query, mirroring the
existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
uid_from_multiget_href helper (percent-decoding, case-insensitive
extension strip, collection hrefs rejected). Exact UID matching also
removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
GET /api/address-books/{id}/contacts: optional ListQuery fields
thread through ContactUseCase::list_contacts to a paginated repo
query. Omitted params keep returning the full book (frontend and
DAV listing/sync paths unchanged).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
The read repository's hash_cache assumed blob hashes were immutable
per file_id, but update_file_content_from_temp remaps the SAME row to
a new hash via swap_blob_hash. After a WebDAV/NC PUT overwrite,
streaming downloads (>=10 MB tier, Range requests, video playback)
kept resolving the OLD blob for the 30 s TTI window — and because
every read refreshes the TTI, a polling client could pin the stale
mapping indefinitely, eventually turning into 500s once the old
blob's refcount hit zero and GC removed it.
The write repository now holds a shared handle to the same moka cache
(clones share storage) and invalidates the entry right after every
content swap and hard delete commits — covering every present and
future caller of the write port rather than one service path. Stale
doc comments claiming the mapping was immutable (and SHA-256) fixed.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Request-path thumbnail generation (REST get_thumbnail and NC preview)
read the full source blob into RAM and decoded it with no concurrency
bound — a first-view gallery of K images stacked K full-size buffers
and K parallel decodes on the blocking pool. The background hook had
the inverse ordering problem: it read the blob eagerly and only then
queued on the decode semaphore, so N concurrent uploads held N
originals in memory while waiting.
All three paths now acquire the decode semaphore first and read the
blob under the permit, capping peak RAM at permits x image size:
- new ThumbnailService::get_thumbnail_from_blob defers the blob read
into the moka init closure (cache and disk hits never touch the
blob); both handlers use it and no longer pre-read.
- get_thumbnail_from_bytes (direct-bytes variant) now also takes a
permit before decoding; shared generate_and_persist core keeps the
two entrypoints duplicate-free.
- generate_all_sizes_background_from_blob (renamed from _from_bytes)
checks blob existence and disk-dedup state, then acquires the
permit, then reads; the redundant pre-read spawn wrapper in
ThumbnailRefreshHook is gone.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Replace the per-row tree_modified_at bump triggers with AFTER ... FOR
EACH STATEMENT triggers using transition tables: each DML statement now
pays one bump covering the distinct ancestor chains of all affected
rows (locked in id order so concurrent bumps over overlapping chains
cannot deadlock), instead of one chain UPDATE per affected row.
Value-based change detection replaces UPDATE OF column lists (PG
forbids those with transition tables), so EXIF media_sort_date syncs
and no-op updates no longer bump at all, and file moves now invalidate
the source chain as well as the destination.
Also fix a latent bug surfaced while testing this: the descendant
path/lpath cascade (trg_folders_cascade_path) was declared AFTER
UPDATE OF path, lpath, but rename/move statements SET name/parent_id,
and BEFORE-trigger rewrites do not count for UPDATE OF — the cascade
never fired, leaving every descendant folder with a stale path/lpath
after any rename or move. The migration canonically repairs existing
trees and re-creates the cascade on the columns the app actually
writes.
delete_expired_bulk now deletes in LIMIT-ed batches (1000 files / 100
folders per round, each its own implicit transaction) ordered by
trashed_at and served by new partial indexes over trashed rows, so
retention purges no longer hold one unbounded transaction.
Verified against a local PG16 cluster: all migrations apply from
scratch, the new one re-runs idempotently, and a 16-case behavioral
battery passes (chain bumps for insert/update/delete/move/trash, EXIF
invisibility, no-op invisibility, cascade repair, FK-cascade depth
guards, batched purge shape and partial-index plan).
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
The five FileLifecycleHook call sites handed dto.etag (format
"{blob_hash[..16]}-{modified_at}") to the blob_hash parameter. Blob
lookups keyed by that value can never resolve, so background thumbnail
generation and audio metadata extraction silently failed on every
upload/copy/update, pushing all thumbnail work onto the request path.
Pass dto.content_hash — the raw full BLAKE3 hash already carried by
FileDto — instead.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Same whitespace-only reformat as branch claude/jolly-johnson-yso7z7:
PR #326 landed three files that fail cargo fmt --check and its CI run
skipped the Rustfmt job, breaking the check for every later Rust PR.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Resolves the four-file conflict with PR #444, which landed a parallel
implementation of the same batch-fsync design (unsynced chunk writes plus
one sync_blobs durability barrier). Both sides converged on identical
trait signatures; main's version is kept for all overlapping code since
it is already merged and carries unit tests:
- blob_storage_ports.rs: doc-comment-only conflict, main's docs kept
- dedup_service.rs: main's pipeline kept. The naive textual merge would
have retained BOTH durability-barrier/INSERT blocks, double-counting
ref_count for every new chunk; the duplicated block from this branch
is dropped
- encrypted_blob_backend.rs, local_blob_backend.rs: main's
implementations kept (fsync_paths_parallel + tests); this branch's
equivalent helpers (fsync_dir, fsync_blob_file, SYNC_BLOBS_CONCURRENCY)
are referenced nowhere else and are dropped
Everything unique to this branch (streaming PROPFIND, range requests,
batch ID resolution, CalDAV/CardDAV indexed lookups, transcode and
breadcrumb changes) merges cleanly and is unchanged.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Every folder navigation awaited rebuildBreadCrumb() — one sequential
fetch per ancestor level — before requesting the first page, so a
depth-8 folder paid ~8 RTTs of dead time before the content even
started loading. And nothing was cached: navigating between sibling
folders re-fetched the same ancestors every time.
- filesModel: session cache id → FolderItem for breadcrumb resolution
(warm navigations rebuild the trail with zero fetches), invalidated
on rename/move (single + batch) via invalidateFolderMeta. The trail
is now built locally and committed atomically, guarded by a
generation token so a rebuild superseded by a faster follow-up
navigation can no longer interleave writes into the newer trail.
- filesView.loadFiles: the rebuild runs concurrently with the first
page fetch; crumbs, history and title update when it resolves.
Navigation latency becomes max(listing, ancestor chain) instead of
their sum — and ~equal to the listing alone once the cache is warm.
The home-folder fallback for inaccessible targets is preserved by
reloading the listing when the rebuild had to reset app.currentPath.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Storing a new file through CDC dedup issued sync_all + a parent-dir
fsync for every ~256 KB chunk (~8,200 fsyncs for a 1 GB upload), plus
one PG INSERT round-trip per chunk. The actual durability boundary is
the manifest INSERT: chunks only need to be durable before any PG row
references them, not one by one.
- BlobStorageBackend grows put_blob_from_bytes_unsynced + sync_blobs
with conservative defaults (unsynced delegates to the synced write,
sync_blobs is a no-op) so backends that don't opt in keep the
per-write durability semantics. Remote stores are durable on PUT.
- LocalBlobBackend writes chunks without fsync and implements
sync_blobs as a parallel sweep: every listed blob file (hard
requirement) plus each distinct prefix directory exactly once
(best-effort, same tier as fsync_parent_dir).
- DedupService::store_chunks writes new chunks unsynced, runs one
sync_blobs sweep, then registers all new chunks in ONE batched
UNNEST INSERT - durability before visibility, and the per-chunk PG
round-trips collapse into one.
- Encrypted/Migration decorators forward both methods so the
optimization survives encrypted-local and live-migration stacks.
https://claude.ai/code/session_013Bk4BMQEvR9QxCU7QXLRwv
Creating a grant for a group dispatched the notification emails one at
a time inside the HTTP request — 30 members × ~500 ms of SMTP ≈ 15 s
holding the POST /api/grants response (the code carried a TODO
acknowledging it).
Dispatches are independent (coalescing and rate-limiting key on the
(granter, recipient) pair, distinct per member), so run them through
`buffered(6)`: ~6× less wall time for group fan-outs while capping
parallel SMTP sessions, with outcome order still matching member order.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The CDC chunk path paid `sync_all` + parent-dir fsync + one ref-count
upsert round-trip PER ~256 KB chunk: a new 1 GB file ≈ 8 000 fsyncs +
4 000 sequential INSERTs on the upload critical path (seconds on SSD,
tens of seconds on HDD).
- New `put_blob_from_bytes_unsynced` + `sync_blobs` on
BlobStorageBackend. Defaults delegate to the durable variants, so
remote backends (S3/Azure — durable on PUT) are untouched.
LocalBlobBackend writes chunks without fsync and `sync_blobs` then
fsyncs all files concurrently (kernel coalesces the writeback) plus
one fsync per DISTINCT shard directory — at most 256 — instead of one
per chunk. EncryptedBlobBackend delegates both so encryption over the
local backend keeps the optimization.
- store_chunks: uploads use the unsynced variant; one `sync_blobs`
barrier runs before anything references the chunks, and the per-chunk
ref-count upserts collapse into a single
`INSERT ... SELECT unnest(...) ON CONFLICT` statement.
Durability contract is unchanged: every chunk is on stable storage
before the manifest row that references it commits. A crash mid-upload
leaves chunk files without DB rows — the same orphan class the
per-chunk scheme already produced, just a wider window.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.
Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every GET of a JPEG ≤5 MB with `Accept: image/webp` ran a full decode +
lossless-WebP encode on the rayon pool (hundreds of ms of CPU), found
the result LARGER than the original — the `image` crate's WebP encoder
is lossless-only, which practically never beats an already-lossy photo —
and threw the work away. Nothing recorded the verdict, so the next GET
of the same file repeated it, forever. A 20-photo gallery saturated the
transcode pool on every visit.
- `can_transcode` no longer accepts `image/jpeg` — photos skip the
pipeline entirely. PNG/GIF stay (lossless WebP genuinely shrinks them).
- Negative verdicts for the remaining formats are now remembered: an
empty-Bytes sentinel in the moka cache (expires with the 10-min TTL)
plus a zero-byte `.skip` marker on disk that survives restarts.
`invalidate()` removes the marker alongside the cached transcode.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Two DAV-surface fixes that replicate patterns the codebase already had:
NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.
WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Four related fixes to stop buffering files in page memory and stop
hammering layout from the selection paths:
- Inline viewer media: video/audio fetched the ENTIRE file into a blob
before the first frame (a 2 GB video = 2 GB of tab heap, no
progressive playback, no seek). The <video>/<audio> src now points
straight at the same-origin API URL — cookies travel automatically
and the browser streams with native Range requests, exactly like the
music player already did.
- Downloads (file, folder ZIP, batch ZIP, viewer button): the
fetch → blob → objectURL pattern materialized the whole payload in
RAM before the save dialog appeared (a 10 GB batch ZIP risked
crashing the tab) with no download progress UI. New shared
utils/download.js hands the URL to the browser, which streams to
disk with its own progress UI. Batch download uses the existing GET
endpoint (same URL contract as the drag-out DownloadURL, now shared
via buildBatchDownloadUrl); selections whose id list cannot fit in a
URL keep the buffered POST fallback. Trade-off: failed downloads now
surface in the browser's download shelf instead of an in-app toast.
- Rubber-band lasso: every mousemove (>100/s) walked all cards
interleaving getBoundingClientRect() reads with class writes — up to
N forced reflows per event, freezing the frame rate on folders with
thousands of loaded rows. Card rects (+ item info) are now snapshot
once per drag (rebuilt on scroll), and a single rAF pass per frame
compares against the cached geometry, touching only cards whose
selection state changed.
- batchToolbar: getSelection() rebuilt the selection by scanning every
.file-item in the document (the TODO admitted it); it now reads the
_selected Map that every selection path already keeps in sync.
clear() scopes its DOM sweep to #files-list (the only container the
toolbar manages) instead of the whole document.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The global childList observer scheduled replaceIconsInElement() with no
scope on ANY node insertion — including bare text nodes — so every
notification-bell progress tick during an upload and every
infinite-scroll batch re-scanned the whole document with an attribute
substring selector. Cost grew with total DOM size (5-10k nodes), not
with what was inserted.
Queue the added element roots per animation frame and scan just those
subtrees (an inserted <i> itself is caught via its parent). Text-node
churn no longer triggers any scan at all. The icon replacement is
idempotent, so overlapping roots are harmless.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
_bindItemEvents attached direct click listeners to the favorite star,
the shared badge and every custom action button of every row — ~3+
listeners per row on top of the container delegation the component
already wires, i.e. ~10-15k listeners with a few thousand rows loaded
via load-more, plus the per-row binding cost on every render.
The container's delegated click handler already dispatches by
closest() checks; move the three button behaviours there (they return
before the card-open branch, preserving the old stopPropagation
semantics) and drop _bindItemEvents entirely. Row creation is now
pure innerHTML — no listener allocation per row.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
displaySearchResults inserted up to 100 results one at a time via
addItem({ scroll: true, highlight: true }) — each insert ran two DOM
scans (duplicate guard + row lookup), a smooth scrollIntoView and a
highlight pulse, so every live-search keystroke triggered ~100 competing
smooth-scrolls and O(n²) container scans.
Add filesView.renderItems(): a single component.render() pass (one
DocumentFragment, zero per-item scans/scrolls). addItem keeps the
scroll/highlight affordances for its documented purpose — optimistic
single-item inserts after upload/create.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
`uploadFiles` sent files one at a time (await per file), so dropping N
files meant N sequential round-trips. Reuse the bounded-concurrency pool
already proven in `uploadFolderEntries` (CONCURRENCY = 10): independent
files now upload up to 10 at a time — ~10x faster for many small files.
Per-file progress (XHR → bell), the legacy progress bar, timeout
notifications and the quota short-circuit are preserved. On a quota error
the pool stops pulling new files while in-flight uploads finish (matching
`uploadFolderEntries`) instead of the old hard `break`.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The entrypoint ran "chown -R" on the storage volume on every container
start. Storage is a content-addressable blob store that can hold millions
of objects, so a blind recursive chown re-stats and rewrites the inode of
every blob on each boot, adding minutes of startup time and saturating the
disk on spinning media.
Guard the chown behind a top-level ownership check: only recurse when the
directory root is not already owned by the oxicloud user. The first boot
fixes a freshly mounted (root-owned) volume; every later boot is a no-op.
The same guard is applied to the static dir, factored into a shared helper.
The target UID is resolved via "id -u oxicloud" instead of hardcoding 1001.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Deleting a multi-photo selection fired one `DELETE /api/files/{id}` per
photo in series — 300 selected photos meant 300 sequential round-trips
with the UI blocked on `confirm()`. Switch to `POST /api/batch/trash`
(already used by the file view), chunked to the backend's MAX_BATCH_SIZE
of 1000, so a selection of any size collapses to ceil(N/1000) requests.
Only items the server reports as successfully trashed are removed from
the grid; the selection bar refreshes to reflect any failures.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The .dockerignore excluded a non-existent doc/ directory (the real one is
docs/) and did not exclude tests/, images/, charts/, or tools/. None of
these are COPYed by the Dockerfile, yet ~24 MB of them were sent to the
daemon as build context on every build. Fix the typo and exclude the
unused asset/tooling directories so the context stays minimal.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
sqlx's postgres driver implements the wire protocol in pure Rust and TLS
goes through rustls, so libpq is never linked. Confirmed via Cargo.lock:
no pq-sys, native-tls, or openssl-sys in the dependency tree.
Removing postgresql-dev from the build base and libpq from the runtime
image shrinks the final image by ~3 MB and removes a C library from the
attack/patch surface. perl/make/gcc/musl-dev are kept for the C builds of
aws-lc-sys.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Closes#323.
LoginLockoutService cached failed-attempt counters keyed only on
the username, so any caller that could reach the auth endpoint and
guess (or enumerate) a username could lock that account out for the
entire lockout window — the rate limiter happily lets each IP make
its share of bad-password attempts before clamping, which is enough
to trip the per-account threshold in seconds. The reporter
demonstrated a complete DOS by spoofing X-Forwarded-For with
OXICLOUD_TRUST_PROXY_HEADERS=true.
Fix: change the lockout cache key from `username` to `username|ip`.
A flood from one IP locks that IP out of that account, but a
legitimate user coming from a different IP is unaffected.
Changes:
- LoginLockoutService::{check, record_failure, record_success} take
client_ip as a second argument; cache key is built via Self::key
(`format!("{username}|{ip}")`).
- middleware/rate_limit.rs: factor out extract_client_ip_from_parts
(HeaderMap + Option<&SocketAddr>) so handlers that don't take a
full Request<B> can still derive the same client identifier
extract_client_ip uses. extract_client_ip now delegates to it.
- auth_handler.rs login: derive client_ip from headers (the only
signal available without ConnectInfo) and pass it through to all
three lockout calls.
- nextcloud/basic_auth_middleware.rs: do the same with the full
Request via extract_client_ip.
Tests:
- Updated existing 4 unit tests to thread an IP arg.
- New does_not_lock_out_other_ips_for_same_account: lock from IP1,
assert IP2 still allowed (the #323 regression).
- New success_resets_only_the_acting_ip: a successful login from
IP2 must NOT clear an attacker's lockout from IP1.
Verification:
- `cargo build` ✅
- `cargo test login_lockout` → 6 passed (4 existing thread an IP
arg without behaviour change, 2 new pin the per-IP scoping).
Signed-off-by: SAY-5 <say.apm35@gmail.com>
The main search endpoint took `limit: params.limit.unwrap_or(100)` with no
ceiling, and that value flows straight into the SQL LIMIT of
search_files_paginated / search_files_in_subtree. A client passing
?limit=<huge> would make Postgres return that many rows into memory and into
the result cache. The suggestions endpoint already clamps (.min(20)); search
did not.
Cap at MAX_SEARCH_LIMIT (500). total_count still reflects the full match set
(COUNT(*) OVER()), so deeper results stay reachable via offset.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
store_chunks bumped storage.blobs.ref_count once per chunk *occurrence* (it
looped over the full chunk list, duplicates included), but
remove_manifest_reference decrements once per *distinct* chunk
(WHERE hash = ANY(chunk_hashes) matches each row a single time). For any file
that repeats a chunk -- zero-filled regions in disk/VM images, repeated
document structures, concatenated archives -- storing added +N while deleting
removed -1, so the blob's ref_count never returned to 0 and the chunk was
never garbage-collected: a permanent storage leak.
Count per distinct chunk on the store side too, matching deletion. This also
makes it faster:
- existing chunks: one batched `UPDATE ... WHERE hash = ANY($1)` instead of
one UPDATE per occurrence;
- a brand-new chunk repeated within a file is read, uploaded and INSERTed
once instead of once per occurrence.
The manifest still stores the full per-occurrence chunk sequence (needed to
reassemble the file). Forward fix: blobs already over-counted by the old path
stay over-counted (a reconcile/verify pass could recompute them), but the
bias is upward (leak), so no data is ever deleted early.
CDC tests pass (12); fmt + clippy clean.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
The /api router added its own predicate-less CompressionLayer (routes.rs),
nested inside the global predicate-aware one in main.rs. As the inner layer it
compressed responses first, so the global predicate that skips already-
compressed media was bypassed for every /api response: video/audio/image/zip
downloads got Brotli-compressed (CPU + first-byte latency for ~0 bytes saved)
and lost their Content-Length (forced to chunked -> no client progress bar).
- Remove the redundant /api CompressionLayer; /api now flows through the
single global layer in main.rs.
- Make that predicate smarter: compress by default so nothing shrinkable is
missed, and skip ONLY already-compressed types. It no longer blanket-excludes
image/*, so image/svg+xml (text, ~70% shrink) now compresses; raster formats
are listed individually. Added the previously-missed already-compressed
types: Office (docx/xlsx/pptx), ODF, epub, jar, apk, 7z/rar/bzip2/zstd/xz,
woff/woff2 fonts, icons.
Net: media downloads keep Content-Length and skip pointless compression, while
text/JSON/JS/CSS/SVG/XML/ttf/otf/wasm still compress. fmt + clippy clean.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
StoragePath::join deep-cloned the whole Vec<String> (every segment String)
just to append one element. Take self by value and push in place. All callers
pass owned values except PathService::create_file_path, which holds a borrow
and now clones explicitly — the same copy the old &self join already made.
File::with_name / with_folder / with_size took &self and rebuilt the struct,
cloning every carried-over field (id, mime_type, folder_id, blob_hash, ...).
Consume self and mutate only the fields that change. Behaviour is identical;
the fallible builders now drop the input on Err, which is fine for these
rename/move/resize transforms (all current callers replace the file).
Impact is small in practice — with_folder/with_size have no callers and
with_name is test-only, while the one hot join caller (create_file_path)
must copy segments regardless — but the consuming form is the idiomatic one.
Verified: cargo fmt + clippy --all-features --all-targets -D warnings clean;
domain tests (path_service::, entities::file::) pass — 30 + 6.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
webdav encode_uri_path runs on every PROPFIND href and did
.map(...).collect::<Vec<_>>().join("/"), allocating a String per segment plus
a joined Vec. Write each utf8_percent_encode Display adapter straight into a
single preallocated String. Behavior is identical (split on '/', encode each
segment, join with '/'), including leading/trailing-slash edge cases.
subject_group list / list_with_counts each issued a second SELECT COUNT(*)
round-trip for the total. Fold it into the page query via COUNT(*) OVER() —
the pattern folder_db_repository already uses — halving the round-trips.
total_count is read from the first row and is 0 on an empty page, matching
folder_db_repository's documented convention.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
list_incoming_grants and list_grants_on_resource ran fetch_all with no
LIMIT, so a pathological number of grants on one resource (or targeting one
subject) would be pulled fully into memory. These back the grant-management
endpoints ("Manage sharing", "Shared with me"), not the hot require() path.
A blind LIMIT is unsafe here: apply_role reads the full grant set to compute
an add/remove diff, so a silently truncated list would be acted on as if
complete (stale grants never revoked). Instead fetch MAX_GRANT_ROWS + 1 and
reject with an audit line (authz.grant_list_rejected / reason=over_row_cap)
when the cap is exceeded, bounding worst-case RAM without ever returning a
partial set. The check is shared via PgAclEngine::guard_grant_row_cap. Cap is
10_000 — orders of magnitude above any realistic single-resource/subject
grant count.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
Claude