Backend — Photos timeline now groups by real capture date instead of upload time. New MediaMetadataService (FileLifecycleHook) extracts EXIF DateTimeOriginal from images and container creation_time from videos (mov/mp4/mkv) via nom-exif, timezone-correct (OffsetTimeOriginal), persisting captured_at so the existing media_sort_date trigger takes over. Adds POST /admin/photos/metadata/reextract to backfill existing media. Falls back to upload date when no embedded date exists.
Frontend — premium grid cards: combined metadata line (relative date · size, owner avatar when shared), custom selection checkbox with a clear checked state, uniform full-width 4:3 thumbnail tiles independent of filename length, centered file-type icons, and a hit-test fix so checkbox/star/kebab clicks reach the controls (the decorative thumbnail no longer captures pointer events). Notification messages internationalised across all 16 locales. Broader polish: design tokens, a11y/focus-visible states, brand + PWA assets.
Chore — bump semver-compatible dependencies (cargo upgrade); add nom-exif 3.6.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both sides added a parameter to create_application_services and a
setup step before it: this branch's storage-usage/quota service (for
the instant-upload path) and main's Tantivy content index (for
SearchService). The resolution keeps both — the signature takes both
arguments and the build runs storage usage as step 3c and the content
index as 3d.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 3 of the delta-sync plan — the inverse direction, so a future
client app holding an older local version can fetch only what changed:
- GET /api/files/{id}/manifest returns the file's chunk recipe
({file_hash, total_size, chunks}). Owner-scoped like the rest of the
delta surface (Read permission through the authz engine first, then
the chunk layer's possession standard; shared files use the regular
download endpoints). A manifest is immutable for a given file_hash,
so it is served with ETag = file_hash and If-None-Match answers 304 —
polling sync clients pay one header round-trip per unchanged file.
Legacy pre-CDC blobs are presented as a single-chunk manifest of
themselves, so clients need no special case.
- POST /api/files/delta/download streams the requested chunks as
[u32 BE length][bytes] frames in request order — the same wire format
the upload direction uses. Entitlement is the same possession rule as
negotiate/commit (chunks reachable through the caller's own files);
anything else returns 404 {not_available} — deliberately
indistinguishable from "never existed" — with a
delta_download.rejected audit event. Batches are bounded by the
chunk_max_bytes budget; Content-Length is exact (sizes come from the
dedup index) and peak RAM is one backend read frame.
Both endpoints share the delta rate limiter. New DedupService
primitives: manifest_chunk_list (with legacy fallback), chunk_sizes,
chunk_stream. OpenAPI regenerated; protocol doc gains the download
section; types.js maps the new wire shapes (plus the delta-upload
typedefs that a container reset had silently dropped from a previous
commit).
Verified end-to-end against PostgreSQL 16 with a simulated two-device
sync: device A uploaded 24 MB by bytes and delta-updated it (2 edits →
2 chunks); device B diffed the manifest against its WASM-chunked local
copy, needed 2/79 chunks, fetched 970 KB instead of 24 MB (96.1%
saved) and rebuilt the file byte-identical with the BLAKE3 verifying.
If-None-Match revalidation returned 304; a second user got 404 on both
the manifest and the chunk batch (with the not_available list and
audit lines); an unknown hash was indistinguishable from a denied one;
an empty hash list returned 400.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 2 — the client side of "upload only what changed", closing the
delta-sync plan.
WASM (wasm/oxicloud-hash): DeltaChunker adds incremental FastCDC with
the server's exact crate and parameters (64K/256K/1M) next to the BLAKE3
hasher. The incremental split is provably identical to a single pass:
every chunk except the last ends on a content/max-size condition whose
decision window was fully buffered, so only the tail is provisional and
re-examined as slices arrive. A mirror test — the client twin of the
server's stream≡slice test — chunks 4 MiB of xorshift noise with
adversarial slice sizes (7 B … 8 MiB) and requires boundary-for-boundary
equality with one FastCDC pass. Vendored artifacts rebuilt (55 KB wasm).
Worker (static/js/workers/deltaWorker.js): the full protocol off the
main thread with OVERLAPPED stages — 8 MiB file slices feed the chunker
while earlier batches (256 hashes) negotiate and their missing chunks
upload through a 2-deep PUT pool (≤8 MiB framed bodies, bytes re-sliced
from the File at send time, never hoarded). Commit handles 409
still_missing by uploading exactly the named hashes and retrying.
Orchestrator (features/files/deltaUpload.js): threshold (8 MiB),
worker lifecycle + size-scaled timeout, progress relay to the upload
bell, conclusive-outcome mapping (201/200, 507 quota, 409 name
conflict) and silent fallback to the byte upload for everything else.
Wired into uploadFiles and uploadFolderEntries, which now surface one
batch summary of the bytes dedup saved. This subsumes the whole-file
instant-upload module — a fully-known file negotiates to nothing
missing and the commit short-circuits on possession — so
instantUpload.js and hashWorker.js are removed (the /api/dedup/check
and /api/files/by-hash endpoints remain for API clients).
Verified end-to-end against PostgreSQL 16 — the cross-boundary proof
the whole design hangs on, in both directions: a 24 MB file byte-
uploaded (server-side CDC) then edited and delta-negotiated with
WASM-computed chunks reported missing 1/74 (boundaries bit-identical),
synced with 344 KB on the wire vs 24 MB (98.6% saved) and downloaded
byte-identical; inversely, a file created via delta then byte-uploaded
as identical content produced a server-side manifest DEDUP HIT with the
same content_hash. Insertion at the head of the file (the adversarial
CDC case) still negotiated missing 1/74. Chunk+hash throughput ≈275 MB/s
in V8 with SIMD128.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.
Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
(storage.search_index_dirty) - every write surface (REST, WebDAV,
NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
batched single-writer Tantivy commits; queue rows are deleted only
after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
reseeds it from Postgres, which remains the single source of truth
SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.
The frontend renders the snippet under the file name in list view.
New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.
https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
Phase 1 of the delta-sync plan, server side. The CDC store already shares
unchanged chunks between file versions after the bytes arrive; these three
stateless endpoints move that detection to the client, so editing a few
bytes of a large file uploads ~1 MiB instead of the whole file:
- POST /api/files/delta/negotiate — given the file's chunk hashes, answer
which ones the caller must upload. User-scoped and purely advisory.
- PUT /api/files/delta/chunks — missing chunks as [u32 BE len][bytes]
frames (streaming parse, ≤1 MiB per frame, chunk_max_bytes per request).
Every hash is recomputed server-side; chunks land as ref_count=0 orphans
that a commit pins or the periodic GC sweeps — no session table.
- POST /api/files/delta/commit — pin one reference per distinct chunk with
a single UPDATE…RETURNING restricted to chunks the caller is entitled
to (reachable through their own non-trashed files, or unreferenced
orphans); anything else returns 409 {still_missing} for the client to
upload and retry. The pinned sequence is then RE-READ and the whole-file
BLAKE3 recomputed before any manifest exists — a declared file_hash is
never trusted, because a forged manifest would poison future whole-file
dedup hits for other users. The manifest accounting is shared with the
byte path (attach_manifest, extracted from store_from_stream); the file
row is created (201) or its content swapped by file_id (200). Owners of
the exact file_hash short-circuit to a pure reference bump.
Supporting pieces: GIN index on chunk_manifests.chunk_hashes (containment
probes were sequential scans), claimable/pin/release/store-loose/verify
primitives on DedupService, update-by-id with Update-permission AuthZ on
FileUploadService, per-caller rate limiter (240/min), audit events with
stable reasons (rate_limited, chunk_verification_failed,
file_hash_mismatch), OpenAPI + docs/delta-upload-protocol.md, framing
parser unit tests and a PG-gated integration suite covering the
entitlement matrix (owned/foreign/orphan/unknown), orphan registration
and the verification read.
Verified end-to-end against PostgreSQL 16 with a node client hashing via
the vendored WASM: a 24 MB file delta-committed in 96 fixed-size chunks;
a 3-byte edit then negotiated missing 1/96 and synced with 278 KB on the
wire vs 24 MB (98.9% saved), downloading byte-identical. A second user
probing the same chunks got nothing (negotiate: all missing; commit: 409
with all 96 still withheld); a forged file_hash returned 400 plus the
audit line; a commit referencing one never-uploaded chunk returned 409
naming exactly that hash; the GIN index serves containment probes
(Bitmap Index Scan) once the planner favors it.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Phase 0 of the delta-sync plan. Re-uploading a file the user already has
(another device, a restore, a duplicate) used to transfer every byte just
for the server to discard them as a dedup hit. The frontend now computes
the file's BLAKE3 locally and, on a hit, registers the file with a single
~150-byte metadata call.
Server — POST /api/files/by-hash:
- All checks live in the application service per the AuthZ rule:
Create permission on the target folder via the authorization engine,
hash ownership via the existing user-scoped query (a non-owned hash
returns 404 — same shape as "no such blob" — and emits an
instant_upload.rejected audit event), quota on the logical size.
- On success: one ref_count bump + the existing save_file_with_blob row
registration (compensation included); is_new_blob=false so lifecycle
hooks skip thumbnail regeneration. ~10 ms warm.
- The storage-usage service is now built before the application services
and injected, instead of only living on AppState.
Client — WASM BLAKE3 + worker:
- wasm/oxicloud-hash: the exact same blake3 crate the server uses,
compiled with WASM SIMD128 (~660 MB/s measured) so browser hashes match
server content addresses bit for bit. Built by scripts/build-wasm.sh;
the artifacts (45 KB wasm + 8 KB glue) are vendored like pdf.js — no
npm dependencies, no wasm toolchain needed for regular builds.
- static/js/workers/hashWorker.js streams the File in 8 MiB slices off
the main thread (constant RAM at any file size).
- features/files/instantUpload.js orchestrates: threshold (8 MiB — below
it the round-trips cost more than the bytes), user-scoped
/api/dedup/check, by-hash registration, and silent fallback to the
normal byte upload on any miss, race or unsupported environment.
Wired into both uploadFiles and uploadFolderEntries.
- biome.json vendors exclusion fixed to cover nested directories
(previous vendors were .mjs and never matched the *.js include).
Verified end-to-end against PostgreSQL 16: node-driven WASM hash equals
the server's content_hash for a 20 MB file; by-hash returns 201 in ~10 ms
warm with a 151-byte request (vs 20,971,873 bytes for the byte upload);
the copy downloads byte-identical and the manifest ref_count goes 1→2;
a second user probing the same hash gets exists:false and 404 plus the
audit line; duplicate name → 409, malformed hash → 400; worker and wasm
are served with correct MIME (application/wasm).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Two follow-ups to the streaming-upload work:
The dedup garbage_collect() pass only ran when a user manually emptied
their trash. Zero-reference rows — chunks orphaned by aborted streaming
uploads (registered at ref_count 0 by the ingest rollback) and blobs
dereferenced by trash expiry itself — could linger indefinitely on
instances where nobody empties trash. The periodic TrashCleanupService
sweep now ends every run with garbage_collect() (maintenance pool,
batched), bounding orphan lifetime to the cleanup interval.
Folder-ZIP creation was strictly sequential: open blob stream, deflate,
close, repeat — every per-file blob-store round-trip (PG lookup + backend
open; a full HTTP round-trip on S3/Azure) added to the wall clock. It now
runs as a 2-stage pipeline: a prefetch task streams the planned files'
content ahead of the writer through a bounded channel (~4 MiB), so the
next file's read latency overlaps the current file's compression. ZIP
entries are still written strictly in order, peak RAM stays flat, and a
writer error hangs up the channel so the prefetcher stops on its own.
Verified end-to-end against PostgreSQL 16: an upload aborted at ~14 MB
left exactly 30 ref_count=0 chunk rows which the GC then reclaimed
(8.5 MB, rows + physical files); a chunked upload completed with a wrong
MD5 returned 400 with the tee-computed digest and the SAME session then
completed successfully with the right checksum (parts persist — the old
assembly deleted them, so the documented retry never actually worked);
a 4-file folder ZIP downloaded and extracted byte-identical.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
JWT validation cache now stores Arc<TokenClaims> and validate_token
returns Arc<TokenClaims>. On a cache hit — the 99% path for every
authenticated request — the moka lookup was deep-cloning the whole
claims struct (5 Strings: sub, jti, username, email, role) on every
call. It is now a refcount bump. Read-only callers (admin middleware)
go through Deref and allocate nothing; the auth middleware clones only
the three fields it moves into CurrentUser (was 5 clones, now 3), and
the admin paths clone only role (was 5, now 1). A new test asserts the
hit path returns a pointer-equal Arc.
TokenServicePort::validate_token is the single trait method touched;
its only implementor is JwtTokenService and the only production callers
are the auth and admin middleware (the WOPI handler uses a separate
WopiTokenService).
Dockerfile: rust:1.94.1-alpine3.23 -> rust:1.96-alpine3.24 and
alpine:3.23.3 -> alpine:3.24.0 for the runtime stage.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Files uploaded before chunk_manifests landed (20260414000000) are stored
as ONE whole-file blob with no manifest. Every legacy fallback in
DedupService exists to serve them, and the cost concentrates on Range
reads: with encryption enabled, seeking inside a legacy video decrypts
the ENTIRE blob (AES-GCM is all-or-nothing), where a CDC file decrypts
only the overlapping <=1 MiB chunks.
This adds a one-time, idempotent background migration (spawned from the
composition root after dedup init, maintenance pool) that converts each
legacy blob into a regular CDC file, indistinguishable from a native
upload:
1. Spool the blob through the normal read path (decrypts when
encryption is on) to a per-attempt-unique temp file, verifying
BLAKE3 == hash; sizes come from the verified spool, never from the
legacy storage.blobs.size column (the manifest's total_size drives
Range arithmetic).
2. CDC-chunk + store chunks via the existing store_chunks (one
manifest reference per distinct chunk).
3. One short accounting TX with the blob row locked: manifest INSERT
with ref_count = N current file references, blob ref_count -= N,
row deleted only at exactly 0 - so single-chunk files (chunk hash
== file hash) keep the physical blob, which IS the chunk; only
bookkeeping moves, no bytes are rewritten.
4. Physical whole-file blob deleted only when its row dropped.
Races lean on the row lock: a concurrent identical upload landing a
legacy reference after commit keeps the blob row alive and that file
readable via the fallback (bounded space leak, never data loss); a
crash between chunk store and the TX over-counts one file's chunk refs
(also a bounded leak). Corrupt blobs (content != hash) are logged,
counted, excluded from the sweep and left untouched, with a hard cap
before aborting.
Per-hash failures never block the sweep; manifests are the resumability
marker, so a restart continues where it left off. The legacy read/write
fallbacks stay in place as the safety net while a deployment converges;
they can be deleted once fleets report "legacy re-chunk: nothing to do".
Opt-out via OXICLOUD_LEGACY_RECHUNK=false (documented in example.env)
for metered remote backends where the one-time re-read should be
scheduled deliberately.
Covered by five integration tests against real PostgreSQL (multi-chunk
accounting + Range across a chunk boundary, single-chunk physical-blob
preservation, corrupt-blob isolation, empty blob, and the full
encrypted-backend roundtrip); they run concurrently, which also
exercises the cross-sweep race handling.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Three targeted frontend fixes:
1. pdf.js smart preload (thumbnail.js, resourceIcon.js). The first PDF
thumbnail of a session stalled 1-2s on the lazy import of the ~1.3 MB
pdf.js stack. buildResourceIcon() now fires thumbnail.preloadPdf()
the moment a PDF row enters the DOM, warming both the module
(~300 KB, via the now promise-memoized getPdfjsLib, shared with real
users) and the worker script (~1 MB, via a cache-priming fetch —
pdf.js only requests it on first getDocument). Only folders that
actually contain PDFs pay the download; idempotent after first call,
resets on failure so transient offline retries.
2. Admin migration poll cleanup (admin.js). The 2s setInterval kept
hitting the API and updating hidden DOM after leaving the Storage
tab, and polled a failing endpoint forever after session expiry
(!resp.ok returned without clearing). New stopMigrationPolling()
helper, invoked on tab switch away from Storage, on non-running
status, and on failed polls; tab re-entry re-arms via loadStorage().
3. resourceList.js render hoisting. Per-row i18n.t() type-cell lookups
and the fully item-invariant _renderCustomActions() HTML were
recomputed for every row; they now resolve once per batch via
_buildItemLabels() (per-category labels memoized, rebuilt each
batch so locale switches keep working). _findLaneByKey() swaps the
container-wide attribute querySelector for an O(1) _lanes Map kept
in sync at the only lane create/wipe sites.
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
Two hot-path fixes for DAV sync clients, which poll continuously:
1. App-password Basic Auth cache TTL 30s → 300s. Every cache miss costs
a full Argon2id verification (~50-100ms CPU) plus two DB queries; at
30s a continuously-syncing client re-paid that every 30s. 5 min cuts
it ~10x. Safe because revoke() already invalidates the user's cached
entries immediately; expiry/deactivation are only re-checked on a
miss, so they now have a <=5 min grace window (comparable to a JWT
access-token lifetime) — documented on the constant.
2. WebDAV lock store: replace the tokio::spawn + sleep scheduled per
acquire/refresh with Moka's per-entry Expiry policy. Office clients
refresh locks constantly, leaving thousands of orphaned sleeping
timers pinned in the runtime that were never cancelled. by_path now
carries the exact per-lock TTL via LockExpiry (no background tasks);
by_token keeps a 24h backstop and resolves through by_path, so a
lingering reverse-index entry can never resurrect an expired lock.
Adds unit tests for the expiry policy and acquire/refresh/release
(the module had none).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.
- Add find_events_by_ical_uids / get_contacts_by_uids through every
layer (domain repo trait → PG repo → storage port/adapter → use case
→ handler) using one indexed `= ANY($2)` query, mirroring the
existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
uid_from_multiget_href helper (percent-decoding, case-insensitive
extension strip, collection hrefs rejected). Exact UID matching also
removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
GET /api/address-books/{id}/contacts: optional ListQuery fields
thread through ContactUseCase::list_contacts to a paginated repo
query. Omitted params keep returning the full book (frontend and
DAV listing/sync paths unchanged).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
The read repository's hash_cache assumed blob hashes were immutable
per file_id, but update_file_content_from_temp remaps the SAME row to
a new hash via swap_blob_hash. After a WebDAV/NC PUT overwrite,
streaming downloads (>=10 MB tier, Range requests, video playback)
kept resolving the OLD blob for the 30 s TTI window — and because
every read refreshes the TTI, a polling client could pin the stale
mapping indefinitely, eventually turning into 500s once the old
blob's refcount hit zero and GC removed it.
The write repository now holds a shared handle to the same moka cache
(clones share storage) and invalidates the entry right after every
content swap and hard delete commits — covering every present and
future caller of the write port rather than one service path. Stale
doc comments claiming the mapping was immutable (and SHA-256) fixed.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Request-path thumbnail generation (REST get_thumbnail and NC preview)
read the full source blob into RAM and decoded it with no concurrency
bound — a first-view gallery of K images stacked K full-size buffers
and K parallel decodes on the blocking pool. The background hook had
the inverse ordering problem: it read the blob eagerly and only then
queued on the decode semaphore, so N concurrent uploads held N
originals in memory while waiting.
All three paths now acquire the decode semaphore first and read the
blob under the permit, capping peak RAM at permits x image size:
- new ThumbnailService::get_thumbnail_from_blob defers the blob read
into the moka init closure (cache and disk hits never touch the
blob); both handlers use it and no longer pre-read.
- get_thumbnail_from_bytes (direct-bytes variant) now also takes a
permit before decoding; shared generate_and_persist core keeps the
two entrypoints duplicate-free.
- generate_all_sizes_background_from_blob (renamed from _from_bytes)
checks blob existence and disk-dedup state, then acquires the
permit, then reads; the redundant pre-read spawn wrapper in
ThumbnailRefreshHook is gone.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Replace the per-row tree_modified_at bump triggers with AFTER ... FOR
EACH STATEMENT triggers using transition tables: each DML statement now
pays one bump covering the distinct ancestor chains of all affected
rows (locked in id order so concurrent bumps over overlapping chains
cannot deadlock), instead of one chain UPDATE per affected row.
Value-based change detection replaces UPDATE OF column lists (PG
forbids those with transition tables), so EXIF media_sort_date syncs
and no-op updates no longer bump at all, and file moves now invalidate
the source chain as well as the destination.
Also fix a latent bug surfaced while testing this: the descendant
path/lpath cascade (trg_folders_cascade_path) was declared AFTER
UPDATE OF path, lpath, but rename/move statements SET name/parent_id,
and BEFORE-trigger rewrites do not count for UPDATE OF — the cascade
never fired, leaving every descendant folder with a stale path/lpath
after any rename or move. The migration canonically repairs existing
trees and re-creates the cascade on the columns the app actually
writes.
delete_expired_bulk now deletes in LIMIT-ed batches (1000 files / 100
folders per round, each its own implicit transaction) ordered by
trashed_at and served by new partial indexes over trashed rows, so
retention purges no longer hold one unbounded transaction.
Verified against a local PG16 cluster: all migrations apply from
scratch, the new one re-runs idempotently, and a 16-case behavioral
battery passes (chain bumps for insert/update/delete/move/trash, EXIF
invisibility, no-op invisibility, cascade repair, FK-cascade depth
guards, batched purge shape and partial-index plan).
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
The five FileLifecycleHook call sites handed dto.etag (format
"{blob_hash[..16]}-{modified_at}") to the blob_hash parameter. Blob
lookups keyed by that value can never resolve, so background thumbnail
generation and audio metadata extraction silently failed on every
upload/copy/update, pushing all thumbnail work onto the request path.
Pass dto.content_hash — the raw full BLAKE3 hash already carried by
FileDto — instead.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Same whitespace-only reformat as branch claude/jolly-johnson-yso7z7:
PR #326 landed three files that fail cargo fmt --check and its CI run
skipped the Rustfmt job, breaking the check for every later Rust PR.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Resolves the four-file conflict with PR #444, which landed a parallel
implementation of the same batch-fsync design (unsynced chunk writes plus
one sync_blobs durability barrier). Both sides converged on identical
trait signatures; main's version is kept for all overlapping code since
it is already merged and carries unit tests:
- blob_storage_ports.rs: doc-comment-only conflict, main's docs kept
- dedup_service.rs: main's pipeline kept. The naive textual merge would
have retained BOTH durability-barrier/INSERT blocks, double-counting
ref_count for every new chunk; the duplicated block from this branch
is dropped
- encrypted_blob_backend.rs, local_blob_backend.rs: main's
implementations kept (fsync_paths_parallel + tests); this branch's
equivalent helpers (fsync_dir, fsync_blob_file, SYNC_BLOBS_CONCURRENCY)
are referenced nowhere else and are dropped
Everything unique to this branch (streaming PROPFIND, range requests,
batch ID resolution, CalDAV/CardDAV indexed lookups, transcode and
breadcrumb changes) merges cleanly and is unchanged.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Every folder navigation awaited rebuildBreadCrumb() — one sequential
fetch per ancestor level — before requesting the first page, so a
depth-8 folder paid ~8 RTTs of dead time before the content even
started loading. And nothing was cached: navigating between sibling
folders re-fetched the same ancestors every time.
- filesModel: session cache id → FolderItem for breadcrumb resolution
(warm navigations rebuild the trail with zero fetches), invalidated
on rename/move (single + batch) via invalidateFolderMeta. The trail
is now built locally and committed atomically, guarded by a
generation token so a rebuild superseded by a faster follow-up
navigation can no longer interleave writes into the newer trail.
- filesView.loadFiles: the rebuild runs concurrently with the first
page fetch; crumbs, history and title update when it resolves.
Navigation latency becomes max(listing, ancestor chain) instead of
their sum — and ~equal to the listing alone once the cache is warm.
The home-folder fallback for inaccessible targets is preserved by
reloading the listing when the rebuild had to reset app.currentPath.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Storing a new file through CDC dedup issued sync_all + a parent-dir
fsync for every ~256 KB chunk (~8,200 fsyncs for a 1 GB upload), plus
one PG INSERT round-trip per chunk. The actual durability boundary is
the manifest INSERT: chunks only need to be durable before any PG row
references them, not one by one.
- BlobStorageBackend grows put_blob_from_bytes_unsynced + sync_blobs
with conservative defaults (unsynced delegates to the synced write,
sync_blobs is a no-op) so backends that don't opt in keep the
per-write durability semantics. Remote stores are durable on PUT.
- LocalBlobBackend writes chunks without fsync and implements
sync_blobs as a parallel sweep: every listed blob file (hard
requirement) plus each distinct prefix directory exactly once
(best-effort, same tier as fsync_parent_dir).
- DedupService::store_chunks writes new chunks unsynced, runs one
sync_blobs sweep, then registers all new chunks in ONE batched
UNNEST INSERT - durability before visibility, and the per-chunk PG
round-trips collapse into one.
- Encrypted/Migration decorators forward both methods so the
optimization survives encrypted-local and live-migration stacks.
https://claude.ai/code/session_013Bk4BMQEvR9QxCU7QXLRwv
Creating a grant for a group dispatched the notification emails one at
a time inside the HTTP request — 30 members × ~500 ms of SMTP ≈ 15 s
holding the POST /api/grants response (the code carried a TODO
acknowledging it).
Dispatches are independent (coalescing and rate-limiting key on the
(granter, recipient) pair, distinct per member), so run them through
`buffered(6)`: ~6× less wall time for group fan-outs while capping
parallel SMTP sessions, with outcome order still matching member order.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The CDC chunk path paid `sync_all` + parent-dir fsync + one ref-count
upsert round-trip PER ~256 KB chunk: a new 1 GB file ≈ 8 000 fsyncs +
4 000 sequential INSERTs on the upload critical path (seconds on SSD,
tens of seconds on HDD).
- New `put_blob_from_bytes_unsynced` + `sync_blobs` on
BlobStorageBackend. Defaults delegate to the durable variants, so
remote backends (S3/Azure — durable on PUT) are untouched.
LocalBlobBackend writes chunks without fsync and `sync_blobs` then
fsyncs all files concurrently (kernel coalesces the writeback) plus
one fsync per DISTINCT shard directory — at most 256 — instead of one
per chunk. EncryptedBlobBackend delegates both so encryption over the
local backend keeps the optimization.
- store_chunks: uploads use the unsynced variant; one `sync_blobs`
barrier runs before anything references the chunks, and the per-chunk
ref-count upserts collapse into a single
`INSERT ... SELECT unnest(...) ON CONFLICT` statement.
Durability contract is unchanged: every chunk is on stable storage
before the manifest row that references it commits. A crash mid-upload
leaves chunk files without DB rows — the same orphan class the
per-chunk scheme already produced, just a wider window.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every CalDAV PUT/GET/DELETE of one .ics loaded the ENTIRE calendar —
every row including its ical_data — and filtered with .find() in Rust,
so importing N events cost O(N²) rows transferred. CardDAV did the exact
same in four places (PROPFIND of one .vcf, PUT existence check, GET,
DELETE), with three JSONB deserializations per discarded contact. The
indexed repo queries (find_event_by_ical_uid, get_contact_by_uid)
existed all along with zero callers.
Wire them end to end: new `get_event_by_ical_uid` /
`get_contact_by_uid` use-case methods (same access checks as
list_events / list_contacts, per the service-layer authz rule) exposed
through the storage ports and adapters, and the seven handler sites now
resolve one row instead of the whole collection.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every GET of a JPEG ≤5 MB with `Accept: image/webp` ran a full decode +
lossless-WebP encode on the rayon pool (hundreds of ms of CPU), found
the result LARGER than the original — the `image` crate's WebP encoder
is lossless-only, which practically never beats an already-lossy photo —
and threw the work away. Nothing recorded the verdict, so the next GET
of the same file repeated it, forever. A 20-photo gallery saturated the
transcode pool on every visit.
- `can_transcode` no longer accepts `image/jpeg` — photos skip the
pipeline entirely. PNG/GIF stay (lossless WebP genuinely shrinks them).
- Negative verdicts for the remaining formats are now remembered: an
empty-Bytes sentinel in the moka cache (expires with the 10-min TTL)
plus a zero-byte `.skip` marker on disk that survives restarts.
`invalidate()` removes the marker alongside the cached transcode.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Two DAV-surface fixes that replicate patterns the codebase already had:
NC PROPFIND (folder case) previously loaded EVERY child via unbounded
list_files/list_folders and serialized the entire multistatus into one
Vec (~2 KB per entry — a 50k-file folder meant ~100 MB of buffer per
request, repeated constantly by sync clients). It now mirrors the native
WebDAV handler's streaming builder: children are fetched in pages of
PROPFIND_BATCH_SIZE (500), each page's favorites and oc:fileids are
resolved with two batch queries, and the XML is yielded chunk by chunk —
memory stays O(batch) and the first byte flows immediately. The
single-file PROPFIND keeps a small buffered variant; the multistatus
opening tag is factored into a shared helper so the namespace set cannot
diverge.
WebDAV GET (native and NC) ignored the Range header and never compared
the ETag it emitted, so mount-style clients (rclone, davfs2, Finder)
re-transferred whole files on every seek, resume, or revalidation. New
shared `interfaces::range_requests` helpers — same semantics as the REST
download endpoint, which now reuses the 304 helper too — give both GETs
If-None-Match → 304, Range → 206/416, and Accept-Ranges advertising.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Four related fixes to stop buffering files in page memory and stop
hammering layout from the selection paths:
- Inline viewer media: video/audio fetched the ENTIRE file into a blob
before the first frame (a 2 GB video = 2 GB of tab heap, no
progressive playback, no seek). The <video>/<audio> src now points
straight at the same-origin API URL — cookies travel automatically
and the browser streams with native Range requests, exactly like the
music player already did.
- Downloads (file, folder ZIP, batch ZIP, viewer button): the
fetch → blob → objectURL pattern materialized the whole payload in
RAM before the save dialog appeared (a 10 GB batch ZIP risked
crashing the tab) with no download progress UI. New shared
utils/download.js hands the URL to the browser, which streams to
disk with its own progress UI. Batch download uses the existing GET
endpoint (same URL contract as the drag-out DownloadURL, now shared
via buildBatchDownloadUrl); selections whose id list cannot fit in a
URL keep the buffered POST fallback. Trade-off: failed downloads now
surface in the browser's download shelf instead of an in-app toast.
- Rubber-band lasso: every mousemove (>100/s) walked all cards
interleaving getBoundingClientRect() reads with class writes — up to
N forced reflows per event, freezing the frame rate on folders with
thousands of loaded rows. Card rects (+ item info) are now snapshot
once per drag (rebuilt on scroll), and a single rAF pass per frame
compares against the cached geometry, touching only cards whose
selection state changed.
- batchToolbar: getSelection() rebuilt the selection by scanning every
.file-item in the document (the TODO admitted it); it now reads the
_selected Map that every selection path already keeps in sync.
clear() scopes its DOM sweep to #files-list (the only container the
toolbar manages) instead of the whole document.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The global childList observer scheduled replaceIconsInElement() with no
scope on ANY node insertion — including bare text nodes — so every
notification-bell progress tick during an upload and every
infinite-scroll batch re-scanned the whole document with an attribute
substring selector. Cost grew with total DOM size (5-10k nodes), not
with what was inserted.
Queue the added element roots per animation frame and scan just those
subtrees (an inserted <i> itself is caught via its parent). Text-node
churn no longer triggers any scan at all. The icon replacement is
idempotent, so overlapping roots are harmless.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
_bindItemEvents attached direct click listeners to the favorite star,
the shared badge and every custom action button of every row — ~3+
listeners per row on top of the container delegation the component
already wires, i.e. ~10-15k listeners with a few thousand rows loaded
via load-more, plus the per-row binding cost on every render.
The container's delegated click handler already dispatches by
closest() checks; move the three button behaviours there (they return
before the card-open branch, preserving the old stopPropagation
semantics) and drop _bindItemEvents entirely. Row creation is now
pure innerHTML — no listener allocation per row.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
displaySearchResults inserted up to 100 results one at a time via
addItem({ scroll: true, highlight: true }) — each insert ran two DOM
scans (duplicate guard + row lookup), a smooth scrollIntoView and a
highlight pulse, so every live-search keystroke triggered ~100 competing
smooth-scrolls and O(n²) container scans.
Add filesView.renderItems(): a single component.render() pass (one
DocumentFragment, zero per-item scans/scrolls). addItem keeps the
scroll/highlight affordances for its documented purpose — optimistic
single-item inserts after upload/create.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
`uploadFiles` sent files one at a time (await per file), so dropping N
files meant N sequential round-trips. Reuse the bounded-concurrency pool
already proven in `uploadFolderEntries` (CONCURRENCY = 10): independent
files now upload up to 10 at a time — ~10x faster for many small files.
Per-file progress (XHR → bell), the legacy progress bar, timeout
notifications and the quota short-circuit are preserved. On a quota error
the pool stops pulling new files while in-flight uploads finish (matching
`uploadFolderEntries`) instead of the old hard `break`.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The entrypoint ran "chown -R" on the storage volume on every container
start. Storage is a content-addressable blob store that can hold millions
of objects, so a blind recursive chown re-stats and rewrites the inode of
every blob on each boot, adding minutes of startup time and saturating the
disk on spinning media.
Guard the chown behind a top-level ownership check: only recurse when the
directory root is not already owned by the oxicloud user. The first boot
fixes a freshly mounted (root-owned) volume; every later boot is a no-op.
The same guard is applied to the static dir, factored into a shared helper.
The target UID is resolved via "id -u oxicloud" instead of hardcoding 1001.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Deleting a multi-photo selection fired one `DELETE /api/files/{id}` per
photo in series — 300 selected photos meant 300 sequential round-trips
with the UI blocked on `confirm()`. Switch to `POST /api/batch/trash`
(already used by the file view), chunked to the backend's MAX_BATCH_SIZE
of 1000, so a selection of any size collapses to ceil(N/1000) requests.
Only items the server reports as successfully trashed are removed from
the grid; the selection bar refreshes to reflect any failures.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
The .dockerignore excluded a non-existent doc/ directory (the real one is
docs/) and did not exclude tests/, images/, charts/, or tools/. None of
these are COPYed by the Dockerfile, yet ~24 MB of them were sent to the
daemon as build context on every build. Fix the typo and exclude the
unused asset/tooling directories so the context stays minimal.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
sqlx's postgres driver implements the wire protocol in pure Rust and TLS
goes through rustls, so libpq is never linked. Confirmed via Cargo.lock:
no pq-sys, native-tls, or openssl-sys in the dependency tree.
Removing postgresql-dev from the build base and libpq from the runtime
image shrinks the final image by ~3 MB and removes a C library from the
attack/patch surface. perl/make/gcc/musl-dev are kept for the C builds of
aws-lc-sys.
https://claude.ai/code/session_01GpprjxjtXFYLfXNkoKnHuL
Closes#323.
LoginLockoutService cached failed-attempt counters keyed only on
the username, so any caller that could reach the auth endpoint and
guess (or enumerate) a username could lock that account out for the
entire lockout window — the rate limiter happily lets each IP make
its share of bad-password attempts before clamping, which is enough
to trip the per-account threshold in seconds. The reporter
demonstrated a complete DOS by spoofing X-Forwarded-For with
OXICLOUD_TRUST_PROXY_HEADERS=true.
Fix: change the lockout cache key from `username` to `username|ip`.
A flood from one IP locks that IP out of that account, but a
legitimate user coming from a different IP is unaffected.
Changes:
- LoginLockoutService::{check, record_failure, record_success} take
client_ip as a second argument; cache key is built via Self::key
(`format!("{username}|{ip}")`).
- middleware/rate_limit.rs: factor out extract_client_ip_from_parts
(HeaderMap + Option<&SocketAddr>) so handlers that don't take a
full Request<B> can still derive the same client identifier
extract_client_ip uses. extract_client_ip now delegates to it.
- auth_handler.rs login: derive client_ip from headers (the only
signal available without ConnectInfo) and pass it through to all
three lockout calls.
- nextcloud/basic_auth_middleware.rs: do the same with the full
Request via extract_client_ip.
Tests:
- Updated existing 4 unit tests to thread an IP arg.
- New does_not_lock_out_other_ips_for_same_account: lock from IP1,
assert IP2 still allowed (the #323 regression).
- New success_resets_only_the_acting_ip: a successful login from
IP2 must NOT clear an attacker's lockout from IP1.
Verification:
- `cargo build` ✅
- `cargo test login_lockout` → 6 passed (4 existing thread an IP
arg without behaviour change, 2 new pin the per-IP scoping).
Signed-off-by: SAY-5 <say.apm35@gmail.com>
DioCrafts