Add an end-to-end and unit test suite for the SvelteKit frontend:
- Playwright e2e specs (tests/e2e/spa) with a throwaway container stack,
codegen scenarios, and an Istanbul-based coverage report pipeline.
- Vitest unit tests across API endpoints, components, stores and composables.
- `data-testid` hooks on interactive elements (AppShell, FileViewer,
ShareDialog, search, photos, files breadcrumbs, login/Nextcloud flows,
public share pages) so the e2e suite can target them deterministically.
- Serve the SPA app-shell CSP from a <meta> policy (svelte.config.js) plus a
middleware that skips the CSP header on HTML; move the Nextcloud Login Flow
v2 grant page to the SvelteKit /nextcloud/login route.
- `just front-codegen` recipe and start-server-spa.sh harness.
Make the test environment robust and consistent:
- Install a deterministic in-memory localStorage/sessionStorage in the Vitest
setup so storage behaves identically across Node versions (Node 26 ships a
native Web Storage global that otherwise shadows jsdom's).
- Pin devenv to Node 26 + PostgreSQL 18 and pin every CI job to Node 26.3.0
so the dev shell and CI run the same toolchain versions.
Repair the API/WebDAV (hurl) suite, which had drifted from the backend:
- Migrate the removed `/api/folders/{id}/listing` endpoint to `/resources`
(cursor-paginated `{items:[{resource_type,resource}]}` shape) across the
batch-copy, grants, nested-group, and WebDAV NC tests + the dav_helpers
wipe routine.
- Stop photos_etag from uploading the dedup-tracked fixture so the dedup
blob-lifecycle test can own its content-addressed blob exclusively.
- dedup_create now asserts the idempotent same-content re-upload (201 +
existing file id) instead of the stale 409 expectation.
Generated coverage reports, nyc output and the e2e server runtime data dir
are gitignored rather than committed.
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).
- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
(infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
extract one representative frame as PNG, bounded by its own semaphore + a
per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
to a size- and time-bounded temp file on the data volume, extract a frame, and
reuse the shared render_and_persist_all_webp helper — so video thumbnails are
WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
path; the server is now the source of truth.
Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.
Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The legacy-frontend removal dropped the deprecated /api/folders/{id}/listing
route, but folders.ts still called it, so every folder view 404'd
("listing failed: 404"). Complete the migration: fetchFolderListing now pages
through the cursor-paginated /api/folders/{id}/resources feed and rebuilds the
combined {folders, files} listing the views expect.
- Pages through next_cursor (limit 200) and splits mixed resource items by
resource_type. 403 still throws; the 304/ETag fast-path is gone (that feed has
no whole-listing ETag) so the in-memory folderCache is the only revalidation.
- Favorite/share badge sets aren't carried by /resources, so they come back
empty for now (no star / share badge until wired from /favorites + /shares).
- folders.test.ts updated to the paginated shape.
npm run check: 0 errors. 58 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to the connection-exhaustion fix. Rather than routing large files to
plain uploads (which kept STORAGE dedup but gave up delta's re-upload bandwidth
savings), keep delta for every file >= 8 MB and instead cap each worker's
concurrent connections so a few large files uploading at once can't blow past
the browser's ~6-per-host budget and starve the small-file plain uploads.
- deltaWorker.js: serialize negotiate (at most one in flight per worker) and
drop chunk-PUT concurrency 2 -> 1, so each worker holds ~2 connections max.
- deltaUpload.ts: revert the 64 MB threshold back to 8 MB — every large file
gets sub-file dedup again. (Storage dedup was never affected: BLAKE3 + CDC +
ref-counting run server-side for plain and delta uploads alike.)
With main upload concurrency at 2, total in-flight upload connections stay <= ~4.
npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With WASM finally enabled, large files (e.g. 32 MB logs) started running the
delta worker, which opens SEVERAL concurrent requests each (overlapping
negotiate batches + chunk PUTs). A few of those running at once blew past the
browser's ~6 connections-per-host limit, so plain uploads of the small files
queued with zero bytes sent until the 30 s stall watchdog cancelled them — the
upload "stuck at 4% / 94%" with N (pending) XHRs in the Network panel. The
session-refresh request got starved too (the spurious 401s).
- Raise the delta-worker threshold to 64 MB (new DELTA_WORKER_MIN_SIZE) so
typical large files take a single-connection plain upload. Delta's payoff is
sub-file dedup on RE-upload; on a first upload it is pure connection overhead.
Client-side instant-hashing still only reads files < 8 MB into memory.
- Lower upload concurrency 3 -> 2, leaving headroom under the 6-connection
budget for session refresh/poll and the occasional delta worker.
npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of folder uploads "freezing at ~95%": the global Content-Security-
Policy `script-src` was `'self'` + inline-script hashes with NO
`'wasm-unsafe-eval'`. Chromium therefore blocked `WebAssembly.instantiate`
("Wasm code generation disallowed by embedder"), so the vendored BLAKE3/FastCDC
WASM threw on instantiation — both on the main thread (instant by-hash uploads
and the batch dedup check) and inside the delta-upload worker. Every file then
fell back to a plain byte upload, and the backend logs showed 0 check-batch /
0 negotiate calls. Large files (32 MB service logs) compounded it and the
session token expired mid-upload, so the last handful failed.
- web/mod.rs: add `'wasm-unsafe-eval'` to `script-src`. WASM-only, safe variant
— does NOT enable `eval()`/`new Function()`. Restores instant uploads, delta
(sub-file dedup), and the client hashing the idempotent re-upload relies on.
- deltaUpload.ts: liveness watchdog on the delta worker. A healthy worker posts
progress sub-second; if it goes silent for 20 s it is wedged (WASM init or
chunking hung without throwing) — disable delta for this file AND every later
one so they fall straight through to a plain upload instead of each burning
the full 120 s+ delta timeout. Defense-in-depth so a broken WASM path can
never again freeze an upload for minutes.
cargo test: pass. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A folder upload with several large files could appear frozen for ~2 min: a few
concurrent uploads stalled and the old 120s per-file timeout neither aborted the
request (leaving zombie XHRs that exhaust the browser's per-host connection pool)
nor recovered quickly.
- uploadFileWithProgress now self-aborts on a stalled connection: the deadline
resets on every upload-progress tick (a slow but *moving* transfer is fine),
and once the body is sent the server gets a fixed window to respond; on a stall
xhr.abort() frees the connection immediately — no zombie, no cascade.
- Lower upload concurrency 4 -> 3 to reduce server contention from large
concurrent uploads.
- The outer per-file timeout is now just a generous backstop for a wedged delta
worker / by-hash request.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend:
- POST /api/dedup/check-batch — returns the subset of submitted whole-file
BLAKE3 hashes the caller already owns, in one query (user-scoped,
anti-enumeration via idx_files_blob_hash). Lets a client learn which of N
files it can skip with a single round trip.
(dedup_service::user_owned_blob_references, dedup_handler, routes) + tests.
Frontend — upload pipeline:
- Instant ("by-hash") upload for content the caller already owns: hash every
in-band file, ONE /api/dedup/check-batch, create the owned ones with zero
content bytes, upload only the rest. Covers all sizes below the 8 MB delta
threshold (delta handles larger files). vendor/hashWasm computes the
whole-file BLAKE3 on the main thread.
- Resilient parallel uploads: bounded concurrency (4) + a per-file deadline,
so one stuck/slow/failing file no longer freezes the whole batch — it blocks
only its own lane and times out / is skipped while the rest proceed. Quota
exhaustion stops the run early; partial results are reported ("N uploaded,
M failed").
- Folder uploads (uploadTree) show live bell progress + a final result and go
through the same dedup + parallel pipeline.
- Storage bar ("Almacenamiento") refreshes after uploads/deletes
(session.refresh) instead of showing the stale login value.
Frontend — i18n / UI fixes:
- Fix literal {{count}} and {{percentage}}/{{used}}/{{total}} (param-name
mismatches) in the selection toolbar and storage line; add es strings.
- Remove the underline on user-menu link rows.
Benchmark (uploadStrategies.bench.test.ts) compares baseline / per-file / batch:
the batch collapses N per-file probes into one check (e.g. a WAN 1000-file run
drops from 1700 to 1001 round trips) while matching per-file's byte savings.
Also includes in-progress group virtual-description i18n work present in the
working tree (groups.ts, ResourceList, locale `groups` keys).
Verified: cargo clippy -D warnings (clean), backend 448 tests; frontend
npm run check (clean), 58 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bundles the backend+frontend delta-upload (content-dedup) feature with a
batch of frontend fixes from this session.
Upload / dedup:
- Client-hashed delta & instant upload (deltaUpload, hashWasm vendor shim)
- Backend dedup batch endpoint (dedup_service, dedup_handler, routes)
- session store owned-hash helpers; unit tests + upload-strategy bench
Frontend UI/UX:
- Colour file-type icons in grid/list (per-type tinted tiles + glyph hue)
- Robust thumbnail fallback; PDFs now show their type icon (backend
generates no PDF thumbnails) instead of a blank tile
- Fix PDF preview: load via a same-origin blob: iframe — the API URL is
blocked by the global X-Frame-Options: DENY in the browser's framed
PDF viewer, matching the existing CSP `frame-src blob:` design
- Groups: localized virtual-group description (no DB schema-note leak),
add nav.groups to the 15 missing locales, fix primary-button contrast
- Repoint --color-text-light → --color-on-accent (was faint grey on accent)
- Nudge the admin role badge off the user-menu header divider
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two related parity gaps from the VanillaJS → Svelte migration (issue #500):
internal-vs-external users weren't badged, and external users in a share's
member list rendered as a bare UUID with a static icon — no avatar, no email.
Both share one root cause: there was no shared user vignette and no resolver
for non-directory (external) users (the system address book lists internal
users only, and ShareDialog hardcoded isExternal=false).
- lib/api/endpoints/users.ts: resolveUser(id) — cached GET /api/users/{id}
(the authenticated per-user profile lookup) → {name, email, image,
isExternal}; returns null when the profile isn't visible so callers keep
their fallback label.
- lib/components/UserVignette.svelte: reusable identity chip — avatar (photo
or coloured initials), name, email, and a building-circle-xmark badge for
external users; resolves lazily and falls back to a caller-supplied label.
- lib/utils/avatar.ts: userInitials() + avatarColorIndex() extracted from
AppShell (now shared by both — no duplicated logic) so vignette and account
button render identically.
- ShareDialog: user member rows now render <UserVignette>; groups keep their
icon+label. Drops the dead hardcoded isExternal.
Backend already exposes everything (UserDto.email/image/is_external via
GET /api/users/{id}); no backend change. Frontend gate green (svelte-check
0/0, eslint, stylelint, prettier) + 47 Vitest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every navigation rebuilt the breadcrumb with one `GET /api/folders/{id}` per path
segment (a depth-D folder = D requests, each no-store) purely to label the trail.
Add an id→name cache, populated wherever a name is already known:
- every listing names its children, so `cacheFolder` records them, and
- `getFolder` records the folder it fetched.
`buildCrumbs` now reads names from the cache and only fetches the ids it hasn't
seen. During normal step-by-step navigation each ancestor was named by its
parent's listing, so the breadcrumb resolves with ZERO extra requests; only a
cold deep-link fetches its unknown ancestors (still in parallel). Folder renames
update the cache immediately so the trail stays correct.
The cache is a small LRU (cap 1000 — names are tiny) and is independent of the
listing cache (names survive a listing invalidation).
Validated: 3 new unit tests (listing populates child names, getFolder records,
rename overwrites) → 46 frontend tests green; npm run check; headless render of
the real files route (list + grid) — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
Every folder navigation re-downloaded the full listing: `listFolder` cache-busted
with `?t=<ts>` + `Cache-Control: no-store`, so back/forward and re-entering a
folder always paid a full round-trip + payload.
Now the files browser caches listings in memory and serves SWR:
- On navigation it paints a previously-visited folder instantly from cache, then
revalidates with `If-None-Match` (the backend ETag covers folders + files +
favorite/share badges, so it's a faithful validator). Unchanged → 304 with an
empty body; changed → 200 refreshes cache + UI.
- A generation token guards against a slow in-flight response clobbering a newer
navigation; breadcrumbs now resolve independently so they never block the grid
paint.
- Mutations (create/upload/rename/move/copy/delete, incl. the move dialog) go
through `reload()`, which drops the cache and refetches fresh — no stale view
after an action.
API layer (`folders.ts`):
- `fetchFolderListing(id, { etag?, forceRefresh? })` does the conditional fetch
(200 → parsed listing + ETag, 304 → empty); `listFolder` stays as a
non-conditional wrapper for the move-dialog tree.
- A small LRU (cap 40) cache with `getCachedFolder` / `cacheFolder` /
`invalidateFolderCache`. `cache: 'no-store'` keeps the browser HTTP cache out
of the way; revalidation is driven entirely by our own ETag.
Net: instant back/forward navigation, and an unchanged folder revalidates with a
0-byte 304 instead of re-downloading the whole listing. Validated: 7 new unit
tests (conditional If-None-Match + 304, LRU eviction/recency, invalidation),
npm run check, and a headless render of the real files route (list + grid).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.
Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
`PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
which already covers public links as 'token' grants — same membership the
/grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).
Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
badge sets straight from the listing and updates them optimistically on
favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
the single, authoritative, fetch-free source.
Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
The merge brought in main's ReBAC→role-grants migration, which changed the
grant contract the Svelte sharing UI (branched before it) was written
against: GrantDto dropped `permission` and now carries an explicit `role`
(owner/editor/viewer/commenter/contributor), and the "admin" role was
renamed "owner". Left unchanged, the share UI derived roles from a
now-absent `permission` field and showed every member as "viewer".
- grants.ts: ShareRole is now viewer|editor|owner; Grant carries `role`
(not `permission`); `roleFromPermissions` → `displayRole`, which collapses
the unexposed commenter→viewer and contributor→editor.
- ShareDialog.svelte: read each subject's role directly (role-grants emits
one row per subject); role picker exposes Owner instead of Admin.
- shared/+page.svelte (My Shares): same owner rename; role badges run
through displayRole so server-only roles render sensibly.
Create/update already POST `role`, so only the read/display path and the
role literal needed fixing. npm run check, test:unit (36) and build pass.
Bring the Photos/People/Places UI that main added (in the legacy vanilla
frontend) into the SvelteKit rewrite, wired to the now-merged backend
(/api/photos/geo, /api/people/*).
Photos page (routes/photos/+page.svelte):
- Moments | Places | People sub-tabs (the People tab appears only when the
faces feature is enabled, via a /api/people capability probe), mirroring
the vanilla photos sub-nav.
- Square ↔ justified layout toggle. Justified uses a Flickr-style
row-packer over the width/height the photos list endpoint returns
(PhotoItem), falling back to 1:1 when dimensions are missing.
New components:
- PhotoLightbox.svelte — the lightbox extracted from the photos page into a
reusable component (items + bindable index, onDelete callback) so the
grid, People and Places all share one implementation (no duplication).
- PlacesMap.svelte — MapLibre GL map with server-clustered markers; the
vector basemap is optional (probed at /basemaps/basemap.pmtiles, themed
fallback otherwise). Cluster click zooms in or opens the lightbox.
- PeopleView.svelte — identity-cluster grid → per-person photo grid, with
rename via the in-app prompt dialog.
Supporting:
- api/endpoints/people.ts (+ peopleEnabled probe); photos.ts gains
fetchPhotosGeo + GeoCluster + PhotoItem; fileThumbnailUrl takes a size.
- lib/vendor/maplibre.ts — minimal typings + lazy loader for the vendored
MapLibre GL + pmtiles globals (kept any-free for ESLint).
- utils/media.ts — shared isVideo / photoTimestamp / minimalPhotoItem.
- Vendored maplibre-gl 5.24.0 + pmtiles 4.4.1 under static/vendors and an
optional static/basemaps dir, matching the PR's vendored-asset pattern.
- New photos.tab_*/layout_*/map_* + people.* keys in en.json.
Verified: npm run check (svelte-check + eslint + stylelint + prettier),
npm run test:unit (36 pass), and npm run build all green.
Edouard Vanbelle