add `include_in_photo_index` and `include_in_music_index` policies
both true for default personal drive
photo is implemented
music is not yet implemented
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
The SvelteKit app under /frontend has fully superseded the legacy
vanilla-JS/CSS frontend in /static, which was only ever served by a
debug `cargo run` / `PROFILE=dev` and never shipped to production.
Remove it together with the whole subsystem that existed only to
support it (~54k lines).
Frontend & assets:
- Delete /static (js/, css/, *.html, sw.js, basemaps/, locales symlink).
- Relocate the brand/PWA assets (logo/, favicon.ico, manifest.webmanifest)
to frontend/static/ so they ship with the SPA. This also fixes the
favicon, which app.html referenced but was missing from the prod bundle.
- Migrate the Nextcloud login-flow redirects from /nextcloud-error.html
to the SvelteKit /nextcloud/error route.
Web layer:
- Simplify resolve_static_path: drop the PROFILE=dev branch; always prefer
the Vite static-dist/ build, fall back to the configured path.
- Resolve i18n locales from the served SPA dir with a frontend/static
fallback so `just dev` works without a prior build.
Build:
- Prune build.rs from 1262 to ~70 lines (git metadata only); the Rust asset
pipeline and the OXICLOUD_RUST_ASSETS rollback flag are gone.
- Drop the now-unused build-dependencies (oxc_*, lightningcss).
- Remove the COPY static lines from the Dockerfile (cacher + builder).
Tooling & docs:
- Delete biome.json, jsconfig.json, tools/check-*.py, identifier.sh.
- Remove the legacy front-* justfile recipes; repoint the design-system
scripts (locales, dead-tokens, brand-drift, token-docs) at the frontend,
and drop check-contrast/check-headings (coupled to the old token
taxonomy / multi-page HTML).
- Repoint docs/DESIGN-SYSTEM.md links; remove 5 superseded docs/plan/*.
Backend dead code:
- Remove the dead `folder_repo` field from FileBlobWriteRepository.
- Remove the deprecated GET /api/folders/{id}/listing endpoint
(superseded by /resources).
Verified: cargo clippy (all-features/all-targets) clean, cargo test
--workspace 448 passed, cargo fmt clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Re-uploading a partially-uploaded folder used to surface hundreds of spurious
"already exists" failures, and a file the watchdog aborted (or one the server
committed just before the client gave up) was lost.
Backend — save_file_with_blob_impl (the shared write path for both plain and
by-hash uploads): on a name conflict (23505), if the existing non-trashed file
holds byte-identical content (same folder, same name, same blob hash), return
that file as success instead of erroring. A different-content clash still
conflicts. Re-upload / re-sync becomes a clean no-op for everything already
stored — only the genuinely missing files transfer.
Frontend — uploadWithRetry: each file gets one automatic retry on a transient
failure (quota is never retried). With backend idempotency, retrying an
already-stored file is an instant no-op and a stalled/aborted file gets a real
second chance, so a folder upload self-completes instead of leaving gaps.
cargo test: 448 passed. npm run check: clean, 58 tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
now Drive is purely a metadata
each drive has always a root folder
this model minimize Oxicloud changes, and simplify
the Drive name is simply the folder's root's name
note: owner of Drive has more permission that an owner of the root folder
Phase 2 increment 3: FacePgRepository implements FaceRepository.
Embeddings stored/read as BYTEA (512 little-endian f32), bbox as REAL[].
Every query is user-scoped. Covers face CRUD, person CRUD (create / rename
/ cover / hide), files-for-person, and a transactional delete-all-for-user
(right to erasure).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Three list endpoints resolved each resource with one query per id:
- GET /api/grants/incoming and /api/grants/outgoing used
join_all(ids.map(get_file)) + join_all(ids.map(get_folder)), so a single
page (limit ≤ 200) could demand ~200 concurrent connections from the
20-connection primary pool, causing acquire-timeouts and head-of-line
blocking under load.
- The NextCloud favorites REPORT (oc:filter-files) fetched get_file/
get_folder once per favorite — up to N serial round-trips per sync.
Add by-ids batch reads that mirror the existing get_file/get_folder column
mapping and NOT is_trashed filter:
- FileBlobReadRepository::get_files_by_ids / FolderDbRepository::get_folders_by_ids
(one SELECT ... WHERE id = ANY($1)), exposed as FileRetrievalService::
get_files_by_ids / FolderService::get_folders_by_ids returning DTOs.
- Both grant handlers and the favorites REPORT now issue two batch queries
total and look results up by id, preserving original order. Missing ids
(stale grants whose resource was deleted, or trashed/removed favorites)
drop out exactly as before. No auth-semantics change: these paths already
resolved ids vetted by the authorization engine / favorites table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TAzLEQDaLak3dnrEN3YT35
Phase 1 server side, gated on OXICLOUD_ENABLE_PLACES (off by default):
- migration: partial index on storage.file_metadata(longitude, latitude).
- FileBlobReadRepository::list_geo_clusters — plain-SQL grid aggregation
(no PostGIS) scoped to the caller's own non-trashed photos, returning a
centroid, count and a representative file id per non-empty cell.
- PlacesService (caller_id-scoped; user-scoped data needs no authz check,
mirroring RecentService) with a zoom→cell-size mapping.
- GET /api/photos/geo?bbox=w,s,e,n&zoom=N returning GeoCluster[]. The route
is mounted only when the Places service is present, and is registered in
the OpenAPI path list.
The map frontend (PMTiles serving + MapLibre module) is deferred pending
the basemap-sourcing and vendoring decision.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
list_media_files now LEFT JOINs storage.file_metadata and returns each
photo's pixel dimensions next to the sort date. The endpoint wraps FileDto
in a flattened PhotoDto carrying width/height, so the gallery can lay tiles
out at their true aspect ratio (justified layout) without a second per-file
metadata round-trip and without layout shift. FileItem gains optional
width/height. No change to FileDto or its other construction sites.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JW6ghFMDtnRYuYNzZhb47M
Threads `new_name: Option<&str>` through FileWritePort::copy_file and
FileManagementUseCase::copy_file_with_perms so a same-folder
COPY /a.txt → /b.txt picks up the destination name via a single
COALESCE($3::text, name) in the CTE. Without it the new row inherits
the source's filename and collides on the (folder_id, name, user_id)
unique index — the "Already Exists" 500 M8 was hitting.
handle_copy in the native WebDAV surface now passes
`(file.name != dest_name).then(|| dest_name.into())`, keeping the
"same name in a different folder" case at None so existing semantics
are preserved.
Closes G9.
DELETE on a folder now flips is_trashed on every descendant folder
and file under it in a single CTE pipeline (lpath <@ root.lpath
covers the whole subtree via the GiST index). Previously only the
root row was flipped — descendants stayed live, directly addressable
via their full path, and confused desktop-sync tree walks that
expected the parent-collection 404 to imply the children were gone
too.
restore_from_trash mirrors the cascade: descendants where the
original_*_parent_id column is NULL are the ones we cascade-trashed,
so they get cascade-restored too. Descendants that were independently
trashed before the parent went to trash have original_*_parent_id
set, so they stay in trash and remain visible as top-level entries
in storage.trash_items.
No schema migration needed — both original_parent_id (folders) and
original_folder_id (files) were already nullable and already encoded
'where this came from when it was independently trashed'; using NULL
as the cascade-marker reuses that existing distinction cleanly.
Test G9 flipped from KNOWN BUG to assert every descendant 404s after
the parent DELETE; new G9b proves the inverse cascade by restoring
the trashed root and verifying every descendant comes back at its
original path.
save_file_with_blob_impl did three sequential DB round-trips per upload:
resolve_user_id (SELECT folders.user_id), the INSERT, then lookup_folder_path
(SELECT folders.path) — the first and third re-reading the same folders row.
Replace them with a single statement: a `parent` CTE reads the folder once and
the INSERT derives user_id from it and returns the path via the CTE, in one
round-trip. An empty CTE (folder vanished between ingest and insert) inserts
zero rows and now surfaces as a clean NotFound instead of a generic owner
error. Deadlock retry, blob-ref compensation, and the 23505 (duplicate name)
mapping are preserved; owner resolution + insert are now atomic (no TOCTOU).
What it does and doesn't buy (benchmarked, honest):
- Server CPU: UNCHANGED. A server-side 50k loop is identical (13.6s vs 13.6s)
— the two extra folder reads are cached point lookups, negligible against
the INSERT + per-statement triggers + 13 indexes.
- Client-observed latency: 2 fewer client<->DB round-trips per upload. At the
measured ~189us/round-trip on localhost that's ~0.38ms/upload; on a
networked DB (RTT 0.5-1ms) ~1-2ms/upload.
- Connection pool: the metadata phase holds a pooled connection for 1
round-trip instead of 3, freeing it ~3x sooner under upload concurrency.
So this is a latency + connection-utilization win (and an atomicity/cleanup),
not a server-CPU win. register_file_deferred keeps its own path.
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
Both sides added a parameter to create_application_services and a
setup step before it: this branch's storage-usage/quota service (for
the instant-upload path) and main's Tantivy content index (for
SearchService). The resolution keeps both — the signature takes both
arguments and the build runs storage usage as step 3c and the content
index as 3d.
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.
Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
(storage.search_index_dirty) - every write surface (REST, WebDAV,
NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
batched single-writer Tantivy commits; queue rows are deleted only
after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
reseeds it from Postgres, which remains the single source of truth
SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.
The frontend renders the snippet under the file name in list view.
New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.
https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
Every upload surface previously wrote each byte to disk twice: the HTTP
body was spooled to a temp file (or assembled from chunk parts), then
mmap-re-read for FastCDC analysis, and finally the new chunks were
written to the blob backend. CDC could not start until the last byte
arrived, so large uploads paid receive + reread + rewrite latency.
The dedup engine now chunks, hashes and settles the stream WHILE it
arrives (fastcdc AsyncStreamCDC + incremental BLAKE3):
- Each batch of distinct chunks is pinned-or-classified by ONE
`UPDATE … RETURNING` (no check-then-bump TOCTOU; pinned chunks can't
be reclaimed mid-upload), and only chunks the store doesn't have are
written — a full dedup hit performs zero content writes.
- Durability before visibility is preserved: one batched fsync sweep,
then one batched INSERT, then the manifest. Identical concurrent
uploads are resolved at the manifest INSERT via ON CONFLICT (the
loser releases its references and becomes a dedup hit).
- A drop guard rolls back pins and surfaces written-but-unregistered
chunks to GC if the request future is cancelled mid-stream.
- MIME sniffing now peeks the first bytes in-flight; client-requested
MD5/SHA-256 checksums are computed by a stream tee — the post-upload
re-read of the assembled file is gone.
All surfaces converge on the new interfaces::upload_ingest helper:
REST multipart, WebDAV PUT, NextCloud PUT, WOPI PutFile, the dedup
endpoint, and both chunked-upload completions (which now stream their
ordered parts straight into the store instead of writing an assembled
file — chunk parts persist until finalize, so completion is genuinely
retryable). The legacy blob re-chunk migration streams from the
backend with no spool file either.
Legacy removed: store_from_file + mmap CDC analysers + temp-path
plumbing through every port (pre_computed_hash, save_file_from_temp,
update_file_content_from_temp), upload_spool + assembled-file
assembly in both chunked services, create_file/update_file byte-slice
variants (no callers), common::temp, the OXICLOUD_UPLOAD_TMPDIR
config, and the memmap2 dependency.
Verified end-to-end against PostgreSQL 16: 8 MB upload (26 chunks),
identical re-upload (dedup hit, zero writes), 3-byte edit re-upload
(26 chunks, 1 written), byte-identical downloads, Range across chunk
boundaries, concurrent identical-upload race (manifest ref 2), and
trash-empty reclaiming exactly the unshared chunk while the shared 25
survive for the edited file. The empty/sub-8KB multipart path found a
post-EOF re-poll panic in the MIME peek (fixed with fuse + regression
test).
https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
A REPORT multiget for a handful of resources previously listed the
whole calendar/address book (every row incl. ical_data/vcard) and
filtered by href in Rust with an O(N×M) substring scan. Large
collections paid full-table latency, RAM and DB CPU on every sync.
- Add find_events_by_ical_uids / get_contacts_by_uids through every
layer (domain repo trait → PG repo → storage port/adapter → use case
→ handler) using one indexed `= ANY($2)` query, mirroring the
existing single-UID lookups.
- Extract UIDs from multiget hrefs with a shared, tested
uid_from_multiget_href helper (percent-decoding, case-insensitive
extension strip, collection hrefs rejected). Exact UID matching also
removes the false positives the old substring filter allowed.
- CalDAV PROPFIND single-event path now uses the existing indexed
get_event_by_ical_uid instead of listing all events.
- Honour the already-documented limit/offset query params on
GET /api/address-books/{id}/contacts: optional ListQuery fields
thread through ContactUseCase::list_contacts to a paginated repo
query. Omitted params keep returning the full book (frontend and
DAV listing/sync paths unchanged).
https://claude.ai/code/session_0193Hff42gaA962wThxMGSd1
The read repository's hash_cache assumed blob hashes were immutable
per file_id, but update_file_content_from_temp remaps the SAME row to
a new hash via swap_blob_hash. After a WebDAV/NC PUT overwrite,
streaming downloads (>=10 MB tier, Range requests, video playback)
kept resolving the OLD blob for the 30 s TTI window — and because
every read refreshes the TTI, a polling client could pin the stale
mapping indefinitely, eventually turning into 500s once the old
blob's refcount hit zero and GC removed it.
The write repository now holds a shared handle to the same moka cache
(clones share storage) and invalidates the entry right after every
content swap and hard delete commits — covering every present and
future caller of the write port rather than one service path. Stale
doc comments claiming the mapping was immutable (and SHA-256) fixed.
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Replace the per-row tree_modified_at bump triggers with AFTER ... FOR
EACH STATEMENT triggers using transition tables: each DML statement now
pays one bump covering the distinct ancestor chains of all affected
rows (locked in id order so concurrent bumps over overlapping chains
cannot deadlock), instead of one chain UPDATE per affected row.
Value-based change detection replaces UPDATE OF column lists (PG
forbids those with transition tables), so EXIF media_sort_date syncs
and no-op updates no longer bump at all, and file moves now invalidate
the source chain as well as the destination.
Also fix a latent bug surfaced while testing this: the descendant
path/lpath cascade (trg_folders_cascade_path) was declared AFTER
UPDATE OF path, lpath, but rename/move statements SET name/parent_id,
and BEFORE-trigger rewrites do not count for UPDATE OF — the cascade
never fired, leaving every descendant folder with a stale path/lpath
after any rename or move. The migration canonically repairs existing
trees and re-creates the cascade on the columns the app actually
writes.
delete_expired_bulk now deletes in LIMIT-ed batches (1000 files / 100
folders per round, each its own implicit transaction) ordered by
trashed_at and served by new partial indexes over trashed rows, so
retention purges no longer hold one unbounded transaction.
Verified against a local PG16 cluster: all migrations apply from
scratch, the new one re-runs idempotently, and a 16-case behavioral
battery passes (chain bumps for insert/update/delete/move/trash, EXIF
invisibility, no-op invisibility, cascade repair, FK-cascade depth
guards, batched purge shape and partial-index plan).
https://claude.ai/code/session_01QxwJDHqQhbMkHK333QtMme
Every WebDAV / CalDAV / CardDAV request paid one full-row user fetch in
require_internal_user_layer just to read `is_external` (and the NC Basic
Auth middleware repeated it right after its own cache hit). That SELECT
includes the `image` column — a data URI of up to 512 KiB — so a sync
client issuing hundreds of PROPFINDs per minute dragged hundreds of MB
of avatar bytes out of Postgres to evaluate a boolean.
- New `UserFlags { role, is_external, active }` + a repo query selecting
only those three columns (inherent method, mirroring `update_image`).
- `AuthApplicationService::get_user_flags`: moka cache, 30 s TTL,
10k capacity. `change_user_role` / `set_user_active` invalidate
eagerly, so admin changes still apply immediately; anything else is
visible within the TTL — preserving the documented "no token rotation
needed" semantics at a per-request cost of zero DB round-trips when
warm.
- `require_internal_user`, `require_admin_user` and the NC Basic Auth
external check now go through the flags lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
Resolving the stable numeric oc:fileid for every child in a NextCloud
listing issued one `INSERT ... ON CONFLICT DO UPDATE` per entry — a write
(row rewrite + WAL + dead tuple) even when the mapping already existed.
A Depth:1 PROPFIND of a folder with N children meant N sequential write
round-trips on a read-only operation that sync clients repeat constantly.
- Repository: replace the single `get_or_create` (DO UPDATE) with
`get_or_create_many` — one idempotent bulk `INSERT ... SELECT unnest(...)
ON CONFLICT DO NOTHING` (existing rows untouched) plus a single
`SELECT ... WHERE object_id = ANY(...)`. Two statements instead of N.
- Service: add an Arc-backed moka cache (uuid -> i64; the mapping is
immutable, so warm entries never go stale) and batch APIs
`get_or_create_file_ids` / `get_or_create_folder_ids` that only query
the misses. Warm listings cost zero queries.
- Handlers (PROPFIND, REPORT favorites/search, trashbin, OCS unified
search): pre-resolve all ids in two batched queries — file and folder
run concurrently via `tokio::join!` — and turn the XML/JSON emission
into a synchronous map lookup.
https://claude.ai/code/session_01Dp3oWon5GBMVn4j3QXZdgx
webdav encode_uri_path runs on every PROPFIND href and did
.map(...).collect::<Vec<_>>().join("/"), allocating a String per segment plus
a joined Vec. Write each utf8_percent_encode Display adapter straight into a
single preallocated String. Behavior is identical (split on '/', encode each
segment, join with '/'), including leading/trailing-slash edge cases.
subject_group list / list_with_counts each issued a second SELECT COUNT(*)
round-trip for the total. Fold it into the page query via COUNT(*) OVER() —
the pattern folder_db_repository already uses — halving the round-trips.
total_count is read from the first row and is 0 on an empty page, matching
folder_db_repository's documented convention.
https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV
GET /api/auth/me ran a synchronous O(N) SUM(size) over all the user's
files plus an unconditional UPDATE of auth.users on every call — one of
the most frequently hit endpoints — adding per-request latency, DB write
load, dead tuples and WAL even when nothing changed.
- /api/auth/me now serves the cached storage_used_bytes column instead of
recomputing it inline.
- New StorageUsageService::start_reconciliation_job runs a periodic sweep
on the maintenance pool that keeps the cached value current for every
mutation (uploads, deletes, trash), so freshness no longer depends on
hitting /me. Interval via OXICLOUD_STORAGE_USAGE_RECONCILE_SECS (default
600s, floored at 30s; first sweep deferred one interval to avoid boot load).
- update_storage_usage only writes when the value actually changes
(IS DISTINCT FROM), so the sweep produces no dead tuple / WAL on no-ops.
- New covering partial index idx_files_user_size_active makes the usage
SUM an index-only scan instead of a heap scan over all the user's files.
Also collapse the same pre-existing clippy collapsible_else_if in
carddav_handler that blocks the -D warnings gate on this base.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Edouard Vanbelle