DeadPropertyStore let PROPPATCH set any namespace/name verbatim,
incl. names the server itself emits as live state (DAV: entirely,
plus oc:/nc:/ocs: names used by write_file_response /
write_folder_response). That either forges a live prop or stores
dead rows nothing ever reads. is_protected_property() denylists
them; both PROPPATCH handlers (native + NC) now return per-property
403 instead of storing. oc:favorite stays writable via its existing
special-case, which runs before the protection check.
add `include_in_photo_index` and `include_in_music_index` policies
both true for default personal drive
photo is implemented
music is not yet implemented
NC PROPPATCH only special-cased oc:favorite, silently dropped every
other prop while claiming 200. PROPFIND had no dead-prop lookup at
all. Reuse native /webdav/ dead-prop plumbing (now pub(crate)) so NC
surface gets real RFC 4918 §4.2 storage. Missing resource on
PROPPATCH is now 404 instead of a fake success no-op.
Complete RFC 4918 litmus test suite compliance (basic 16/16,
copymove 13/13, props 30/30) by fixing six categories of failures:
- PropPatchOp: parse PROPPATCH in document order (RFC 4918 §9.2)
so that remove-then-set and set-then-remove yield different results
- Null namespace: resolve_name now correctly maps xmlns="" to empty-ns
QualifiedName instead of falling through to DAV: namespace
- Unicode chars: handle quick-xml 0.39's Event::GeneralRef for character
references (𐀀) — the parser emits these as GeneralRef, not Text
- MOVE preserves dead props: call rename_resource on MOVE, clearing
stale destination data even when source has no registered properties
- Malformed PROPFIND: return 400 when body lacks a complete <propfind>
element (test 2: bare <foo>, test 3: invalid xmlns:prefix="" binding)
- PROPPATCH document order: process ops via Vec<PropPatchOp> instead of
separate (sets, removes) to honour interleaved remove/set sequences
- Return unknown properties in a separate 404 propstat (RFC 4918 §9.2)
via folder_prop_is_known/file_prop_is_known classifiers
- Detect unclosed <propfind> element and return 400 Bad Request
- Add write_folder/file_entry_with_dead_props to include stored dead
properties in PROPFIND responses
- Change write_folder/file_requested_props to accept &[&QualifiedName]
to avoid clone overhead when using partition output
Add an in-memory DeadPropertyStore backed by RwLock<HashMap> that
stores arbitrary client-supplied XML properties per resource path.
Wire it into AppState so PROPPATCH can persist dead props and PROPFIND
can retrieve them across requests.
Mounts are now configurable at runtime without DB surgery.
Backend (admin-gated, /api/admin/external-mounts):
- GET list all configured mounts
- POST create: validate the provider config up front, create a mount-root
folder under the admin's drive, insert the row, hot-reload the registry
- DELETE remove the mount row + its root folder (host content is left intact),
then hot-reload
- ExternalMountRepositoryPort gains create/delete (PG impl); registry.reload()
runs in-process so changes are live immediately
- audit lines event="external_mount.config" action=create|delete
Frontend:
- admin.ts endpoints: listExternalMounts / createExternalMount / deleteExternalMount
- new "External Mounts" tab in the admin page: add form (name + host path +
read-only) and a list with delete
Integration test for the repo create/delete round-trip (testcontainers).
Mounts are now browsable and downloadable over both WebDAV surfaces
(/webdav/ and NextCloud /remote.php/dav) and NextCloud clients. The work
funnels through the path-based service methods all three surfaces already use,
so both WebDAV handlers gain mount support with no handler changes.
- get_folder_by_path / get_file_by_path resolve a path that descends past a
mount root to a synthetic ext: DTO (the mount root itself stays a real row)
- list_folders_paginated_with_perms / list_files_batch_with_perms branch to the
provider, so PROPFIND Depth:1 enumerates mount directory contents in both
WebDAV handlers
- get_file_stream / get_file_range_stream branch ext: file ids to the provider,
so WebDAV GET streams mount content (Pin<Box<dyn Stream>> re-boxed)
- router injected into FileRetrievalService; new MountRouter::find_path delegates
to the registry's (drive_id, mount_path) index
- WebDAV mkdir/delete/move/rename by path work via path→ext:id→the P2 service
methods (no extra wiring)
Fixes a leading-slash normalization bug in the registry path index: materialized
folder paths arrive both as `Personal/Media` and `/Personal/Media`; keys and
lookups now normalize the leading slash (would have broken real WebDAV paths).
Known follow-up: WebDAV PUT (upload/update by path) still ingests to the CAS;
streaming a WebDAV PUT straight to the provider needs a pre-ingest branch in the
two WebDAV PUT handlers (mirrors the REST upload branch).
Integration test: get_folder_by_path/get_file_by_path resolution, PROPFIND
Depth:1 folder+file listing, and content streaming on a real mount + Postgres.
Adds full CRUD on mount contents, mirroring the P1 read pattern (handlers/
services classify; authorization stays in the service via the mount-root
folder grant; the provider does the I/O).
- mkdir / rename / delete / move-within branch inside FolderService and
FileManagementService (router injected into both)
- streaming upload via a new ExternalUploadService: the upload handler detects a
mount destination BEFORE the CAS ingest and streams the multipart body
straight to the provider (no BLAKE3/dedup). `write_stream` now takes a
lifetime-bound boxed stream so the borrowing multipart field can be passed
without buffering.
- deletes on mounts are permanent (no trash): the trash-first folder handler
routes `ext:` ids straight to the provider delete; file delete goes through the
branched delete_and_cleanup
- cross-backend move/copy (mount ↔ native, or between mounts) is forbidden
(UnsupportedOperation); the mount root itself cannot be renamed/moved/deleted
- every mutation emits a `target:"audit" event="external_mount.write"` line
- shared mount_dto builders synthesize FolderDto/FileDto from a provider MountStat
Tests: 529 unit + integration tests for mkdir/rename/delete, file rename/delete,
streaming upload, cross-boundary forbid, and stranger-denied — all against real
Postgres + a real provider (testcontainers).
Adds the foundation for external file mounts: admin-configured backends
(raw host filesystem in v1; sftp/webdav/… as future provider kinds) surfaced
as a folder inside a user's drive. Mount contents are virtual/live-passthrough
— read straight from the backend, never stored in storage.files — and are a
deliberately separate, limited storage type (no dedup/sharing/trash/search).
The feature is dark by default (OXICLOUD_ENABLE_EXTERNAL_MOUNTS=false).
P1 scope (this PR): data model, the pluggable provider abstraction, and the
read-only REST surface (mount listing + download). Read-write (P2),
WebDAV/NextCloud path resolution (P3), and the admin UI (P4) follow.
Core model
- Mount root = a real storage.folders row; authorization for everything inside
collapses onto that folder UUID (ltree-ancestry grant cascade).
- Children are virtual, addressed by ext:<mount_id>:<base64url(node_id)> where
node_id is provider-owned and opaque to the rest of the system.
- A lock-free (arc-swap) MountRegistry maps mount-root UUID -> provider; a thin
MountRouter::classify() is the single cheap hook handlers call before parsing
an id as a UUID. With no mounts configured it always returns Regular, so
existing code paths are unchanged.
Added
- migrations/20260805000000_external_mounts.sql (storage.external_mounts, kind + config JSONB)
- domain/services/external_mount_id (id envelope + virtual etags)
- application/ports/external_mount_ports (ExternalMountProvider, MountProviderFactory, repo port)
- infrastructure local_fs_mount_provider (tokio::fs, symlink-escape-safe) + factory
- application MountRegistry + MountRouter, pg ExternalMountRepository
- DI wiring (AppState.mount_router), FeaturesConfig.enable_external_mounts
- listing branch (FolderService::list_mount_dir_with_perms + folder_handler) and
download branch (FileRetrievalService stat/open mount methods + file_handler)
Authorization stays in the service layer (authz.require(Resource::Folder(mount_id)));
handlers only classify. Cross-backend operations are out of scope for P1.
Tests: 529 unit tests + 5 testcontainers integration tests (real Postgres 17),
including end-to-end authorization (owner allowed, stranger denied). Line
coverage of the new modules is 84–100% (cargo-llvm-cov). Known gap:
file_handler::download_mount_file (HTTP glue) needs a full-app test (P4).
- permit shared drive creation from oxicloud admin (for now)
- prepare other personal drive creation (Not implemented), need to validate
quota policies and strategy first
- add hurl test to verify permissions
initially the recent was done client side
recent files are now directly updated on serverside when accessing a file
note: nextcloud and webdav voluntary not included
- quota per drive
- add 2 internal API endpoints to test purpose
disabled by default, enable it via `OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS=true`
this enable:
/api/admin/internal/trigger-sweep
to sweep the trash and recalculated quota
/api/admin/internal/trigger-gc
to garbage orphan blobs
use full for end to end tests and validate lifecycles
OptimizedFileContent::Mmap was constructed nowhere — the documented "Tier 2:
memory-mapped I/O (10-100 MB)" path was never wired, so optimized_inner only
ever returns Bytes (<10 MB) or Stream (>=10 MB). The variant survived only as
an enum case plus two dead match arms in the file and share download handlers.
Remove the variant and its arms, and fix the now-misleading retrieval-service
tier docs (everything >=10 MB streams via CDC chunk reassembly with the
backend read-ahead; there is no mmap tier). Behaviour is unchanged — the
deleted arms were unreachable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
The local backend inherited the trait's conservative read_prefetch() = 1
(strictly sequential chunk reassembly), while S3/Azure already use 8. The
prior rationale was that concurrent opens over scattered content-addressed
chunk files turn one sequential read into competing random I/O ("slower
cold"). Benchmarked that assumption with examples/bench_blob_prefetch
(sweeps the buffered(N) depth over a real LocalBlobBackend under disk-bound
vs network-bound consumers and warm vs cold page cache).
Result on SSD-class storage (median MB/s vs N=1):
warm disk-bound N=2 +11.8% N=8 +3.9% N=16 -4.4%
cold disk-bound N=2 +7.2% (no cold regression on SSD)
network-bound (throttled) ~0% at any N — the socket, not the disk, caps it
So N=8 is wrong for local (leaves gain on the table, risks HDD seek thrash)
and the network-bound win the analysis assumed doesn't materialize: buffered()
here overlaps the per-chunk File::open (cheap on local disk), not the data
read. N=2 captures most of the disk-bound gain — which covers localhost/LAN
downloads AND the internal blob reads that drain as fast as the disk delivers
(thumbnail render, transcode, ZIP export, content extraction) — at the lowest
fan-out. Env-tunable via OXICLOUD_LOCAL_READ_PREFETCH (set 1 on seek-bound
HDDs to restore the old behaviour; raise on fast NVMe). Signature unchanged,
so all ~16 LocalBlobBackend::new call sites are untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JG5yYZ9s868mJwqT2Qz7ez
Videos now get a thumbnail generated eagerly server-side on upload, through
the same WebP/blob-hash pipeline as photos — instead of the old browser path
that only ran when the Photos grid first rendered a video tile, re-downloaded
the whole video to seek a frame, and PUT 3 JPEGs back (and produced nothing at
all for HEVC/.mov, which a browser <video> cannot decode).
- New VideoFramePort (application) + FfmpegVideoFrameService / NoopVideoFrameService
(infrastructure): shell out to the system ffmpeg (no compile-time libav dep),
extract one representative frame as PNG, bounded by its own semaphore + a
per-process timeout + kill_on_drop. Noop when ffmpeg is absent/disabled, so
videos degrade gracefully to no thumbnail.
- ThumbnailRefreshHook.on_file_created routes video/* to
generate_video_thumbnails_background: stream the (decrypted, reassembled) blob
to a size- and time-bounded temp file on the data volume, extract a frame, and
reuse the shared render_and_persist_all_webp helper — so video thumbnails are
WebP, blob-hash keyed (dedup'd) and content-negotiated, exactly like photos.
- GET thumbnail serves the video's WebP to every client (byte-sniffed
Content-Type); a genuine miss returns 204.
- Config: OXICLOUD_ENABLE_VIDEO_THUMBNAILS (default true, needs ffmpeg detected
at startup) + OXICLOUD_FFMPEG_PATH / _CONCURRENCY / _TIMEOUT_SECS / _MAX_MB.
- Dockerfile installs ffmpeg in the runtime image.
- Frontend: drop the client-side generateVideoThumb/frameFromVideo re-download
path; the server is now the source of truth.
Benchmark (examples/bench_video_thumbnails.rs, needs ffmpeg): 4/4 codecs incl.
HEVC/.mov produce a thumbnail server-side (was 0% for HEVC); ~50-70 ms/frame in
the background; ~3.9 KB preview WebP; up to ~23x less per-first-view transfer on
the test corpus (far more on real multi-MB clips). Methodology in
benches/VIDEO-THUMB.md.
Hardening from an adversarial review: video render holds the decode_semaphore
like the image path; the ffmpeg scale filter bounds both dimensions; the blob
stream has a timeout; the temp file lives on the data volume; the size cap uses
saturating_mul.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Thumbnails are now generated eagerly as lossy WebP (the primary codec) and
served to clients that advertise `Accept: image/webp`; JPEG is kept as a lazy
fallback for older clients and NextCloud, generated on first request and then
cached like WebP.
- ThumbnailFormat{Webp,Jpeg} enum threaded through encode/render/generate, the
on-disk path ({hash}.webp / {hash}.jpg), the moka cache key
(file_id, size, format), and cleanup (both formats removed).
- file_handler: parse Accept -> format, format-keyed ETag, `Vary: Accept` on
every response (incl. 304) so shared caches never serve the wrong codec;
Content-Type is byte-sniffed (infer) so it always matches the bytes.
- preview_handler (NextCloud) pins JPEG.
- webp = "0.3" (vendored libwebp via cc, no system dependency).
WEBP_QUALITY=82, chosen via a quality sweep (bench Table E1): SSIM within
~0.005 of JPEG q80 (imperceptible at thumbnail scale) for ~62% fewer bytes. On
the photo-realistic bench corpus the full set (3 sizes x 3 photos) drops 65.6%
(213->73 KB); real photos with edges/text land nearer ~25-40%. Encode is +5ms,
paid once in the eager background generator (off the request path).
The bench corpus is now photo-realistic (per-channel sums of low-frequency
sinusoids) instead of white noise, which had distorted codec byte ratios.
Methodology + numbers in benches/WEBP.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OIDC SSO login intermittently ended on a 403 "Invalid or expired OIDC state
— possible CSRF attack" even though the login had already succeeded
server-side.
Root cause: the (now-removed) legacy vanilla-JS frontend registered a
`/sw.js` service worker that, with navigation preload enabled, double-fetched
the top-level navigation to `/api/auth/oidc/callback`. The OIDC `state` is
single-use, so the first callback consumed it and logged the user in while
the duplicate (~0.4s later) found the state gone and returned the 403 the
browser rendered.
Backend — idempotent callback: after a successful web login, remember
`state -> exchange_code` in a short-lived (120s) cache. A duplicate callback
whose state was already consumed now replays that same redirect instead of
403-ing, returning the cached result directly without re-running the IdP code
exchange (the authorization `code` is single-use too). Keyed by the
unguessable 32-byte state, so it adds no new attack surface and fixes the 403
for everyone — including browsers still running a stale legacy service worker.
Frontend — evict the stale worker: the current SvelteKit app registers no
service worker, so fresh clients can't double-fire. But a browser that
previously loaded the legacy frontend still has `/sw.js` registered and
controlling pages (and `/sw.js` now 404s, so vendor self-cleanup is
inconsistent). killLegacyServiceWorker() runs first in the root layout's
onMount: it surgically unregisters only `/sw.js` workers, drops only the
legacy `oxicloud-cache-*` caches, and reloads once (guarded).
Fixes#510.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An audit (backend /api routes vs SvelteKit frontend usage, adversarially
verified across the whole repo) found these 5 routes have ZERO callers — no
frontend, no test, no protocol layer, no internal caller — and are superseded:
- GET /api/folders/paginated no-op duplicate of GET /api/folders
(discards the page arg); superseded by
the cursor-paginated /{id}/resources.
- POST /api/dedup/upload superseded by /api/files/upload, which
does the identical CDC dedup ingest.
- POST /api/people/{id}/hide the hide-person toggle was never built
into the UI (is_hidden never read).
- GET /api/admin/settings/general never called anywhere.
- GET /api/admin/settings/registration only the PUT is used; the GET had no
caller (PUT kept).
Removes each route, its handler + _impl, the now-orphaned DedupUploadResponse
DTO + its two serialization tests, the set_hidden service method (only caller
was hide_person), and the OpenAPI path/schema registrations.
Also cleans 4 stale markers: two #[allow(deprecated)] that no longer suppress
anything (zero #[deprecated] remain), the "Legacy folder endpoints (contents,
listing)" comment (both already removed), and a "Re-export AppError for backward
compatibility" comment describing a re-export that doesn't exist.
Net -323 lines. The 31 other unused-by-frontend routes (device-code auth,
CardDAV contact-groups, people/photos & music WIP, dedup/admin debug, i18n,
openapi.json) are intentional surface and were left untouched.
cargo clippy --all-features --all-targets -D warnings: clean. cargo test: 446 passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend — tail latency & throughput:
- FileContentCache, image transcode, and search now use moka single-flight
(try_get_with / get_or_load) so N concurrent misses for the same key
collapse to one disk read / transcode / query instead of a thundering herd.
Microbenchmark (128 concurrent on one hot key): 128 loads / p99 ~1023ms
before vs 1 load / p99 ~32ms after.
- DB: configurable per-statement timeout on the primary pool
(OXICLOUD_DB_STATEMENT_TIMEOUT_SECS, default 30; maintenance pool exempt) so
a runaway query can't pin a connection and starve the pool.
- DB: background pool-saturation monitor
(OXICLOUD_DB_POOL_MONITOR_INTERVAL_SECS) that WARNs as the primary pool nears
exhaustion — the early signal before tail latency cliffs.
- mimalloc: set MIMALLOC_PURGE_DELAY=0 (Dockerfile + compose) so freed pages
return to the OS and RSS tracks the live working set; benchmarked on
musl/aarch64 at ~400MB reclaimed vs 0MB with the default.
Frontend — UI / i18n fixes:
- i18n: fix literal "{{count}}" and "{{percentage}}/{{used}}/{{total}}" in the
selection toolbar and storage line — the call sites passed param names that
didn't match the locale placeholders; unify on `count` and pass the storage
template its params. Add es files.selected_count.
- sidebar: hide the drive picker when there's only one drive (the redundant
"Personal" row); remove the coloured left accent on the active nav item.
- logo: stop clipping the cloud's left bulge — viewBox recentred on the cloud's
true bbox with proportional SVG size so it keeps the same rendered scale.
- user menu: drop the default <a> underline on the link rows.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The folder listing now carries the favorite/share badge state for exactly the
items it returns, so the files browser stops fetching favorites and outgoing
shares separately. This removes the last per-navigation badge round-trips AND
fixes the correctness hole of the previous approaches: badges were derived from
only the first 200 global favorites / shares, so a favorited or shared item
outside that window showed no badge. Now every listed item is correct, and the
work is scoped to the items on screen.
Backend (`GET /api/folders/{id}/listing`):
- `FolderListingDto` gains `favorite_ids` and `shared_ids` (sorted) — listing-
level metadata, so no churn to the many FileDto/FolderDto constructors.
- The handler computes both with two batched, index-backed queries run
concurrently: `FavoritesService::favorited_ids` (auth.user_favorites, ANY) and
`PgAclEngine::shared_resource_ids` (storage.role_grants by granted_by + ANY,
which already covers public links as 'token' grants — same membership the
/grants/outgoing/resources endpoint exposes). Both fold into the ETag.
- Public-share browsing passes empty sets (anonymous, read-only context).
Frontend:
- `listFolder` reads `favorite_ids` / `shared_ids`; the files view seeds local
badge sets straight from the listing and updates them optimistically on
favorite toggle / batch / share creation (via ShareDialog's `onshared`).
- Removes the session `badges` store + its fetches entirely — the listing is now
the single, authoritative, fetch-free source.
Net: favorite/share badges cost zero extra client requests per navigation and
are correct regardless of how many favorites/shares the user has. Validated:
cargo check + clippy -D warnings (backend; integration tests need Postgres,
unavailable here), frontend npm run check + unit tests, and a headless render of
the real files route (list + grid) with the new flags present — no errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M8Vb9QHmLZnEMzHz7MrFy6
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
Bring chroot to nextcloud capability: login on Nextcloud via username="{user}~{folder-uuid}
Doing a such login will chroot the folder folder-uuid
If user has several folder as root (parent=None), the login flow will request which
folder user want to chroot
Edouard Vanbelle