Every thumbnail request paid an uncached storage.file_attached_blobs
point query before it could answer — including 304 revalidations and
RAM thumbnail hits, where the ETag path (thumbnail_content_id) probes
the row every time and tier 2b probes it again with the same key. A
photos grid revalidating 60 thumbnails per visit meant 60+ point
queries per browse, repeated on every visit.
find_attached_blob now reads through a process-local moka cache in
DedupService, keyed by the row's (file_id, kind, variant) PK, holding
positive and negative entries (most files have no attached preview, so
the negative side carries the win). Two rules keep it honest:
- DB faults are surfaced as Err and never cached — a transient outage
cannot freeze "no attached blob" into a negative entry (a read
failure is never proof that data is absent). The public signature is
unchanged; the SQL body moved to find_attached_blob_uncached.
- Writes invalidate eagerly: store_attached_blob and the Inserted arm
of store_attached_blob_if_absent on success, and deletions via
ThumbnailRefreshHook::on_file_deleted, which all three production
delete paths (single file, folder cascade, trash clear) fire after
the DELETE commits. The 60s TTL bounds only what the process cannot
see (bare SQL, copy_file_satellites races).
The Nextcloud preview endpoint rides the same lookup and benefits
identically. Five in-memory contract tests pin the cache behaviour,
including the fault-not-cached rule.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Step 12. The two tables existed with COMMENT ON text, but nothing
explained the pair together — and the relationship is the part that
matters: they hold the same kind of artifact under two different keys,
and the keying difference is a security boundary.
Content keying shares one derivation across identical bytes, which is
free and correct for something the server derived. Apply it to
user-supplied bytes and uploading a file whose content matches someone
else's lets you replace the preview they see. A single table with a
`kind` column cannot express that: the key has to be one thing or the
other, and either choice is wrong for half the rows. The split is the
enforcement, which is why the two import jobs each refuse the other's
filenames rather than one job handling both trees.
Covers structure, negative rows and what may not become one, the NULL
trap (comparison against NULL silently excludes negative rows —
correct for refcounts, wrong for dangling checks, fatal for
enumeration; all three have been hit), why `variant` carries the format
on one table and not the other, lifecycle and which consistency job
covers which failure, worked examples, and a decision rule for adding a
third artifact type.
Records `content_type`-as-key as a rejected alternative: reasonable
until negative rows made the column nullable, and PostgreSQL does not
allow a nullable column in a primary key. Worth writing down because a
later feature retroactively eliminated an option that would have looked
sound at the time.
Named for the two things rather than "satellite tables" — that is
internal shorthand nobody would search for, while `derived` and
`attached` are the words the schema and jobs already use. Mentioned once
in the intro so the code's collective noun still resolves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`blobs_consistency` probed `blob_exists` once per row and, under
`?deep=true`, read and re-hashed every blob. `backend_consistency`
already reports the same `blob_missing_from_backend` from its
merge-join — so the probe was duplicated work that found strictly less
(a DB walk cannot see backend-only orphans by construction) at N round
-trips instead of one enumeration. Every scheduled sweep paid for it.
All three physical checks move to `backend_consistency`:
* `blob_missing_from_backend` was already there; the duplicate is gone.
* `blob_corrupted` / `blob_unreadable` hook the matched arm of the
merge-join, which holds exactly the key pairs worth reading. Guarded
by `in_range` so a pair past the horizon is not read twice, and
`params.deep` is persisted on a fresh run and read back on resume so
a paused deep scan does not silently continue shallow.
Deep mode belongs there because it is backend work end to end: the
only DB input is the hash. Keeping it in `blobs_consistency` forced
that tenant to carry a backend for one flag.
What remains is the half that needs no backend: `refcount_mismatch`
and its repair. The constructor drops from five parameters to two —
no backend, no storage_entries, no storage_path_fallback — and
`?storage=<name>` / `?deep=true` are now inert there, which the
job description says outright.
`affected_files` is needed by both tenants, so it moves to a shared
`blob_diagnostics` module rather than being copied.
`PROBED_STORAGE_PARAM` moves to `backend_consistency`: it was defined
in `blobs_consistency` and re-exported, which is backwards once the
DB-only tenant has no entry to scope. The create-grace window goes
with the probe — it existed to avoid flagging a blob whose bytes had
landed before its row, and the refcount comparison reads one
consistent snapshot.
Known cost: `backend_consistency` returns `backend_unenumerable` on
Azure and mid-migration, so on those configs missing bytes now go
unreported where the per-row probe caught them. That argues for the
Azure enumeration impl, not for keeping the probe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
permit an admin to specify `oidc` only as the only method to login/register
note that if OIDC is enabled, the engine always append oidc in OXICLOUD_AUTH_METHODS
we could move to an explicit declaration in a major release
- grants: permission moved to roles
- new resources (Drive, Caldav, Carddav, Playlist) now using ReBAC
- expired shared now cleaned up
- drive visible in Webdav
- new login/registration options (domain allow list, policies, etc)
- upgrade of external user into internal user
The normalized cursor-paginated /resources API is live and the bundled
frontend already uses it for favorites, recent, trash and grants. These five
deprecated old-format endpoints had no remaining frontend or protocol
consumers (the Nextcloud handlers call the service layer directly, not these
HTTP routes), so remove them for a uniform API and less duplicate listing
logic:
- GET /api/folders/{id}/contents → use /api/folders/{id}/resources
- GET /api/folders/{id}/contents/paginated → use /api/folders/{id}/resources
- GET /api/favorites → use /api/favorites/resources
- GET /api/recent → use /api/recent/resources
- GET /api/trash → use /api/trash/resources
Removes the HTTP handlers, their routes, OpenAPI path registrations, and the
now-dead list_folder_contents{,_paginated}_impl helpers + unused imports. The
underlying service methods (favorites_service.get_favorites,
trash_service.get_trash_items, etc.) are KEPT — the Nextcloud OCS/trashbin
handlers depend on them.
Deliberately NOT removed: GET /api/folders/{id}/listing. It is still the Files
view's primary data path and offers ETag/304 conditional caching plus
one-shot favorite/share badge sets that /resources does not yet provide;
migrating it needs a separate parity pass on /resources first.
Updates the OpenAPI structure test and the two docs that referenced the
removed paths. `cargo clippy -D warnings` clean; 443 lib tests pass; OpenAPI
regenerates with the 5 paths gone and the /resources replacements present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
remove create_home_folder() & ensure_home_folder()
now: on_user_created() and on_user_login both() call provision_if_needed()
which calls **create_personal_drive_atomic()**
add a helper to find Personal drive for a user and also it's root directorry
1. Identity model — email is the identity, three optional credential slots (username, password_hash, oidc_subject), the @-in-username ban that makes namespaces provably disjoint.
2. Login dispatcher — @-in-input decides the path. Single DB lookup.
3. Login paths — four ways in (username+password, email+password, email+magic-link, OIDC redirect) with their pre-conditions.
4. Magic-link eligibility ladder — the three-branch table with OIDC unconditional reject, password flag-gated, no-credential allowed.
5. Registration paths — four ways (/api/auth/register with full creds, email-only, grant-by-email invitation, OIDC JIT).
6. Anti-enumeration — per-endpoint table showing what each surface returns and why; explicit note that instance-wide policy stays visible.
7. Security trade-offs — mailbox-as-bypass (lenient vs strict), no native MFA, magic-link as bearer token, enumeration via timing.
8. Rate limits — the five caps from PR 12 + classic auth, with env knobs.
9. Audit events — table of 12 event names with reason keys; every entry verified grep-able in src/.
10. Migration path — what existing instances see when PR 16-20 land (forward-only, non-destructive).
11. Future direction — login_strategy — the seven-row matrix sketch as the explicit-policy north star.
12. Out of scope — TOTP/WebAuthn, external→internal promotion, session_kind, differentiated session TTL, OCM, email-verified gates (PR 23), device-bound tokens (PR 22), anti-enum timing parity, per-user opt-out, login_strategy implementation.
13. Related documents — cross-refs to magic-link-auth, ReBAC, share-integration, env-vars.
prepare identity service for external users, support of:
- magic_link (url challenge via email)
- self issued oidc (eventually social login)
- open cloud mesh
- AuthzCacheLifecycleHook — invalidates the user_groups_cache Moka entry on logout/delete.
- SessionRevocationLifecycleHook — explicit per-session firing of on_user_logout (currently per-call); session revocation inside the user-delete transaction.
- DeletionMode-driven policy in HomeFolderLifecycleHook::on_user_deleted (trash vs hard-delete based on AdminDelete / GdprPurge).
- Refactor delete_user_admin to expose a transaction handle so on_user_deleted can abort atomically.
- Add src/application/dtos/cursor.rs with three shared types:
· PageCursor trait — default base64url+JSON encode/decode; one bare
impl line per cursor struct
· CursorQuery struct — standard limit/cursor/sort_by query params with
limit_clamped() and decode_cursor<C>() helpers; compose via flatten
· CursorListResponse<T> — standard {items, next_cursor?} envelope with
from_oversized() and with_cursor() builders
- Migrate GrantCursor to impl PageCursor (remove duplicate encode/decode)
- Update GET /api/grants/incoming/resources:
· SharedWithMeQuery now embeds CursorQuery via #[serde(flatten)]
· Replace file/folder nullable pair with ResourceContentDto (untagged
enum) under a single always-present 'resource' field
· SharedWithMeDto is now a type alias for CursorListResponse<SharedWithMeItemDto>
· Handler uses q.paging.limit_clamped() and decode_cursor<GrantCursor>()
- Add docs/architecture/resource-listing.md — authoritative contract for
all listing endpoints (cursor design, SQL keyset WHERE, sort_by naming,
Rust + JS skeletons, compliance table, migration guide)
- Register doc in VitePress sidebar and architecture index
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* make more coherent lifecycles
* remove specific implementation on different handlers (they do not need to know existence of ThumbnailSerice nor AudioMetadataService)
* reduce risk of orphean objects
* ensure additional services are correctly wired (ex: Thumbnail generation was not covering all upload cases)
* more details on docs/architecture/file-and-blob-lifecycle.md :
```rust
// application/ports/file_lifecycle.rs
pub trait FileLifecycleHook {
fn on_file_created(file_id, blob_hash, content_type, is_new_blob);
fn on_file_updated(file_id, blob_hash, content_type);
fn on_file_copied(file_id, blob_hash, content_type, source_id)
fn on_file_deleted(file_id);
}
// application/ports/blob_lifecycle.rs
pub trait BlobLifecycleHook {
fn on_blob_created(blob_hash, content_type);
fn on_blob_deleted(blob_hash);
}
```
cjw
Edouard Vanbelle