507bc2e98df24ce7c6db3fde2d14bb2296d0455c
10 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Edouard Vanbelle
|
507bc2e98d | feat(recoverable-job): add backend_consistency (storage) | ||
|
DioCrafts
|
d66956824c | Optimize storage, GC, and upload hot paths | ||
|
Claude
|
5b2bb8f883 |
perf(round26): drive-policy JSONB decode, CachedBlobBackend shard-dir pre-create, delta-upload foldhash
Three benchmark-gated optimizations from the ROUND25 backlog (benches/ROUND26.md),
each with a BEFORE/AFTER gate that rolls back if AFTER does not beat BEFORE:
- P1 drive_pg_repository policy reads: decode d.policies through
sqlx::types::Json<DrivePolicies> (one from_slice over the raw JSONB bytes)
instead of a throwaway serde_json::Value DOM + DrivePolicies::from_value —
6 -> 0 allocs/read, 2.77x wall. A shared policies_from_row helper preserves
the lenient unwrap_or_default fallback (malformed bag -> all-false).
- D1 CachedBlobBackend: pre-create the 256 {00..ff} shard dirs at initialize()
(mirroring LocalBlobBackend, reusing HEX_PREFIXES) and drop the redundant
per-write create_dir_all on already-existing shards — ~45us + a blocking-pool
dispatch removed per cache write on cached-remote deployments.
- G1 delta-upload have/need hash sets (distinct_hashes, authorize_chunk_download):
SipHash -> foldhash::quality::RandomState — a fast hasher that stays
DoS-resistant via a per-instance random seed, the required property for the
attacker-controlled 64-hex client hashes — 2.37x wall on a 40k-hash
negotiation. foldhash was already in the lockfile transitively (hashbrown).
Tested and REVERTED (kept as-is): moving the moka eviction unlink off the reactor
via spawn_blocking. The benchmark refuted it — on the local cache dir the
spawn_blocking dispatch (~20us) costs more than the inline unlink (~7us) it would
replace. See ROUND26.md §D2.
Adds bench_round26_{micro,diskio,hasher} (counting allocator / async wall / wall).
Verified: cargo fmt clean, cargo clippy --features bench -D warnings clean,
cargo test --lib --features bench = 529 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L8gs91AhmazoxMsDcNk3KT
|
||
|
Claude
|
50eca0627f |
perf: round 12 — auth write-path narrowing, fused quota gate, moka blob-cache index, media single-read, sized listing JSON
Benchmark-gated round (benches/ROUND12.md; every change ships with a BEFORE/AFTER harness + equivalence gates, one candidate rejected by its own bench): DB / query shapes (bench_round12_queries): - NC sharee search: username-only projection instead of the 21-column row (incl. the <=512 KiB avatar) per match, + gin_trgm_ops indexes on auth.users for the leading-wildcard ILIKE (4.98x; 54.7x with index). - Password login: delete the redundant full-row update_user — create_session already stamps last_login_at in its own txn (4.45x per login). - Email-verified stamp: narrow conditional UPDATE (8.9x); OIDC repeat login now compares profile state in memory and issues ZERO queries when nothing changed (was: full 17-column rewrite per login). - Refresh rotation: revoke+insert+stamp fused into one transaction via new rotate_session port method (1.18x). - WOPI CheckFileInfo / authorize_wopi_access: require(Read) + get_file + check(Update) overlapped with tokio::join!, original result precedence (cold 1.34x). - Upload quota gate: user-envelope + drive-cap checks fused into ONE round-trip (check_upload_quotas) — the NC chunked PUT pays this per chunk (1.81x, 2 -> 1 queries/chunk); shared verdict evaluators keep error shapes byte-identical. CPU / allocs (bench_round12_micro): - sized_json: pre-sized listing serialization replacing axum Json's 128 B seed + doubling-realloc chain on files/folder-resources/photos/search responses (1.40x, 13 -> 2 allocs per 500-row page; byte-identical). - Security headers: 4 SetResponseHeaderLayer folded into the CSP middleware pass (5 layers -> 1; 1.43x per request, -26 allocs; header set gated byte-identical incl. 304s). - Media capture-metadata: single-read extraction — nom-exif now parses the buffer kamadak already read (zero-copy Bytes) and videos open once with a kind() dispatch; per-image opens 2-3 -> 1 (1.44x warm geomean, 1.6-3.2x cold cache; extraction outputs gated identical incl. the MIME-mislabel track fallback). - Chunked-upload session ops: owner gate folded into the operation's own DashMap lookup + stack-encoded uuid compare (5 -> 3 lookups, -2 allocs, 1.28x per chunk). Blob cache (bench_blob_cache_index + round-3 regression guard): - CachedBlobBackend index: tokio::sync::Mutex<LruCache> -> moka::sync::Cache with byte weigher. The mutex serialized every cached chunk read and scaled NEGATIVELY (2.08 -> 1.07 Mops/s from 1 -> 2 readers); moka probes are lock-free (2.17x at K=2). Byte budget now enforced by moka (manual current_size + collect_evictions machinery deleted); eviction listener unlinks size-evicted files only (Replaced entries keep their file — gated). Single-flight miss gate unchanged (16 concurrent misses -> 1 fetch re-verified via the round-3 harness). - put_blob now populates the cache BEFORE the inner backend consumes the source file (the old order failed 100% of the time — local renames, S3/Azure delete the source — so the first read after a whole-file put re-downloaded from the remote); inner-put failure invalidates the entry. Frontend (vitest gates): - List-view thumbnails request the 150px icon rendition instead of 400px preview into a 40px slot (~7.1x fewer pixels, ~4-5x fewer bytes per thumbnail across list views); grid keeps preview. Rejected by its own bench (kept as evidence in bench_round12_micro §2): - Single-pass compression predicate: the monomorphized And-chain already costs ~4.6 ns / 0 allocs total; the fused node measured within noise. New migration: 20260719000000_users_search_trgm.sql (trgm indexes). Deferred with prepared design: grouped file/grid view virtualization (single-VirtualRows flatten, the photos pattern) — next round's headline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BfidAJD5AHw23jtvBUNamB |
||
|
Claude
|
fdf445d2b0 |
perf: round 9 — decorator PUT reactivation, session/search/dedup alloc purges, PROPFIND join!, folder-level cascade
Benchmark-gated round (benches/ROUND9.md): every change carries a BEFORE/AFTER bench with equivalence/safety gates; verdicts below are from the committed harnesses on 4 cores / local PG 16. Backend: - Blob decorators (Retry/Cached) now forward put_blob_from_bytes_unsynced + sync_blobs — the trait default had silently reinstated HEAD-before-PUT per chunk on decorated remote stacks, undoing ROUND3 §8. Full production stack: 500 probes -> 0, 1.9x wall at 10 ms RTT (bench_s3_put §3). - NC PROPFIND per-page enrichment triple (favorites / oc:fileid / dead props) overlapped with tokio::join!: 2.07x local, 2.86x at 5 ms RTT (bench_nc_enrich_join, injected-latency decide-by-bench). - Search enrichment consumes its DTOs and carries the interned Arc<str> display fields end-to-end (SearchFileResultDto type change, OpenAPI shape preserved): enrich_file 2.0x, 11.6 -> 2.2 allocs/row; the NC REPORT conversion stops re-running all three classifiers per row (bench_search_enrich). - NC session Arc end-to-end: SharedNcSession extractor (8 -> 0 allocs), Arc<FolderDto> chroot cache (4 -> 0/hit), single shared Arc<CurrentUser> + lazy span render (11 -> 6/build) (bench_nc_session). - Storage micro-pack: atomic create_new chunk writes (2.1x fresh), stream_chunks over the manifest Arc (4097 -> 0 allocs/read incl. the Range path), manifest single-flight (herd 64 -> 1 loads), hex_lower for chunk Content-MD5 (18 -> 1 allocs) (bench_storage_micro). - OCS capabilities memoized into OnceLock<[Bytes;2]>: 237x, 102 -> 0 allocs/poll, byte-identical (bench_capabilities_static). - Drive::is_empty COUNT(*) sum -> EXISTS: 34.4x on a 100k-file drive (bench_drive_is_empty). - favorites/recents row-map ROUND7 port: path/name/blob_hash moved, -2.75 allocs/row (bench_resource_row_map §2). - Folder rows decode binary UUIDs (ROUND6 §10 port): 1.03-1.07x page fetch, honest verdict incl. one noise-band wash documented (bench_folder_uuid_decode). - Authz: file cascade decision decomposed into memoized folder-level decision + direct-grant lookup (ROUND8 deferred item). Cold shared-album first view 592 -> 418 µs/thumb; warm path unchanged; safety gates incl. new direct-grant sibling isolation, revoke-flush re-verified, full integration authz suite green (bench_thumbnail_cascade_cache). Frontend (vitest gates committed beside the code): - resolveLabel/resolveRecipient O(directory) scan -> id-keyed Map: 13.9x (recipients.bench.test.ts). - ResourceList selection-prune effect skips when nothing is selected (100 -> 0 Set builds per drain) and the photos timeline reads a listener-fed mobile flag instead of matchMedia per recompute (listDerives.bench.test.ts). Verification: cargo fmt + clippy --all-features --all-targets -D warnings clean; 524 unit + 554 integration (--cfg integration_tests) tests pass; frontend npm run check clean with 293 vitest tests green. Deferred with rationale in ROUND9.md: CalDAV authz-before-fetch reorder (maintainer sign-off), per-page batched parent resolution, JWT-claims Arc<str>, batch_operations signature widening. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XDc9VtXvskJ6dnMRraSndn |
||
|
Claude
|
cd4c62042a |
perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr |
||
|
Claude
|
f70dffeaf5 |
perf(dedup): backend-aware chunk read-ahead for CDC reassembly
read_blob_stream / read_blob_range_stream reassembled a CDC file by fetching its chunks with `buffered(1)` — strictly sequential, so the next chunk's backend fetch (a file `open` locally; a full request round-trip on S3/Azure) only started after the current chunk was fully drained. A benchmark of the exact pipeline (stream::iter(chunks).map(get).buffered(K) .try_flatten()) showed a blind `buffered(4)` is the WRONG fix: on a local disk it is neutral on a warm page cache and ~37% SLOWER cold, because concurrent opens turn one sequential read into several competing random-I/O streams over content-addressed (scattered) chunk files. The win is entirely on remote backends, where per-chunk request latency dominates and overlapping fetches hide it (≈ linear in K). So the read-ahead depth is now a backend hint, not a constant: - BlobStorageBackend::read_prefetch() default 1 (sequential; safe for local). - S3 / Azure override to 8 (overlap GETs to hide TTFB). - cached / encrypted / retry / migration delegate to the backend that serves the bytes. - Both CDC read paths use `self.backend.read_prefetch().max(1)`. Net: local backend unchanged (no regression); remote reassembly ~4-8x faster. Ordered `buffered` (not buffer_unordered) keeps chunks in sequence. Bench (per-chunk fetch-latency model): buffered(1)->(4)/(8) = x3.9 / x7.8 @1ms, x4.0 / x8.1 @5ms, x4.0 / x8.0 @20ms. Local warm: 230ms@1 vs 227ms@4 (noise); local cold: 425ms@1 vs 585ms@4 (why local stays at 1). https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK |
||
|
Claude
|
fc55e92299 |
perf(blob-cache,db): release index mutex before disk I/O; skip per-acquire DB ping
CachedBlobBackend held its single tokio::Mutex<LruCache> across filesystem syscalls, serializing every concurrent cache operation behind one lock: - get_blob_stream / get_blob_range_stream: held across File::open()/seek() - delete_blob: held across remove_file() - initialize: held across the full cache-dir walk - eviction (insert + fetch paths): held across remove_file() loops Now the lock only guards the in-memory LRU. Presence checks bump recency and release the guard before touching the filesystem (a vanished file falls through to the existing fetch-and-cache path, covering the race), and eviction selects victims under the lock then unlinks them after releasing it. The duplicated eviction loop is extracted into CachedRef::collect_evictions. db: set test_before_acquire(false). With warm min_connections and a bounded max_lifetime, the liveness ping sqlx issues on every acquire() costs more than the rare dead connection it catches; stale sockets surface as a query error and the pool recycles them either way. https://claude.ai/code/session_01UtfkS3nZF1vrF5jNAps6wV |
||
|
Diocrafts
|
761d159a92 |
feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking (min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing - Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[]) - Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends) - 3-phase store_chunks pipeline: Phase 0: batch-check existing chunks (single PG query) Phase 1: selective disk read (skip existing chunks entirely) Phase 2: parallel upload with buffer_unordered(8) - CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback - Transactional manifest + chunk ref-count cascade on remove_reference - 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.) - Update deduplication.md to reflect new architecture |
||
|
Diocrafts
|
cd3733b459 |
feat: pluggable storage backends (S3, Azure, local) with admin UI
Implement 4-phase external storage backends architecture: Phase 1 - Foundation: - BlobStorageBackend trait (application/ports/blob_storage_ports.rs) - LocalBlobBackend: extracted all tokio::fs ops from DedupService - S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2) - DedupService refactored to use Arc<dyn BlobStorageBackend> Phase 2 - Admin Panel: - StorageSettingsService with DB persistence + env override - Storage tab in admin panel (backend selector, S3 form, provider presets) - GET/PUT/POST endpoints for storage settings + connection test - i18n keys (en/es) and BEM CSS Phase 3 - Migration: - MigrationBlobBackend decorator (dual-read: target-first + source fallback) - Background migration job with parallel transfers + progress tracking - Migration UI (progress bar, ETA, pause/resume/verify/complete) - 6 admin API endpoints for migration lifecycle Phase 4 - Enterprise Extras: - CachedBlobBackend: LRU disk cache for remote backends - EncryptedBlobBackend: AES-256-GCM at-rest encryption - AzureBlobBackend: Azure Blob Storage support - RetryBlobBackend: exponential backoff for transient errors - Decorator composition in DI: retry → encryption → cache All 223 tests passing, clippy clean, fmt verified. |