fdf445d2b0
perf: round 9 — decorator PUT reactivation, session/search/dedup alloc purges, PROPFIND join!, folder-level cascade
...
Benchmark-gated round (benches/ROUND9.md): every change carries a
BEFORE/AFTER bench with equivalence/safety gates; verdicts below are from
the committed harnesses on 4 cores / local PG 16.
Backend:
- Blob decorators (Retry/Cached) now forward put_blob_from_bytes_unsynced
+ sync_blobs — the trait default had silently reinstated HEAD-before-PUT
per chunk on decorated remote stacks, undoing ROUND3 §8. Full production
stack: 500 probes -> 0, 1.9x wall at 10 ms RTT (bench_s3_put §3).
- NC PROPFIND per-page enrichment triple (favorites / oc:fileid / dead
props) overlapped with tokio::join!: 2.07x local, 2.86x at 5 ms RTT
(bench_nc_enrich_join, injected-latency decide-by-bench).
- Search enrichment consumes its DTOs and carries the interned Arc<str>
display fields end-to-end (SearchFileResultDto type change, OpenAPI
shape preserved): enrich_file 2.0x, 11.6 -> 2.2 allocs/row; the NC
REPORT conversion stops re-running all three classifiers per row
(bench_search_enrich).
- NC session Arc end-to-end: SharedNcSession extractor (8 -> 0 allocs),
Arc<FolderDto> chroot cache (4 -> 0/hit), single shared Arc<CurrentUser>
+ lazy span render (11 -> 6/build) (bench_nc_session).
- Storage micro-pack: atomic create_new chunk writes (2.1x fresh),
stream_chunks over the manifest Arc (4097 -> 0 allocs/read incl. the
Range path), manifest single-flight (herd 64 -> 1 loads), hex_lower for
chunk Content-MD5 (18 -> 1 allocs) (bench_storage_micro).
- OCS capabilities memoized into OnceLock<[Bytes;2]>: 237x, 102 -> 0
allocs/poll, byte-identical (bench_capabilities_static).
- Drive::is_empty COUNT(*) sum -> EXISTS: 34.4x on a 100k-file drive
(bench_drive_is_empty).
- favorites/recents row-map ROUND7 port: path/name/blob_hash moved,
-2.75 allocs/row (bench_resource_row_map §2).
- Folder rows decode binary UUIDs (ROUND6 §10 port): 1.03-1.07x page
fetch, honest verdict incl. one noise-band wash documented
(bench_folder_uuid_decode).
- Authz: file cascade decision decomposed into memoized folder-level
decision + direct-grant lookup (ROUND8 deferred item). Cold shared-album
first view 592 -> 418 µs/thumb; warm path unchanged; safety gates incl.
new direct-grant sibling isolation, revoke-flush re-verified, full
integration authz suite green (bench_thumbnail_cascade_cache).
Frontend (vitest gates committed beside the code):
- resolveLabel/resolveRecipient O(directory) scan -> id-keyed Map: 13.9x
(recipients.bench.test.ts).
- ResourceList selection-prune effect skips when nothing is selected
(100 -> 0 Set builds per drain) and the photos timeline reads a
listener-fed mobile flag instead of matchMedia per recompute
(listDerives.bench.test.ts).
Verification: cargo fmt + clippy --all-features --all-targets -D warnings
clean; 524 unit + 554 integration (--cfg integration_tests) tests pass;
frontend npm run check clean with 293 vitest tests green.
Deferred with rationale in ROUND9.md: CalDAV authz-before-fetch reorder
(maintainer sign-off), per-page batched parent resolution, JWT-claims
Arc<str>, batch_operations signature widening.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
Claude-Session: https://claude.ai/code/session_01XDc9VtXvskJ6dnMRraSndn
2026-07-18 16:12:04 +00:00
f70dffeaf5
perf(dedup): backend-aware chunk read-ahead for CDC reassembly
...
read_blob_stream / read_blob_range_stream reassembled a CDC file by fetching
its chunks with `buffered(1)` — strictly sequential, so the next chunk's
backend fetch (a file `open` locally; a full request round-trip on S3/Azure)
only started after the current chunk was fully drained.
A benchmark of the exact pipeline (stream::iter(chunks).map(get).buffered(K)
.try_flatten()) showed a blind `buffered(4)` is the WRONG fix: on a local
disk it is neutral on a warm page cache and ~37% SLOWER cold, because
concurrent opens turn one sequential read into several competing random-I/O
streams over content-addressed (scattered) chunk files. The win is entirely
on remote backends, where per-chunk request latency dominates and overlapping
fetches hide it (≈ linear in K).
So the read-ahead depth is now a backend hint, not a constant:
- BlobStorageBackend::read_prefetch() default 1 (sequential; safe for local).
- S3 / Azure override to 8 (overlap GETs to hide TTFB).
- cached / encrypted / retry / migration delegate to the backend that serves
the bytes.
- Both CDC read paths use `self.backend.read_prefetch().max(1)`.
Net: local backend unchanged (no regression); remote reassembly ~4-8x faster.
Ordered `buffered` (not buffer_unordered) keeps chunks in sequence.
Bench (per-chunk fetch-latency model): buffered(1)->(4)/(8) = x3.9 / x7.8
@1ms, x4.0 / x8.1 @5ms, x4.0 / x8.0 @20ms. Local warm: 230ms@1 vs 227ms@4
(noise); local cold: 425ms@1 vs 585ms@4 (why local stays at 1).
https://claude.ai/code/session_01DCszkkU11LYxMEUWr4setK
2026-06-15 12:34:56 +00:00
761d159a92
feat(dedup): CDC sub-file deduplication with FastCDC + parallel chunk storage + dedup skip
...
- Replace whole-file SHA-256 dedup with FastCDC 2020 content-defined chunking
(min 64KB, avg 256KB, max 1MB) + BLAKE3 hashing
- Add chunk_manifests table (file_hash → chunk_hashes[] + chunk_sizes[])
- Add put_blob_from_bytes to BlobStorageBackend trait (all 7 backends)
- 3-phase store_chunks pipeline:
Phase 0: batch-check existing chunks (single PG query)
Phase 1: selective disk read (skip existing chunks entirely)
Phase 2: parallel upload with buffer_unordered(8)
- CDC-aware read_blob_stream and read_blob_range_stream with legacy fallback
- Transactional manifest + chunk ref-count cascade on remove_reference
- 12 CDC tests (determinism, reassembly, contiguity, sub-file dedup, etc.)
- Update deduplication.md to reflect new architecture
2026-04-14 23:17:39 +02:00
cd3733b459
feat: pluggable storage backends (S3, Azure, local) with admin UI
...
Implement 4-phase external storage backends architecture:
Phase 1 - Foundation:
- BlobStorageBackend trait (application/ports/blob_storage_ports.rs)
- LocalBlobBackend: extracted all tokio::fs ops from DedupService
- S3BlobBackend: AWS SDK with custom endpoint support (MinIO, R2, B2)
- DedupService refactored to use Arc<dyn BlobStorageBackend>
Phase 2 - Admin Panel:
- StorageSettingsService with DB persistence + env override
- Storage tab in admin panel (backend selector, S3 form, provider presets)
- GET/PUT/POST endpoints for storage settings + connection test
- i18n keys (en/es) and BEM CSS
Phase 3 - Migration:
- MigrationBlobBackend decorator (dual-read: target-first + source fallback)
- Background migration job with parallel transfers + progress tracking
- Migration UI (progress bar, ETA, pause/resume/verify/complete)
- 6 admin API endpoints for migration lifecycle
Phase 4 - Enterprise Extras:
- CachedBlobBackend: LRU disk cache for remote backends
- EncryptedBlobBackend: AES-256-GCM at-rest encryption
- AzureBlobBackend: Azure Blob Storage support
- RetryBlobBackend: exponential backoff for transient errors
- Decorator composition in DI: retry → encryption → cache
All 223 tests passing, clippy clean, fmt verified.
2026-04-14 21:33:38 +02:00
Claude