Sweep aborted-upload orphans from the periodic trash job; pipeline ZIP reads
Two follow-ups to the streaming-upload work: The dedup garbage_collect() pass only ran when a user manually emptied their trash. Zero-reference rows — chunks orphaned by aborted streaming uploads (registered at ref_count 0 by the ingest rollback) and blobs dereferenced by trash expiry itself — could linger indefinitely on instances where nobody empties trash. The periodic TrashCleanupService sweep now ends every run with garbage_collect() (maintenance pool, batched), bounding orphan lifetime to the cleanup interval. Folder-ZIP creation was strictly sequential: open blob stream, deflate, close, repeat — every per-file blob-store round-trip (PG lookup + backend open; a full HTTP round-trip on S3/Azure) added to the wall clock. It now runs as a 2-stage pipeline: a prefetch task streams the planned files' content ahead of the writer through a bounded channel (~4 MiB), so the next file's read latency overlaps the current file's compression. ZIP entries are still written strictly in order, peak RAM stays flat, and a writer error hangs up the channel so the prefetcher stops on its own. Verified end-to-end against PostgreSQL 16: an upload aborted at ~14 MB left exactly 30 ref_count=0 chunk rows which the GC then reclaimed (8.5 MB, rows + physical files); a chunked upload completed with a wrong MD5 returned 400 with the tee-computed digest and the SAME session then completed successfully with the right checksum (parts persist — the old assembly deleted them, so the documented retry never actually worked); a 4-file folder ZIP downloaded and extracted byte-identical. https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
+4
-1
@@ -557,9 +557,12 @@ impl AppServiceFactory {
|
||||
.with_file_deleted_hook(core.file_lifecycle.clone()),
|
||||
);
|
||||
|
||||
// Initialize cleanup service (bulk-deletes expired items in 2 SQL queries)
|
||||
// Initialize cleanup service (bulk-deletes expired items in 2 SQL
|
||||
// queries, then GCs zero-reference blobs — including chunks orphaned
|
||||
// by aborted streaming uploads).
|
||||
let cleanup_service = TrashCleanupService::new(
|
||||
trash_repo.clone(),
|
||||
core.dedup_service.clone(),
|
||||
24, // Run cleanup every 24 hours
|
||||
);
|
||||
|
||||
|
||||
Reference in New Issue
Block a user