Sweep aborted-upload orphans from the periodic trash job; pipeline ZIP reads

Two follow-ups to the streaming-upload work:

The dedup garbage_collect() pass only ran when a user manually emptied
their trash. Zero-reference rows — chunks orphaned by aborted streaming
uploads (registered at ref_count 0 by the ingest rollback) and blobs
dereferenced by trash expiry itself — could linger indefinitely on
instances where nobody empties trash. The periodic TrashCleanupService
sweep now ends every run with garbage_collect() (maintenance pool,
batched), bounding orphan lifetime to the cleanup interval.

Folder-ZIP creation was strictly sequential: open blob stream, deflate,
close, repeat — every per-file blob-store round-trip (PG lookup + backend
open; a full HTTP round-trip on S3/Azure) added to the wall clock. It now
runs as a 2-stage pipeline: a prefetch task streams the planned files'
content ahead of the writer through a bounded channel (~4 MiB), so the
next file's read latency overlaps the current file's compression. ZIP
entries are still written strictly in order, peak RAM stays flat, and a
writer error hangs up the channel so the prefetcher stops on its own.

Verified end-to-end against PostgreSQL 16: an upload aborted at ~14 MB
left exactly 30 ref_count=0 chunk rows which the GC then reclaimed
(8.5 MB, rows + physical files); a chunked upload completed with a wrong
MD5 returned 400 with the tee-computed digest and the SAME session then
completed successfully with the right checksum (parts persist — the old
assembly deleted them, so the documented retry never actually worked);
a 4-file folder ZIP downloaded and extracted byte-identical.

https://claude.ai/code/session_01WdNenpnujNR2sc32XVvwfS
This commit is contained in:
Claude
2026-06-11 13:19:29 +00:00
parent e3f04d58aa
commit 944c833787
3 changed files with 193 additions and 61 deletions
+4 -1
View File
@@ -557,9 +557,12 @@ impl AppServiceFactory {
.with_file_deleted_hook(core.file_lifecycle.clone()),
);
// Initialize cleanup service (bulk-deletes expired items in 2 SQL queries)
// Initialize cleanup service (bulk-deletes expired items in 2 SQL
// queries, then GCs zero-reference blobs — including chunks orphaned
// by aborted streaming uploads).
let cleanup_service = TrashCleanupService::new(
trash_repo.clone(),
core.dedup_service.clone(),
24, // Run cleanup every 24 hours
);