Edouard Vanbelle
213e1c553a
Step 3 (prerequisite 2) of docs/plan/derived-blobs.md, and the last one
before the thumbnail slice.
There are two reference counters and only one was ever verified.
add_reference bumps chunk_manifests.ref_count first and only falls back
to storage.blobs.ref_count, so a reference lands on whichever counter
its hash names: chunk references feed storage.blobs and are reconciled
by blobs_consistency::refcount_mismatch, while Blob references — every
CDC file, and every derived artifact once those exist — feed
chunk_manifests.ref_count, which nothing reconciled.
That gap was survivable only because dedup_gc's reap predicate carried a
second clause ("no storage.files row references this manifest") that
quietly compensated for drift on the bulk-delete paths where ref_count
is never decremented. Generalising that clause to the reference registry
in 1c8ead49 — so thumbnails stop being reaped — removed the
compensation, which is precisely why the counter now needs checking
directly. The two changes have to ship together.
Adds manifests_consistency, a recoverable job reporting
manifest_refcount_mismatch (severity inconsistent). The finding carries
reap_risk so an operator can triage: an under-count means GC reaps a
manifest whose content is still reachable, taking its chunks with it,
while an over-count merely pins storage.
A separate job rather than a second phase of blobs_consistency: one
subject per job, as the other five consistency tenants do, and it avoids
changing the cursor format of an existing recoverable job — which would
strand any run paused across the deploy.
The page query is assembled from the same registry dedup_gc reaps from
(via DedupService::reference_registry), built once at construction, and
pinned by a golden test. Two invariants the test guards: the files term
carries no NOT EXISTS guard — that guard keeps CDC rows out of the
*chunk* level and here would count nothing — and chunk_hashes appears
nowhere, since a manifest citing its own chunks is not a referrer of
itself.
fmt, clippy --all-features --all-targets, and 3 new unit tests clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
69 lines
1.9 KiB
Rust
69 lines
1.9 KiB
Rust
pub mod audio_metadata_service;
|
|
pub mod azure_blob_backend;
|
|
pub mod backend_consistency_service;
|
|
pub mod backend_migration_service;
|
|
pub mod backend_rotate_service;
|
|
pub mod blobs_consistency_service;
|
|
pub mod cached_blob_backend;
|
|
pub mod chunked_upload_service;
|
|
pub mod compression_service;
|
|
pub mod consistency_batch_service;
|
|
pub mod db_pool_monitor;
|
|
pub mod dedup_service;
|
|
pub mod dpop_nonce_service;
|
|
pub mod dpop_replay_cache;
|
|
pub mod dpop_verifier;
|
|
pub mod drives_consistency_service;
|
|
pub mod encrypted_blob_backend;
|
|
pub mod entry_backend;
|
|
pub mod exif_service;
|
|
pub mod face_geometry;
|
|
pub mod face_indexing_service;
|
|
pub mod ffmpeg_video_frame_service;
|
|
pub mod file_content_cache;
|
|
pub mod file_system_i18n_service;
|
|
pub mod files_consistency_service;
|
|
pub mod folders_consistency_service;
|
|
pub mod grant_cleanup_service;
|
|
pub mod image_transcode_service;
|
|
pub mod jwt_service;
|
|
pub mod last_seen_tracker;
|
|
pub mod local_blob_backend;
|
|
pub mod local_fs_mount_provider;
|
|
pub mod login_lockout_service;
|
|
pub mod manifests_consistency_service;
|
|
pub mod media_metadata_service;
|
|
pub mod mock_email_sender;
|
|
pub mod mount_provider_factory;
|
|
pub mod nextcloud_chunked_upload_service;
|
|
pub mod noop_face_analyzer;
|
|
pub mod oidc_service;
|
|
#[cfg(feature = "faces-onnx")]
|
|
pub mod onnx_face_analyzer;
|
|
pub mod opaque_login_exchange;
|
|
pub mod opaque_service;
|
|
pub mod password_hasher;
|
|
pub mod path_resolver_service;
|
|
pub mod path_service;
|
|
pub mod pg_acl_engine;
|
|
#[cfg(feature = "plugins")]
|
|
pub mod plugins;
|
|
pub mod recent_recording_hook;
|
|
pub mod retry_blob_backend;
|
|
pub mod s3_blob_backend;
|
|
pub mod search_index;
|
|
pub mod session_cleanup_service;
|
|
pub mod session_liveness_gauges;
|
|
pub mod share_unlock_cookie;
|
|
pub mod smtp_email_sender;
|
|
pub mod swappable_blob_backend;
|
|
pub mod thumbnail_service;
|
|
#[cfg(test)]
|
|
mod thumbnail_service_test;
|
|
pub mod trash_cleanup_service;
|
|
pub mod tree_etag_flush_service;
|
|
pub mod webdav_dead_property_store;
|
|
pub mod webdav_lock_service;
|
|
pub mod wopi_discovery_service;
|
|
pub mod zip_service;
|