Add embedded Tantivy full-text content search
/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.
Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
(storage.search_index_dirty) - every write surface (REST, WebDAV,
NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
batched single-writer Tantivy commits; queue rows are deleted only
after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
reseeds it from Postgres, which remains the single source of truth
SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.
The frontend renders the snippet under the file name in list view.
New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.
https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
This commit is contained in:
@@ -0,0 +1,49 @@
|
||||
//! Content Index Port - Application layer abstraction for full-text search
|
||||
//! over file names and extracted file content.
|
||||
//!
|
||||
//! The implementation (an embedded Tantivy BM25 index) lives in the
|
||||
//! infrastructure layer; `SearchService` only sees this port. The index is a
|
||||
//! DERIVED artifact fed asynchronously by a background worker — it never sits
|
||||
//! on a request path, and PostgreSQL remains the source of truth (hits are
|
||||
//! re-validated and hydrated through SQL before they reach the caller, so a
|
||||
//! stale index can only ever produce a dropped candidate, never a leak).
|
||||
|
||||
use async_trait::async_trait;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::common::errors::DomainError;
|
||||
|
||||
/// One content-index hit: a candidate file id with its BM25 score and an
|
||||
/// optional plain-text snippet around the first match.
|
||||
///
|
||||
/// `file_id` is a CANDIDATE — callers must hydrate it through the metadata
|
||||
/// repository (which re-applies user scoping, trash state and the active
|
||||
/// search filters) before exposing it.
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct ContentHitDto {
|
||||
/// File UUID as string (matches `storage.files.id`).
|
||||
pub file_id: String,
|
||||
/// BM25 relevance score (positive, unbounded — normalize per result set).
|
||||
pub score: f32,
|
||||
/// Plain-text fragment around the first matched term, when available.
|
||||
pub snippet: Option<String>,
|
||||
}
|
||||
|
||||
/// Port for querying the full-text content index.
|
||||
///
|
||||
/// `#[async_trait]` is used so the trait is dyn-compatible — `SearchService`
|
||||
/// holds an `Option<Arc<dyn ContentIndexPort>>` (the feature is toggleable).
|
||||
#[async_trait]
|
||||
pub trait ContentIndexPort: Send + Sync + 'static {
|
||||
/// Search indexed file names + content for `query`, scoped to `user_id`.
|
||||
///
|
||||
/// Returns up to `limit` hits sorted by BM25 score descending. Matching is
|
||||
/// tokenized (not substring): exact terms, typo-tolerant fuzzy terms
|
||||
/// (edit distance 1) and prefix expansion on the last query token.
|
||||
async fn search_content(
|
||||
&self,
|
||||
user_id: Uuid,
|
||||
query: &str,
|
||||
limit: usize,
|
||||
) -> Result<Vec<ContentHitDto>, DomainError>;
|
||||
}
|
||||
@@ -7,6 +7,7 @@ pub mod calendar_ports;
|
||||
pub mod carddav_ports;
|
||||
pub mod chunked_upload_ports;
|
||||
pub mod compression_ports;
|
||||
pub mod content_index_ports;
|
||||
pub mod dedup_ports;
|
||||
pub mod email_sender;
|
||||
pub mod favorites_ports;
|
||||
|
||||
Reference in New Issue
Block a user