Add embedded Tantivy full-text content search

/api/search now finds files by CONTENT as well as by name: BM25-ranked
matches over extracted text (PDF, Office OOXML/ODF, plain text/code)
with typo-tolerant fuzzy terms and search-as-you-type prefix matching,
served from an embedded Tantivy index at {storage}/.search-index.

Pipeline (all off the request path, mirroring tree-etag + thumbnails):
- statement triggers on storage.files append to a durable dirty queue
  (storage.search_index_dirty) - every write surface (REST, WebDAV,
  NextCloud, WOPI, trash) is covered, crash-safe by construction
- ContentIndexWorker drains the queue on the maintenance pool, extracts
  text once per unique BLAKE3 blob (storage.blob_extracted_text cache:
  N copies = 1 extraction, renames/moves = 0 re-extraction) and applies
  batched single-writer Tantivy commits; queue rows are deleted only
  after the commit succeeds (at-least-once, idempotent upserts)
- the index is a derived artifact: a version-marker mismatch wipes and
  reseeds it from Postgres, which remains the single source of truth

SearchService merges content hits into the existing name search: hits
are hydrated through ONE SQL round-trip that re-applies user scope,
trash state and every active filter (a stale index id can never leak),
scored below name matches, and returned with a plain-text snippet and
a match_source field. Index failure or
OXICLOUD_ENABLE_CONTENT_SEARCH=false degrades to name-only search; a
discard-only janitor keeps the trigger-fed queue bounded while disabled.

The frontend renders the snippet under the file name in list view.

New dependencies: tantivy 0.26, zip 8.6 (deflate only), pdf-extract 0.10.

https://claude.ai/code/session_01Sc7F4xbo83YbFAQ4xEeDrX
This commit is contained in:
Claude
2026-06-11 15:16:03 +00:00
parent 7157454afd
commit 8dab135090
19 changed files with 2813 additions and 70 deletions
+20
View File
@@ -204,6 +204,26 @@ DATABASE_URL=postgres://postgres:postgres@localhost:5432/oxicloud
# Enable search functionality (default: true)
#OXICLOUD_ENABLE_SEARCH=true
# Full-text content search (embedded Tantivy index over file names AND file
# content: PDF, Office, plain text/code). Indexing runs on the maintenance
# pool, off the request path. (default: true)
#OXICLOUD_ENABLE_CONTENT_SEARCH=true
# Content index directory (default: {OXICLOUD_STORAGE_PATH}/.search-index)
#OXICLOUD_CONTENT_INDEX_DIR=
# Index worker drain cadence in ms — upper bound on how long a new upload
# takes to become content-searchable (default: 1500)
#OXICLOUD_CONTENT_INDEX_FLUSH_MS=1500
# Files larger than this are indexed by name only, no text extraction
# (default: 33554432 = 32 MiB)
#OXICLOUD_CONTENT_INDEX_MAX_FILE_BYTES=33554432
# Cap on extracted text per unique blob fed to the index
# (default: 1048576 = 1 MiB)
#OXICLOUD_CONTENT_INDEX_MAX_TEXT_BYTES=1048576
# Enable music playlists and audio metadata (default: true)
#OXICLOUD_ENABLE_MUSIC=true