Merge pull request #657 from EdouardVanbelle/fix/services-use-blob-abstraction
fix: prevent services accessing directly to localstorage (audio, media metadata and face indexing)
This commit is contained in:
@@ -8,6 +8,7 @@ Most runtime variables use the `OXICLOUD_` prefix. A few build-time or allocator
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `OXICLOUD_STORAGE_PATH` | `./storage` | Root storage directory |
|
| `OXICLOUD_STORAGE_PATH` | `./storage` | Root storage directory |
|
||||||
| `OXICLOUD_STATIC_PATH` | `./static` | Static files directory |
|
| `OXICLOUD_STATIC_PATH` | `./static` | Static files directory |
|
||||||
|
| `OXICLOUD_TEMP_DIR` | `std::env::temp_dir()` (`$TMPDIR`) | Directory for tier-1 temporary data — pure scratch, safe to lose at reboot. Backend services stream blobs here when extractors need a `&Path` (id3, mp3_duration, ffprobe, nom-exif video). Files are auto-removed after use. On Linux `/tmp` is often tmpfs (RAM-backed); point at a disk-backed dir under RAM-constrained deployments. |
|
||||||
| `OXICLOUD_SERVER_PORT` | `8086` | Server port |
|
| `OXICLOUD_SERVER_PORT` | `8086` | Server port |
|
||||||
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
| `OXICLOUD_SERVER_HOST` | `127.0.0.1` | Server bind address (IPv4 or IPv6 allowed) |
|
||||||
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
| `OXICLOUD_BASE_URL` | (auto) | Public base URL for share links; defaults to `http://{host}:{port}` |
|
||||||
|
|||||||
@@ -0,0 +1,306 @@
|
|||||||
|
# Plan — Derived content as blobs (tier-2 refactor)
|
||||||
|
|
||||||
|
**Status:** design captured 2026-08-02, not implemented. Follow-up to
|
||||||
|
`fix/services-use-blob-abstraction` — that PR normalised the
|
||||||
|
**read-side** (services consume blobs through `BlobStorageBackend`
|
||||||
|
uniformly). This plan tackles the **write-side**: services that
|
||||||
|
today write derived artifacts (thumbnails, transcodes) to a local
|
||||||
|
sidecar directory and would benefit from writing them through the
|
||||||
|
backend abstraction instead.
|
||||||
|
|
||||||
|
## Context — the three-tier storage taxonomy
|
||||||
|
|
||||||
|
Today the codebase runs three implicit tiers with no explicit
|
||||||
|
separation:
|
||||||
|
|
||||||
|
| Tier | Purpose | Loss on reboot? | Where today |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **1 — Temp** | Pure scratch, deletable at reboot | ✅ fine | `std::env::temp_dir()` (ad-hoc callers). Now unified under `OXICLOUD_TEMP_DIR` (`AppConfig::temp_dir`). |
|
||||||
|
| **2 — Persistent spool** | Caches; expensive but rebuildable | ⚠️ possible but painful | `<storage_path>/.thumbnails/`, `<storage_path>/.transcoded/`, `<storage_path>/.blob-cache/`, `<storage_path>/.search-index/`, `<storage_path>/.plugin-logs/` — all mixed into tier-3 storage today. |
|
||||||
|
| **3 — Persistent data** | Source of truth | ❌ never | `<storage_path>/.blobs/` (Local) OR S3/Azure bucket, via `BlobStorageBackend`. Already correctly configured via `OXICLOUD_STORAGE_ENTRIES`. |
|
||||||
|
|
||||||
|
**Today's misclassification**: tier-2 sidecars live under
|
||||||
|
`<storage_path>` — the same directory as tier-3 source-of-truth
|
||||||
|
data. Ops resizing / moving / backing up tier-3 accidentally moves
|
||||||
|
tier-2 caches with it. Loss of tier 2 is expensive (regenerate
|
||||||
|
thumbnails for every photo) but not data loss; conflating them
|
||||||
|
means backup policies can't distinguish "must preserve" from "can
|
||||||
|
rebuild".
|
||||||
|
|
||||||
|
## Multi-instance driver
|
||||||
|
|
||||||
|
Single-instance: tier-2-as-local-cache works fine. Rebuild after
|
||||||
|
reboot is annoying but bounded.
|
||||||
|
|
||||||
|
Multi-instance (2+ app servers behind a load balancer):
|
||||||
|
|
||||||
|
- Request for thumbnail `abc123.jpg` lands on instance A → generates
|
||||||
|
it → stores locally at `.thumbnails/abc123.jpg`.
|
||||||
|
- Same-URL retry lands on instance B → cache miss → regenerates
|
||||||
|
from source.
|
||||||
|
- Every derived asset gets recomputed N times (N = instance count)
|
||||||
|
at worst.
|
||||||
|
|
||||||
|
Wasteful compute, wasteful storage, inconsistent latency. The
|
||||||
|
long-term fix is to put derived content on tier 3 (shared) with a
|
||||||
|
local read-through cache in front. Multi-instance isn't the near-
|
||||||
|
term target, but the design should leave the door open.
|
||||||
|
|
||||||
|
## Design decision — derived content IS a blob
|
||||||
|
|
||||||
|
The blob storage abstraction is already:
|
||||||
|
|
||||||
|
- Backend-agnostic (Local / S3 / Azure)
|
||||||
|
- Encrypted uniformly (`EncryptedBlobBackend` wrapper)
|
||||||
|
- Consistency-checked (`blobs_consistency`)
|
||||||
|
- Migratable (`backend_migration`)
|
||||||
|
- Rotatable (`backend_rotate`)
|
||||||
|
- Multi-instance-ready (S3/Azure natively; Local via network mount)
|
||||||
|
|
||||||
|
Reusing it for derived artifacts means no second abstraction to
|
||||||
|
build and maintain, and all the operational surface (audit,
|
||||||
|
migration, key rotation) applies to derived content by default.
|
||||||
|
|
||||||
|
### Keying
|
||||||
|
|
||||||
|
Content-addressable via BLAKE3, same as source blobs. For
|
||||||
|
server-derived content the hash is over the produced bytes (not
|
||||||
|
the source), so:
|
||||||
|
|
||||||
|
- Two files with **identical thumbnails** (e.g. same 256px WebP
|
||||||
|
crop of the same underlying image → identical bytes → identical
|
||||||
|
hash) share the physical blob. Dedup wins for free.
|
||||||
|
- Two files with **identical originals** but **different variant
|
||||||
|
specs** (256px vs 512px thumb) produce different blobs. Also
|
||||||
|
correct.
|
||||||
|
|
||||||
|
The variant spec (what was rendered) lives in the referring DB row
|
||||||
|
alongside the blob hash — not in the storage key. Storage stays
|
||||||
|
one keyspace; ownership stays per-service.
|
||||||
|
|
||||||
|
### Client-uploaded thumbnails
|
||||||
|
|
||||||
|
Some clients (NC desktop, mobile apps) upload their own encoded
|
||||||
|
previews alongside the file. These are **not derivable** — losing
|
||||||
|
them means asking the client to regenerate, which may not be
|
||||||
|
possible (client offline, original file no longer present on
|
||||||
|
device).
|
||||||
|
|
||||||
|
Same storage shape: BLAKE3 of the client-provided bytes → blob.
|
||||||
|
The DB row distinguishes `origin = 'server_derived' | 'client_provided'`
|
||||||
|
so consistency-check policy can differ (missing client-provided
|
||||||
|
thumbnail = data loss finding; missing server-derived = warning,
|
||||||
|
regenerable).
|
||||||
|
|
||||||
|
## `BlobReferenceSource` — reference tracking abstraction
|
||||||
|
|
||||||
|
Adding new blob-owning services without teaching the ref-count +
|
||||||
|
consistency machinery about them causes silent orphaning risk:
|
||||||
|
`dedup_gc` sees `ref_count = 0` and reaps live content.
|
||||||
|
|
||||||
|
The extension point:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
#[async_trait]
|
||||||
|
pub trait BlobReferenceSource: Send + Sync {
|
||||||
|
/// Short stable identifier for logs / consistency finding
|
||||||
|
/// `source` fields. Suggested: `"files"`, `"chunks"`,
|
||||||
|
/// `"thumbnails"`, `"transcodes"`.
|
||||||
|
fn source_name(&self) -> &'static str;
|
||||||
|
|
||||||
|
/// Count of references this source holds on `blob_hash`.
|
||||||
|
/// Called by `blobs_consistency` when recomputing
|
||||||
|
/// `refcount_mismatch` findings.
|
||||||
|
async fn count_references(&self, blob_hash: &str) -> Result<u64, DomainError>;
|
||||||
|
|
||||||
|
/// Iterate the source's referenced blobs, paged by the
|
||||||
|
/// implementation's natural cursor (typically a DB PK). Used
|
||||||
|
/// by `backend_consistency` to walk the backend against the
|
||||||
|
/// union of all sources.
|
||||||
|
async fn list_referenced_blobs(
|
||||||
|
&self,
|
||||||
|
cursor: Option<Vec<u8>>,
|
||||||
|
limit: usize,
|
||||||
|
) -> Result<(Vec<String>, Option<Vec<u8>>), DomainError>;
|
||||||
|
|
||||||
|
/// Optional notify hook: `dedup_gc` reaped this blob. Sources
|
||||||
|
/// that maintain their own denormalised refcount table can
|
||||||
|
/// clean up here. Most sources leave this as the trait default
|
||||||
|
/// (noop).
|
||||||
|
fn on_blob_reaped(&self, _blob_hash: &str) {}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Wired via a `BlobReferenceRegistry`:
|
||||||
|
|
||||||
|
```rust
|
||||||
|
pub struct BlobReferenceRegistry {
|
||||||
|
sources: Vec<Arc<dyn BlobReferenceSource>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BlobReferenceRegistry {
|
||||||
|
pub fn register(&mut self, source: Arc<dyn BlobReferenceSource>);
|
||||||
|
pub async fn total_references(&self, hash: &str) -> Result<u64, DomainError>;
|
||||||
|
// ... etc.
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Current implicit sources become the first two explicit
|
||||||
|
registrations:
|
||||||
|
|
||||||
|
- `FilesReferenceSource` — wraps `storage.files.blob_hash`
|
||||||
|
- `ChunksReferenceSource` — wraps `storage.chunk_manifests.chunk_hashes[]`
|
||||||
|
|
||||||
|
Tier-2 migration adds:
|
||||||
|
|
||||||
|
- `ThumbnailsReferenceSource` — wraps a new
|
||||||
|
`storage.thumbnails(hash, blob_hash, variant_spec, origin)` table
|
||||||
|
- `TranscodesReferenceSource` — wraps
|
||||||
|
`storage.transcodes(hash, blob_hash, target_format)` table
|
||||||
|
|
||||||
|
Then:
|
||||||
|
|
||||||
|
- **`dedup_gc`** — orphan iff `registry.total_references(hash) == 0`
|
||||||
|
(with the existing grace window). No per-service GC changes.
|
||||||
|
- **`blobs_consistency`** — `refcount_mismatch` recomputes via
|
||||||
|
`registry.total_references`. New services register → automatically
|
||||||
|
covered.
|
||||||
|
- **`backend_consistency`** — walks the backend and unions all
|
||||||
|
`list_referenced_blobs` streams for the "did we lose bytes"
|
||||||
|
check.
|
||||||
|
|
||||||
|
## Sidecar directories after this refactor
|
||||||
|
|
||||||
|
| Sidecar today | After |
|
||||||
|
|---|---|
|
||||||
|
| `.thumbnails/` | Persisted as derived blobs in tier 3. `.thumbnails/` becomes a pure read-through cache (tier 1-ish; ephemeral, per-instance). |
|
||||||
|
| `.transcoded/` | Same shape as thumbnails. |
|
||||||
|
| `.blob-cache/` | Already a cache; stays. Owned by `CachedBlobBackend`. |
|
||||||
|
| `.search-index/` | Open question — see non-goals. |
|
||||||
|
| `.plugin-logs/` | Ops-local; stays. |
|
||||||
|
| `.uploads/` | Tier 1 already; migrates to `OXICLOUD_TEMP_DIR`. |
|
||||||
|
|
||||||
|
The persistent-spool env var reserved:
|
||||||
|
|
||||||
|
- **`OXICLOUD_SPOOL_DIR`** — path for the local read-through
|
||||||
|
caches (`.thumbnails/`, `.transcoded/`, `.blob-cache/`). Default
|
||||||
|
`<storage_path>/spool`. Ops can point it at a different disk
|
||||||
|
than tier-3 storage; multi-instance deployments accept per-
|
||||||
|
instance rebuild OR mount a shared FS here.
|
||||||
|
|
||||||
|
## Delivery order
|
||||||
|
|
||||||
|
Coarse — the trait + registry ship first (empty-impl for
|
||||||
|
`FilesReferenceSource` + `ChunksReferenceSource` mirroring today's
|
||||||
|
hardcoded SQL). New sources bolt on independently.
|
||||||
|
|
||||||
|
1. **`BlobReferenceSource` trait + registry** in
|
||||||
|
`application/ports/`. `FilesReferenceSource` and
|
||||||
|
`ChunksReferenceSource` implementations mirroring current SQL;
|
||||||
|
wire into `dedup_gc` + `blobs_consistency` behind an integration
|
||||||
|
test that proves the union equals the pre-refactor count on a
|
||||||
|
real DB.
|
||||||
|
2. **`OXICLOUD_SPOOL_DIR`** — config + `example.env` + docs +
|
||||||
|
`AppConfig::spool_dir`. Migrate `CachedBlobBackend` cache path
|
||||||
|
default to `<spool_dir>/blob-cache/`.
|
||||||
|
3. **`ThumbnailService` writes go through the backend**. New
|
||||||
|
`storage.thumbnails` table + `ThumbnailsReferenceSource`. Local
|
||||||
|
`.thumbnails/` sidecar becomes a read-through cache pattern.
|
||||||
|
4. **`ImageTranscodeService`** — same shape as thumbnails.
|
||||||
|
5. **Client-uploaded thumbnails** — new `origin` column + upload
|
||||||
|
API path if needed.
|
||||||
|
|
||||||
|
Each slice is independently mergeable. Delivery span: rough
|
||||||
|
estimate ~2 weeks end-to-end.
|
||||||
|
|
||||||
|
## Naming clarifications to land alongside this refactor
|
||||||
|
|
||||||
|
Two consumer-facing terminology issues that surfaced during the
|
||||||
|
read-side normalisation (2026-08-02). They're not code-breakers,
|
||||||
|
but they cost every new implementor a mental round-trip, so
|
||||||
|
they belong in the tier-2 sweep:
|
||||||
|
|
||||||
|
### 1. `DedupService` name is implementation-shaped, not consumer-shaped
|
||||||
|
|
||||||
|
From a consumer's perspective the service is "the thing that
|
||||||
|
reads and writes file content by hash." Deduplication is one
|
||||||
|
internal responsibility (alongside CDC chunking, ref-counting,
|
||||||
|
GC). The name `DedupService` narrates HOW it works, not WHAT it
|
||||||
|
is — new service authors read the name and don't realise they
|
||||||
|
should be routing every blob read through it.
|
||||||
|
|
||||||
|
Suggested rename: **`BlobHandler`** (or `ContentStore` /
|
||||||
|
`BlobStore` — pick one and commit). Public surface stays
|
||||||
|
identical; consumers write `Arc<BlobHandler>` and call
|
||||||
|
`blob_handler.read_blob_bytes(hash)`. Internal doc-comments
|
||||||
|
document dedup + CDC + GC as strategies.
|
||||||
|
|
||||||
|
Scope: ~35 files (grep `DedupService|dedup_service`), mechanical.
|
||||||
|
Keep as one commit inside the tier-2 refactor so reviewers see
|
||||||
|
"rename" independently from the substantive changes.
|
||||||
|
|
||||||
|
### 2. `blob` overloaded across two scales
|
||||||
|
|
||||||
|
Current usage:
|
||||||
|
|
||||||
|
- `storage.blobs` — the physical storage table; rows are BYTES
|
||||||
|
written to a backend. Post-CDC, most entries are chunks
|
||||||
|
(fragments), not whole files.
|
||||||
|
- `storage.chunk_manifests` — the CDC manifest that references a
|
||||||
|
set of `storage.blobs` rows to reconstitute a file.
|
||||||
|
- `file.blob_hash` — the hash a file row points at; either a
|
||||||
|
whole-file blob (legacy) OR a chunk-manifest (post-CDC).
|
||||||
|
|
||||||
|
The word "blob" carries two meanings: *whole-file content* (what
|
||||||
|
a user thinks of when they say "download the blob") vs
|
||||||
|
*physical byte-payload on disk* (what the storage backend
|
||||||
|
holds — may be a whole file, may be a chunk fragment).
|
||||||
|
|
||||||
|
Proposed clarification for the tier-2 sweep:
|
||||||
|
|
||||||
|
- **Blob** = the abstraction of "content of a file", identified
|
||||||
|
by BLAKE3 of the plaintext. Consumers work at this level. What
|
||||||
|
`DedupService`/`BlobHandler` returns.
|
||||||
|
- **Chunk** = a physical byte-payload written to the backend,
|
||||||
|
identified by its own BLAKE3. Storage-backend-internal.
|
||||||
|
- **Manifest** = the map from a Blob to one or more Chunks.
|
||||||
|
|
||||||
|
Schema rename (deferred, requires migration):
|
||||||
|
|
||||||
|
- `storage.blobs` → `storage.chunks` (that's what it actually holds now)
|
||||||
|
- `storage.chunk_manifests` → `storage.blob_manifests` (or keep — arguable)
|
||||||
|
- `BlobStorageBackend` trait → `ChunkStorageBackend` — reads and
|
||||||
|
writes physical chunks, not blobs
|
||||||
|
|
||||||
|
`file.blob_hash` semantics stay — references a Blob via its
|
||||||
|
manifest OR (for pre-CDC legacy) points directly at a single-chunk
|
||||||
|
Blob whose hash equals its lone chunk's hash.
|
||||||
|
|
||||||
|
Scope for this rename: ~23 files touch the SQL, plus a migration
|
||||||
|
for the table rename. Not free. Ship AFTER the tier-2 write-side
|
||||||
|
lands so we don't stack schema changes.
|
||||||
|
|
||||||
|
## Non-goals
|
||||||
|
|
||||||
|
- **Tantivy `.search-index/`** — memory-mapped by design, doesn't
|
||||||
|
fit the blob-storage abstraction. Separate future decision:
|
||||||
|
keep local, snapshot-to-backend periodically, or retire Tantivy
|
||||||
|
for PG-native full-text.
|
||||||
|
- **`.plugin-logs/`** — ops-local operational data, not user
|
||||||
|
content. Stays local.
|
||||||
|
- **Client thumbnail negotiation protocol** — the wire-level API
|
||||||
|
for how clients push their previews. Design piece for the
|
||||||
|
photo/mobile team when there's a real feature ask.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- `docs/architecture/backend-storage.md` — the wrapper stack, header
|
||||||
|
format, consistency check, migration semantics that derived
|
||||||
|
content inherits.
|
||||||
|
- `docs/plan/storage-multi-entry.md` — tier-3 configuration model.
|
||||||
|
- `docs/plan/storage-key-rotation.md` — encryption/rotation applies
|
||||||
|
to derived blobs too.
|
||||||
|
- `src/AGENTS.md` — the read-side rule enforcing backend
|
||||||
|
abstraction (already shipped alongside this plan doc).
|
||||||
|
- Memory note `project_services_bypassing_blob_backend` — audit
|
||||||
|
history of the pre-normalisation bypasses.
|
||||||
+11
@@ -17,6 +17,17 @@ OXICLOUD_STORAGE_PATH=./storage
|
|||||||
# Path to static files directory (default: ./static)
|
# Path to static files directory (default: ./static)
|
||||||
OXICLOUD_STATIC_PATH=./static
|
OXICLOUD_STATIC_PATH=./static
|
||||||
|
|
||||||
|
# Directory for tier-1 temporary data — pure scratch, safe to lose at reboot.
|
||||||
|
# Services that need a filesystem path (audio ID3/MP3 duration, video EXIF via
|
||||||
|
# ffprobe/nom-exif, etc.) stream blobs here from the active backend before
|
||||||
|
# handing the path to the extractor. The file is auto-removed after each use.
|
||||||
|
#
|
||||||
|
# Default: system tempdir (`std::env::temp_dir()`, honours $TMPDIR / $TMP).
|
||||||
|
# On Linux `/tmp` is often mounted as tmpfs (RAM-backed); high-concurrency
|
||||||
|
# production deployments concerned about RAM should point this at a
|
||||||
|
# disk-backed dir (e.g. /var/lib/oxicloud/tmp).
|
||||||
|
#OXICLOUD_TEMP_DIR=/var/lib/oxicloud/tmp
|
||||||
|
|
||||||
# Server port (default: 8086)
|
# Server port (default: 8086)
|
||||||
OXICLOUD_SERVER_PORT=8086
|
OXICLOUD_SERVER_PORT=8086
|
||||||
|
|
||||||
|
|||||||
@@ -369,3 +369,7 @@ load-baseline:
|
|||||||
# just db
|
# just db
|
||||||
load-seed:
|
load-seed:
|
||||||
cargo run --bin load-seed -- --depth 5 --fanout 4 --files-per-leaf 3
|
cargo run --bin load-seed -- --depth 5 --fanout 4 --files-per-leaf 3
|
||||||
|
|
||||||
|
# Check and test everything
|
||||||
|
# recommanded before pull request
|
||||||
|
pre-pull-request: check fe-check audit check-migrations test test-integration fe-test build api-test fe-build-e2e front-test
|
||||||
|
|||||||
@@ -14,3 +14,11 @@ Non-obvious rules that trip up new code. Terse on purpose.
|
|||||||
|
|
||||||
- Any new endpoint that mints or consumes credentials/tokens must consult one of the `is_*_login_allowed()` helpers, not the raw allowlist.
|
- Any new endpoint that mints or consumes credentials/tokens must consult one of the `is_*_login_allowed()` helpers, not the raw allowlist.
|
||||||
- Any new "policy-disabled" refusal must emit an `audit`-target line before returning — matches `auth.login_rejected`, `magic_link.redemption_rejected` conventions.
|
- Any new "policy-disabled" refusal must emit an `audit`-target line before returning — matches `auth.login_rejected`, `magic_link.redemption_rejected` conventions.
|
||||||
|
|
||||||
|
## Storage backend access
|
||||||
|
|
||||||
|
- **Read blob content through `Arc<DedupService>`.** It's the ONE canonical read abstraction — CDC-manifest-aware (`file.blob_hash` may reference a chunk manifest, not a blob), backend-agnostic (Local/S3/Azure), wrapper-transparent (encryption/retry/cache). Never take `Arc<dyn BlobStorageBackend>` directly in a service that reads content; you'll silently break on any file ≥ 64 KiB (`CDC_MIN_CHUNK`). Follow `thumbnail_service`, `audio_metadata_service`, `media_metadata_service`, `face_indexing_service`, `search_index::content_index_worker` as reference impls.
|
||||||
|
- **Reads use `DedupService` methods**: `dedup.read_blob_bytes(hash)` for byte-slice analyzers (ONNX, EXIF, ID3-via-Reader), `dedup.stream_blob_to_tempfile(hash, &temp_dir, ".ext")` for crates that only accept `&Path` (mp3_duration, ffprobe, `nom-exif` video), `dedup.read_blob_stream(hash)` for streaming to a downstream `Stream` consumer.
|
||||||
|
- **Never hand-craft blob paths.** No `blob_root: PathBuf` fields, no `<storage>/.blobs/<xx>/<hash>.blob` constructions. `BlobStorageBackend::local_blob_path` returns `None` under `EncryptedBlobBackend`; do not rely on it. The three services that did this pre-2026-08 (audio/media/face) are the anti-pattern — see memory `project_services_bypassing_blob_backend`.
|
||||||
|
- **Persistent state = backend**, not `<storage_path>/*` sidecars. Local sidecars (`.thumbnails/`, `.transcoded/`, `.blob-cache/`, `.search-index/`, `.plugin-logs/`, `.uploads/`) are only for caches (regenerable) or truly-temp scratch (deleted on drop). Anything a user would notice losing → blob backend. Tier-2 migration plan: `docs/plan/derived-blobs.md`.
|
||||||
|
- **Temp files use `OXICLOUD_TEMP_DIR`** via the shared config path (`AppConfig::temp_dir`) — not raw `std::env::temp_dir()`. Ops point it at real disk on RAM-constrained Linux deployments (default `/tmp` = tmpfs = RAM).
|
||||||
|
|||||||
@@ -2252,6 +2252,19 @@ pub struct AppConfig {
|
|||||||
pub storage_path: PathBuf,
|
pub storage_path: PathBuf,
|
||||||
/// Static files directory path
|
/// Static files directory path
|
||||||
pub static_path: PathBuf,
|
pub static_path: PathBuf,
|
||||||
|
/// Directory for tier-1 temporary data — pure scratch, safe to
|
||||||
|
/// lose at reboot. Backend `stream_to_tempfile` writes here so
|
||||||
|
/// extractors that require a `&Path` (id3, mp3_duration,
|
||||||
|
/// ffprobe, nom-exif video) can operate on a local file
|
||||||
|
/// without the service ever seeing a raw blob path.
|
||||||
|
///
|
||||||
|
/// Env: `OXICLOUD_TEMP_DIR`. Default: `std::env::temp_dir()`
|
||||||
|
/// (respects `$TMPDIR`). On Linux this is typically `/tmp`,
|
||||||
|
/// often mounted as tmpfs (RAM-backed); production
|
||||||
|
/// deployments concerned about physical RAM under
|
||||||
|
/// high concurrency should point this at a disk-backed
|
||||||
|
/// directory (e.g. `/var/lib/oxicloud/tmp`).
|
||||||
|
pub temp_dir: PathBuf,
|
||||||
/// Server port
|
/// Server port
|
||||||
pub server_port: u16,
|
pub server_port: u16,
|
||||||
/// Server host
|
/// Server host
|
||||||
@@ -2356,6 +2369,7 @@ impl Default for AppConfig {
|
|||||||
Self {
|
Self {
|
||||||
storage_path: PathBuf::from("./storage"),
|
storage_path: PathBuf::from("./storage"),
|
||||||
static_path: PathBuf::from("./static"),
|
static_path: PathBuf::from("./static"),
|
||||||
|
temp_dir: env::temp_dir(),
|
||||||
server_port: 8086,
|
server_port: 8086,
|
||||||
server_host: "127.0.0.1".to_string(),
|
server_host: "127.0.0.1".to_string(),
|
||||||
cache: CacheConfig::default(),
|
cache: CacheConfig::default(),
|
||||||
@@ -2394,6 +2408,10 @@ impl AppConfig {
|
|||||||
config.static_path = PathBuf::from(static_path);
|
config.static_path = PathBuf::from(static_path);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if let Ok(temp_dir) = env::var("OXICLOUD_TEMP_DIR") {
|
||||||
|
config.temp_dir = PathBuf::from(temp_dir);
|
||||||
|
}
|
||||||
|
|
||||||
if let Ok(server_port) = env::var("OXICLOUD_SERVER_PORT")
|
if let Ok(server_port) = env::var("OXICLOUD_SERVER_PORT")
|
||||||
&& let Ok(port) = server_port.parse::<u16>()
|
&& let Ok(port) = server_port.parse::<u16>()
|
||||||
{
|
{
|
||||||
|
|||||||
+14
-9
@@ -468,11 +468,11 @@ impl AppServiceFactory {
|
|||||||
);
|
);
|
||||||
|
|
||||||
// Audio metadata service — created here so it can be wired into file_lifecycle.
|
// Audio metadata service — created here so it can be wired into file_lifecycle.
|
||||||
let audio_metadata_service = self.create_audio_metadata_service(db_pool);
|
let audio_metadata_service = self.create_audio_metadata_service(db_pool, &dedup_service);
|
||||||
|
|
||||||
// Image/video capture-metadata service — extracts EXIF/container capture
|
// Image/video capture-metadata service — extracts EXIF/container capture
|
||||||
// dates so the Photos timeline groups by real capture time, not upload time.
|
// dates so the Photos timeline groups by real capture time, not upload time.
|
||||||
let media_metadata_service = self.create_media_metadata_service(db_pool);
|
let media_metadata_service = self.create_media_metadata_service(db_pool, &dedup_service);
|
||||||
|
|
||||||
// ThumbnailRefreshHook: handles FileLifecycleHook events (create/update/delete).
|
// ThumbnailRefreshHook: handles FileLifecycleHook events (create/update/delete).
|
||||||
// Implemented on ThumbnailRefreshHook (not ThumbnailService) to avoid circular Arc:
|
// Implemented on ThumbnailRefreshHook (not ThumbnailService) to avoid circular Arc:
|
||||||
@@ -544,7 +544,7 @@ impl AppServiceFactory {
|
|||||||
}
|
}
|
||||||
fls = fls.with_hook(media_metadata_service.clone());
|
fls = fls.with_hook(media_metadata_service.clone());
|
||||||
if self.config.features.enable_faces {
|
if self.config.features.enable_faces {
|
||||||
fls = fls.with_hook(self.create_face_indexing_service(db_pool));
|
fls = fls.with_hook(self.create_face_indexing_service(db_pool, &dedup_service));
|
||||||
}
|
}
|
||||||
let file_lifecycle = Arc::new(fls);
|
let file_lifecycle = Arc::new(fls);
|
||||||
|
|
||||||
@@ -950,15 +950,16 @@ impl AppServiceFactory {
|
|||||||
pub fn create_audio_metadata_service(
|
pub fn create_audio_metadata_service(
|
||||||
&self,
|
&self,
|
||||||
db_pool: &Arc<PgPool>,
|
db_pool: &Arc<PgPool>,
|
||||||
|
dedup: &Arc<crate::infrastructure::services::dedup_service::DedupService>,
|
||||||
) -> Option<Arc<AudioMetadataService>> {
|
) -> Option<Arc<AudioMetadataService>> {
|
||||||
if !self.config.features.enable_music {
|
if !self.config.features.enable_music {
|
||||||
tracing::info!("Audio metadata service is disabled (music feature disabled)");
|
tracing::info!("Audio metadata service is disabled (music feature disabled)");
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
let blob_root = self.storage_path.join(".blobs");
|
|
||||||
Some(Arc::new(AudioMetadataService::new(
|
Some(Arc::new(AudioMetadataService::new(
|
||||||
db_pool.clone(),
|
db_pool.clone(),
|
||||||
blob_root,
|
dedup.clone(),
|
||||||
|
self.config.temp_dir.clone(),
|
||||||
)))
|
)))
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -967,9 +968,13 @@ impl AppServiceFactory {
|
|||||||
pub fn create_media_metadata_service(
|
pub fn create_media_metadata_service(
|
||||||
&self,
|
&self,
|
||||||
db_pool: &Arc<PgPool>,
|
db_pool: &Arc<PgPool>,
|
||||||
|
dedup: &Arc<crate::infrastructure::services::dedup_service::DedupService>,
|
||||||
) -> Arc<MediaMetadataService> {
|
) -> Arc<MediaMetadataService> {
|
||||||
let blob_root = self.storage_path.join(".blobs");
|
Arc::new(MediaMetadataService::new(
|
||||||
Arc::new(MediaMetadataService::new(db_pool.clone(), blob_root))
|
db_pool.clone(),
|
||||||
|
dedup.clone(),
|
||||||
|
self.config.temp_dir.clone(),
|
||||||
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Creates the trash service
|
/// Creates the trash service
|
||||||
@@ -1137,13 +1142,13 @@ impl AppServiceFactory {
|
|||||||
pub fn create_face_indexing_service(
|
pub fn create_face_indexing_service(
|
||||||
&self,
|
&self,
|
||||||
db_pool: &Arc<PgPool>,
|
db_pool: &Arc<PgPool>,
|
||||||
|
dedup: &Arc<crate::infrastructure::services::dedup_service::DedupService>,
|
||||||
) -> Arc<crate::infrastructure::services::face_indexing_service::FaceIndexingService> {
|
) -> Arc<crate::infrastructure::services::face_indexing_service::FaceIndexingService> {
|
||||||
let blob_root = self.storage_path.join(".blobs");
|
|
||||||
let analyzer = self.build_face_analyzer();
|
let analyzer = self.build_face_analyzer();
|
||||||
Arc::new(
|
Arc::new(
|
||||||
crate::infrastructure::services::face_indexing_service::FaceIndexingService::new(
|
crate::infrastructure::services::face_indexing_service::FaceIndexingService::new(
|
||||||
db_pool.clone(),
|
db_pool.clone(),
|
||||||
blob_root,
|
dedup.clone(),
|
||||||
analyzer,
|
analyzer,
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ use uuid::Uuid;
|
|||||||
|
|
||||||
use crate::application::ports::file_lifecycle::FileLifecycleHook;
|
use crate::application::ports::file_lifecycle::FileLifecycleHook;
|
||||||
use crate::common::errors::DomainError;
|
use crate::common::errors::DomainError;
|
||||||
|
use crate::infrastructure::services::dedup_service::DedupService;
|
||||||
|
|
||||||
#[derive(Debug, FromRow)]
|
#[derive(Debug, FromRow)]
|
||||||
pub struct AudioFileRow {
|
pub struct AudioFileRow {
|
||||||
@@ -17,22 +18,32 @@ pub struct AudioFileRow {
|
|||||||
|
|
||||||
pub struct AudioMetadataService {
|
pub struct AudioMetadataService {
|
||||||
pool: Arc<PgPool>,
|
pool: Arc<PgPool>,
|
||||||
blob_root: PathBuf,
|
/// CDC-aware blob reader. Same abstraction `thumbnail_service` uses —
|
||||||
|
/// hides both the chunk-manifest concatenation and the underlying
|
||||||
|
/// `BlobStorageBackend` wrapper stack.
|
||||||
|
dedup: Arc<DedupService>,
|
||||||
|
/// Tier-1 scratch directory for `stream_blob_to_tempfile`. Pulled
|
||||||
|
/// from `AppConfig::temp_dir` (env `OXICLOUD_TEMP_DIR`) at DI time.
|
||||||
|
temp_dir: PathBuf,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl AudioMetadataService {
|
impl AudioMetadataService {
|
||||||
pub fn new(pool: Arc<PgPool>, blob_root: PathBuf) -> Self {
|
pub fn new(pool: Arc<PgPool>, dedup: Arc<DedupService>, temp_dir: PathBuf) -> Self {
|
||||||
Self { pool, blob_root }
|
Self {
|
||||||
|
pool,
|
||||||
|
dedup,
|
||||||
|
temp_dir,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn is_audio_file(mime_type: &str) -> bool {
|
pub fn is_audio_file(mime_type: &str) -> bool {
|
||||||
mime_type.starts_with("audio/")
|
mime_type.starts_with("audio/")
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn spawn_extraction_background(service: Arc<Self>, file_id: Uuid, file_path: PathBuf) {
|
pub fn spawn_extraction_background(service: Arc<Self>, file_id: Uuid, blob_hash: String) {
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
tracing::info!("🎵 Extracting audio metadata for: {}", file_id);
|
tracing::info!("🎵 Extracting audio metadata for: {}", file_id);
|
||||||
if let Err(e) = service.extract_and_save(&file_id, &file_path).await {
|
if let Err(e) = service.extract_and_save(&file_id, &blob_hash).await {
|
||||||
tracing::warn!("Failed to extract audio metadata: {}", e);
|
tracing::warn!("Failed to extract audio metadata: {}", e);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
@@ -41,22 +52,17 @@ impl AudioMetadataService {
|
|||||||
pub fn spawn_extraction_with_delete_background(
|
pub fn spawn_extraction_with_delete_background(
|
||||||
service: Arc<Self>,
|
service: Arc<Self>,
|
||||||
file_id: Uuid,
|
file_id: Uuid,
|
||||||
file_path: PathBuf,
|
blob_hash: String,
|
||||||
) {
|
) {
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
tracing::info!("🎵 Updating audio metadata for: {}", file_id);
|
tracing::info!("🎵 Updating audio metadata for: {}", file_id);
|
||||||
let _ = service.delete_metadata(&file_id).await;
|
let _ = service.delete_metadata(&file_id).await;
|
||||||
if let Err(e) = service.extract_and_save(&file_id, &file_path).await {
|
if let Err(e) = service.extract_and_save(&file_id, &blob_hash).await {
|
||||||
tracing::warn!("Failed to update audio metadata: {}", e);
|
tracing::warn!("Failed to update audio metadata: {}", e);
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
fn blob_path(&self, hash: &str) -> PathBuf {
|
|
||||||
let prefix = &hash[0..2];
|
|
||||||
self.blob_root.join(prefix).join(format!("{}.blob", hash))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Extract ID3 tag and MP3 duration from a file.
|
/// Extract ID3 tag and MP3 duration from a file.
|
||||||
///
|
///
|
||||||
/// All I/O is synchronous (id3 + mp3_duration crates), so this MUST
|
/// All I/O is synchronous (id3 + mp3_duration crates), so this MUST
|
||||||
@@ -104,15 +110,26 @@ impl AudioMetadataService {
|
|||||||
pub async fn extract_and_save(
|
pub async fn extract_and_save(
|
||||||
&self,
|
&self,
|
||||||
file_id: &Uuid,
|
file_id: &Uuid,
|
||||||
file_path: &Path,
|
blob_hash: &str,
|
||||||
) -> Result<(), DomainError> {
|
) -> Result<(), DomainError> {
|
||||||
info!(
|
info!(
|
||||||
"AudioMetadataService: blob_root={:?}, file_id={}, file_path={:?}",
|
"AudioMetadataService: extracting file_id={}, blob_hash={}",
|
||||||
self.blob_root, file_id, file_path,
|
file_id, blob_hash,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Stream the blob (CDC-aware — chunks concatenated on the fly for
|
||||||
|
// chunked files; wrapper stack handles encryption + retry + cache)
|
||||||
|
// to a tempfile in the configured tier-1 temp dir, then hand its
|
||||||
|
// `.path()` to the id3 + mp3_duration crates which only expose
|
||||||
|
// `from_path` APIs. Peak process-heap = one chunk (~1 MiB)
|
||||||
|
// regardless of MP3 size. Guard drops → tempfile auto-removed.
|
||||||
|
let named = self
|
||||||
|
.dedup
|
||||||
|
.stream_blob_to_tempfile(blob_hash, &self.temp_dir, ".mp3")
|
||||||
|
.await?;
|
||||||
|
let path = named.path().to_path_buf();
|
||||||
|
|
||||||
// ── Sync I/O on the blocking thread pool (never stalls Tokio workers) ──
|
// ── Sync I/O on the blocking thread pool (never stalls Tokio workers) ──
|
||||||
let path = file_path.to_path_buf();
|
|
||||||
let metadata = tokio::task::spawn_blocking(move || Self::extract_metadata_blocking(&path))
|
let metadata = tokio::task::spawn_blocking(move || Self::extract_metadata_blocking(&path))
|
||||||
.await
|
.await
|
||||||
.map_err(|e| {
|
.map_err(|e| {
|
||||||
@@ -121,6 +138,10 @@ impl AudioMetadataService {
|
|||||||
format!("spawn_blocking join error: {e}"),
|
format!("spawn_blocking join error: {e}"),
|
||||||
)
|
)
|
||||||
})?;
|
})?;
|
||||||
|
// Explicitly hold `named` alive until after the extraction — the
|
||||||
|
// `spawn_blocking` closure only borrows the raw `path`, so the
|
||||||
|
// guard must not drop while the extractor is running.
|
||||||
|
drop(named);
|
||||||
|
|
||||||
let Some(m) = metadata else {
|
let Some(m) = metadata else {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
@@ -208,8 +229,10 @@ impl AudioMetadataService {
|
|||||||
let audio_file = row.map_err(|e| {
|
let audio_file = row.map_err(|e| {
|
||||||
DomainError::database_error(format!("Failed to fetch audio file row: {}", e))
|
DomainError::database_error(format!("Failed to fetch audio file row: {}", e))
|
||||||
})?;
|
})?;
|
||||||
let file_path = self.blob_path(&audio_file.blob_hash);
|
match self
|
||||||
match self.extract_and_save(&audio_file.file_id, &file_path).await {
|
.extract_and_save(&audio_file.file_id, &audio_file.blob_hash)
|
||||||
|
.await
|
||||||
|
{
|
||||||
Ok(()) => processed += 1,
|
Ok(()) => processed += 1,
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
warn!(
|
warn!(
|
||||||
@@ -313,8 +336,7 @@ impl AudioMetadataService {
|
|||||||
}
|
}
|
||||||
Ok(_) => {
|
Ok(_) => {
|
||||||
// No existing metadata found — original not yet processed; fall back.
|
// No existing metadata found — original not yet processed; fall back.
|
||||||
let file_path = service.blob_path(&blob_hash);
|
if let Err(e) = service.extract_and_save(&new_file_id, &blob_hash).await {
|
||||||
if let Err(e) = service.extract_and_save(&new_file_id, &file_path).await {
|
|
||||||
warn!(
|
warn!(
|
||||||
"Failed to extract audio metadata for {}: {}",
|
"Failed to extract audio metadata for {}: {}",
|
||||||
new_file_id, e
|
new_file_id, e
|
||||||
@@ -368,10 +390,11 @@ impl FileLifecycleHook for AudioMetadataService {
|
|||||||
};
|
};
|
||||||
let service = Arc::new(Self {
|
let service = Arc::new(Self {
|
||||||
pool: self.pool.clone(),
|
pool: self.pool.clone(),
|
||||||
blob_root: self.blob_root.clone(),
|
dedup: self.dedup.clone(),
|
||||||
|
temp_dir: self.temp_dir.clone(),
|
||||||
});
|
});
|
||||||
if is_new_blob {
|
if is_new_blob {
|
||||||
Self::spawn_extraction_background(service, uuid, self.blob_path(blob_hash));
|
Self::spawn_extraction_background(service, uuid, blob_hash.to_string());
|
||||||
} else {
|
} else {
|
||||||
Self::clone_or_extract_background(service, uuid, blob_hash.to_string());
|
Self::clone_or_extract_background(service, uuid, blob_hash.to_string());
|
||||||
}
|
}
|
||||||
@@ -400,7 +423,8 @@ impl FileLifecycleHook for AudioMetadataService {
|
|||||||
};
|
};
|
||||||
let service = Arc::new(Self {
|
let service = Arc::new(Self {
|
||||||
pool: self.pool.clone(),
|
pool: self.pool.clone(),
|
||||||
blob_root: self.blob_root.clone(),
|
dedup: self.dedup.clone(),
|
||||||
|
temp_dir: self.temp_dir.clone(),
|
||||||
});
|
});
|
||||||
Self::clone_from_source_background(service, uuid, source_uuid, blob_hash.to_string());
|
Self::clone_from_source_background(service, uuid, source_uuid, blob_hash.to_string());
|
||||||
}
|
}
|
||||||
@@ -415,9 +439,10 @@ impl FileLifecycleHook for AudioMetadataService {
|
|||||||
};
|
};
|
||||||
let service = Arc::new(Self {
|
let service = Arc::new(Self {
|
||||||
pool: self.pool.clone(),
|
pool: self.pool.clone(),
|
||||||
blob_root: self.blob_root.clone(),
|
dedup: self.dedup.clone(),
|
||||||
|
temp_dir: self.temp_dir.clone(),
|
||||||
});
|
});
|
||||||
Self::spawn_extraction_with_delete_background(service, uuid, self.blob_path(blob_hash));
|
Self::spawn_extraction_with_delete_background(service, uuid, blob_hash.to_string());
|
||||||
}
|
}
|
||||||
|
|
||||||
fn on_file_deleted(&self, _file_id: &str) {
|
fn on_file_deleted(&self, _file_id: &str) {
|
||||||
|
|||||||
@@ -2056,6 +2056,56 @@ impl DedupService {
|
|||||||
Ok(Bytes::from(data))
|
Ok(Bytes::from(data))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Stream a blob to a temp file for extractors that only accept a
|
||||||
|
/// filesystem `Path` (id3, mp3_duration, ffprobe, nom-exif video).
|
||||||
|
/// CDC-aware — reads through [`Self::read_blob_stream`] so a chunked
|
||||||
|
/// file's chunks are concatenated on the fly. Peak process-heap =
|
||||||
|
/// one chunk (~1 MiB) regardless of blob size.
|
||||||
|
///
|
||||||
|
/// `temp_dir` is the destination directory (typically
|
||||||
|
/// `AppConfig::temp_dir`, from env `OXICLOUD_TEMP_DIR`). `suffix`
|
||||||
|
/// is appended to the tempfile name (e.g. `".mp3"`, `".jpg"`) so
|
||||||
|
/// content-sniffing extractors get a hint. The returned
|
||||||
|
/// `NamedTempFile` auto-removes on drop; callers pass `.path()`
|
||||||
|
/// to the extractor, then let the guard fall out of scope.
|
||||||
|
pub async fn stream_blob_to_tempfile(
|
||||||
|
&self,
|
||||||
|
hash: &str,
|
||||||
|
temp_dir: &std::path::Path,
|
||||||
|
suffix: &str,
|
||||||
|
) -> Result<tempfile::NamedTempFile, DomainError> {
|
||||||
|
use tokio::io::AsyncWriteExt;
|
||||||
|
let named = tempfile::Builder::new()
|
||||||
|
.prefix("oxi-blob-")
|
||||||
|
.suffix(suffix)
|
||||||
|
.tempfile_in(temp_dir)
|
||||||
|
.map_err(|e| {
|
||||||
|
DomainError::internal_error("Dedup", format!("mktemp in {:?}: {e}", temp_dir))
|
||||||
|
})?;
|
||||||
|
// Re-open with tokio's async File so we can await writes.
|
||||||
|
let path = named.path().to_path_buf();
|
||||||
|
let mut file = tokio::fs::OpenOptions::new()
|
||||||
|
.write(true)
|
||||||
|
.truncate(true)
|
||||||
|
.open(&path)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("Dedup", format!("reopen temp: {e}")))?;
|
||||||
|
// CDC-aware: manifest lookup + chunk concat OR legacy backend passthrough.
|
||||||
|
let mut stream = self.read_blob_stream(hash).await?;
|
||||||
|
while let Some(chunk) = stream.next().await {
|
||||||
|
let bytes = chunk
|
||||||
|
.map_err(|e| DomainError::internal_error("Dedup", format!("stream chunk: {e}")))?;
|
||||||
|
file.write_all(&bytes)
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("Dedup", format!("temp write: {e}")))?;
|
||||||
|
}
|
||||||
|
file.flush()
|
||||||
|
.await
|
||||||
|
.map_err(|e| DomainError::internal_error("Dedup", format!("temp flush: {e}")))?;
|
||||||
|
drop(file);
|
||||||
|
Ok(named)
|
||||||
|
}
|
||||||
|
|
||||||
/// Stream a byte range — CDC-aware with legacy fallback.
|
/// Stream a byte range — CDC-aware with legacy fallback.
|
||||||
///
|
///
|
||||||
/// For CDC files: calculates which chunks overlap the requested range,
|
/// For CDC files: calculates which chunks overlap the requested range,
|
||||||
@@ -3971,6 +4021,83 @@ mod rechunk_integration_tests {
|
|||||||
|
|
||||||
cleanup(&pool, &hash, &files).await;
|
cleanup(&pool, &hash, &files).await;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ─── stream_blob_to_tempfile — CDC-aware read to a filesystem path ───
|
||||||
|
//
|
||||||
|
// Regression tests for the fix landed on `fix/services-use-blob-abstraction`:
|
||||||
|
// audio_metadata_service, media_metadata_service, and face_indexing_service
|
||||||
|
// all read blob content via DedupService (`read_blob_bytes` /
|
||||||
|
// `stream_blob_to_tempfile`), NOT the raw `BlobStorageBackend`. If someone
|
||||||
|
// reverts a service to `backend.get_blob_stream(hash)`, this test fails
|
||||||
|
// because `hash` is a chunk-manifest hash — the physical backend has no
|
||||||
|
// blob at that key. Bug returns silently otherwise; these tests catch it.
|
||||||
|
|
||||||
|
/// Local backend: seed a > 64 KiB blob, rechunk to CDC, then call
|
||||||
|
/// `stream_blob_to_tempfile` and verify the tempfile contents match
|
||||||
|
/// the original. Proves the CDC chunk-concat path works.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn stream_blob_to_tempfile_reads_cdc_chunked_local() {
|
||||||
|
let pool = test_pool().await;
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let svc = local_svc(&pool, &dir).await;
|
||||||
|
|
||||||
|
// 200 KiB → forced multi-chunk after rechunk_legacy_blobs.
|
||||||
|
let data = content(200 * 1024, 33);
|
||||||
|
let (hash, files) = seed_legacy(&svc, &pool, &dir, &data, 1, "cdc-local", None).await;
|
||||||
|
svc.rechunk_legacy_blobs().await.expect("sweep");
|
||||||
|
|
||||||
|
// Sanity: rechunk actually produced a manifest (i.e. we're on the
|
||||||
|
// CDC path, not the legacy-fallback branch of read_blob_stream).
|
||||||
|
assert!(
|
||||||
|
manifest(&pool, &hash).await.is_some(),
|
||||||
|
"expected a CDC manifest after rechunk (test wouldn't cover the bug otherwise)"
|
||||||
|
);
|
||||||
|
|
||||||
|
// New method — the entry point audio/media services use.
|
||||||
|
let temp_dir = TempDir::new().unwrap();
|
||||||
|
let named = svc
|
||||||
|
.stream_blob_to_tempfile(&hash, temp_dir.path(), ".bin")
|
||||||
|
.await
|
||||||
|
.expect("stream_blob_to_tempfile must succeed on CDC-chunked blob");
|
||||||
|
|
||||||
|
let round_tripped = tokio::fs::read(named.path()).await.expect("read tempfile");
|
||||||
|
assert_eq!(round_tripped, data, "tempfile content must match original");
|
||||||
|
|
||||||
|
cleanup(&pool, &hash, &files).await;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Encrypted backend variant — proves the wrapper stack (decryption
|
||||||
|
/// on read) is honoured. Same regression class: if a service reads
|
||||||
|
/// raw ciphertext instead of going through DedupService, this fails.
|
||||||
|
#[tokio::test]
|
||||||
|
async fn stream_blob_to_tempfile_reads_cdc_chunked_encrypted() {
|
||||||
|
let pool = test_pool().await;
|
||||||
|
let dir = TempDir::new().unwrap();
|
||||||
|
let svc = encrypted_svc(&pool, &dir).await;
|
||||||
|
|
||||||
|
let data = content(150 * 1024, 77);
|
||||||
|
let (hash, files) = seed_legacy(&svc, &pool, &dir, &data, 1, "cdc-enc", None).await;
|
||||||
|
svc.rechunk_legacy_blobs().await.expect("sweep");
|
||||||
|
|
||||||
|
assert!(
|
||||||
|
manifest(&pool, &hash).await.is_some(),
|
||||||
|
"expected a CDC manifest after rechunk"
|
||||||
|
);
|
||||||
|
|
||||||
|
let temp_dir = TempDir::new().unwrap();
|
||||||
|
let named = svc
|
||||||
|
.stream_blob_to_tempfile(&hash, temp_dir.path(), ".bin")
|
||||||
|
.await
|
||||||
|
.expect("stream_blob_to_tempfile must succeed on encrypted CDC blob");
|
||||||
|
|
||||||
|
let round_tripped = tokio::fs::read(named.path()).await.expect("read tempfile");
|
||||||
|
assert_eq!(
|
||||||
|
round_tripped, data,
|
||||||
|
"tempfile content must match original plaintext (wrapper stack must decrypt transparently)"
|
||||||
|
);
|
||||||
|
|
||||||
|
cleanup(&pool, &hash, &files).await;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// ─────────────────────────────────────────────────────────────────────────────
|
// ─────────────────────────────────────────────────────────────────────────────
|
||||||
|
|||||||
@@ -1,14 +1,15 @@
|
|||||||
//! Face indexing as a `FileLifecycleHook`.
|
//! Face indexing as a `FileLifecycleHook`.
|
||||||
//!
|
//!
|
||||||
//! On image upload it detects + embeds faces (off the request path, in a
|
//! On image upload it detects + embeds faces (off the request path, in a
|
||||||
//! background task) and stores them. It mirrors `MediaMetadataService`: reads
|
//! background task) and stores them. Mirrors `ThumbnailService`: reads the
|
||||||
//! the blob from the local `.blobs` tree, is dedup-aware (identical uploads
|
//! blob through `DedupService` (CDC-manifest lookup, wrapper-stack
|
||||||
//! clone an existing file's faces instead of re-running inference), and is
|
//! delegation, encryption transparency — the service sees none of that),
|
||||||
//! completely inert when no model is configured (`FaceAnalyzerPort::is_ready()
|
//! is dedup-aware (identical uploads clone an existing file's faces
|
||||||
//! == false`) — so the feature compiles and runs with the default no-op
|
//! instead of re-running inference), and is completely inert when no
|
||||||
//! analyzer until the operator wires a real ONNX model.
|
//! model is configured (`FaceAnalyzerPort::is_ready() == false`) so the
|
||||||
|
//! feature compiles and runs with the default no-op analyzer until the
|
||||||
|
//! operator wires a real ONNX model.
|
||||||
|
|
||||||
use std::path::{Path, PathBuf};
|
|
||||||
use std::sync::Arc;
|
use std::sync::Arc;
|
||||||
|
|
||||||
use chrono::Utc;
|
use chrono::Utc;
|
||||||
@@ -20,6 +21,7 @@ use crate::application::ports::file_lifecycle::FileLifecycleHook;
|
|||||||
use crate::common::errors::DomainError;
|
use crate::common::errors::DomainError;
|
||||||
use crate::domain::entities::face::Face;
|
use crate::domain::entities::face::Face;
|
||||||
use crate::infrastructure::repositories::pg::FacePgRepository;
|
use crate::infrastructure::repositories::pg::FacePgRepository;
|
||||||
|
use crate::infrastructure::services::dedup_service::DedupService;
|
||||||
|
|
||||||
/// Minimum detector confidence for a face to be stored.
|
/// Minimum detector confidence for a face to be stored.
|
||||||
const MIN_DET_SCORE: f32 = 0.6;
|
const MIN_DET_SCORE: f32 = 0.6;
|
||||||
@@ -48,7 +50,10 @@ pub struct FaceIndexingService {
|
|||||||
pool: Arc<PgPool>,
|
pool: Arc<PgPool>,
|
||||||
repo: Arc<FacePgRepository>,
|
repo: Arc<FacePgRepository>,
|
||||||
analyzer: Arc<dyn FaceAnalyzerPort>,
|
analyzer: Arc<dyn FaceAnalyzerPort>,
|
||||||
blob_root: PathBuf,
|
/// CDC-aware blob reader. Same abstraction `thumbnail_service` uses —
|
||||||
|
/// hides both the chunk-manifest concatenation and the underlying
|
||||||
|
/// `BlobStorageBackend` wrapper stack.
|
||||||
|
dedup: Arc<DedupService>,
|
||||||
/// Bounds concurrent indexing tasks. The lifecycle hooks spawn one
|
/// Bounds concurrent indexing tasks. The lifecycle hooks spawn one
|
||||||
/// task per uploaded/copied image with no ceiling, so a bulk upload
|
/// task per uploaded/copied image with no ceiling, so a bulk upload
|
||||||
/// used to fan out N simultaneous full-image reads + decodes +
|
/// used to fan out N simultaneous full-image reads + decodes +
|
||||||
@@ -60,23 +65,21 @@ pub struct FaceIndexingService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
impl FaceIndexingService {
|
impl FaceIndexingService {
|
||||||
pub fn new(pool: Arc<PgPool>, blob_root: PathBuf, analyzer: Arc<dyn FaceAnalyzerPort>) -> Self {
|
pub fn new(
|
||||||
|
pool: Arc<PgPool>,
|
||||||
|
dedup: Arc<DedupService>,
|
||||||
|
analyzer: Arc<dyn FaceAnalyzerPort>,
|
||||||
|
) -> Self {
|
||||||
let repo = Arc::new(FacePgRepository::new(pool.clone()));
|
let repo = Arc::new(FacePgRepository::new(pool.clone()));
|
||||||
Self {
|
Self {
|
||||||
pool,
|
pool,
|
||||||
repo,
|
repo,
|
||||||
analyzer,
|
analyzer,
|
||||||
blob_root,
|
dedup,
|
||||||
index_semaphore: Arc::new(tokio::sync::Semaphore::new(max_concurrent_index())),
|
index_semaphore: Arc::new(tokio::sync::Semaphore::new(max_concurrent_index())),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Local path of a blob: `.blobs/{prefix}/{hash}.blob`.
|
|
||||||
fn blob_path(&self, hash: &str) -> PathBuf {
|
|
||||||
let prefix = if hash.len() >= 2 { &hash[0..2] } else { hash };
|
|
||||||
self.blob_root.join(prefix).join(format!("{hash}.blob"))
|
|
||||||
}
|
|
||||||
|
|
||||||
/// Spawn a background indexing task. `reuse_dedup` clones faces from an
|
/// Spawn a background indexing task. `reuse_dedup` clones faces from an
|
||||||
/// existing file with the same blob hash instead of re-running inference;
|
/// existing file with the same blob hash instead of re-running inference;
|
||||||
/// `delete_first` clears prior faces (used on overwrite).
|
/// `delete_first` clears prior faces (used on overwrite).
|
||||||
@@ -84,7 +87,7 @@ impl FaceIndexingService {
|
|||||||
let pool = self.pool.clone();
|
let pool = self.pool.clone();
|
||||||
let repo = self.repo.clone();
|
let repo = self.repo.clone();
|
||||||
let analyzer = self.analyzer.clone();
|
let analyzer = self.analyzer.clone();
|
||||||
let blob_path = self.blob_path(&blob_hash);
|
let dedup = self.dedup.clone();
|
||||||
let semaphore = self.index_semaphore.clone();
|
let semaphore = self.index_semaphore.clone();
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
// Queue behind the concurrency budget BEFORE touching the
|
// Queue behind the concurrency budget BEFORE touching the
|
||||||
@@ -102,7 +105,7 @@ impl FaceIndexingService {
|
|||||||
&repo,
|
&repo,
|
||||||
analyzer.as_ref(),
|
analyzer.as_ref(),
|
||||||
file_id,
|
file_id,
|
||||||
&blob_path,
|
&dedup,
|
||||||
&blob_hash,
|
&blob_hash,
|
||||||
reuse_dedup,
|
reuse_dedup,
|
||||||
)
|
)
|
||||||
@@ -161,7 +164,12 @@ impl FileLifecycleHook for FaceIndexingService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
async fn lookup_user(pool: &PgPool, file_id: Uuid) -> Result<Uuid, DomainError> {
|
async fn lookup_user(pool: &PgPool, file_id: Uuid) -> Result<Uuid, DomainError> {
|
||||||
let row: (Uuid,) = sqlx::query_as("SELECT user_id FROM storage.files WHERE id = $1")
|
// Post-D7: `storage.files.user_id` was dropped in
|
||||||
|
// migrations/20260904000000_drop_files_folders_user_id.sql —
|
||||||
|
// provenance moved to `created_by` / `updated_by`. For the
|
||||||
|
// faces.user_id anchor, the file's original creator is what we
|
||||||
|
// want (matches the pre-D7 semantic of the dropped column).
|
||||||
|
let row: (Uuid,) = sqlx::query_as("SELECT created_by FROM storage.files WHERE id = $1")
|
||||||
.bind(file_id)
|
.bind(file_id)
|
||||||
.fetch_one(pool)
|
.fetch_one(pool)
|
||||||
.await
|
.await
|
||||||
@@ -174,7 +182,7 @@ async fn index_file(
|
|||||||
repo: &FacePgRepository,
|
repo: &FacePgRepository,
|
||||||
analyzer: &dyn FaceAnalyzerPort,
|
analyzer: &dyn FaceAnalyzerPort,
|
||||||
file_id: Uuid,
|
file_id: Uuid,
|
||||||
blob_path: &Path,
|
dedup: &Arc<DedupService>,
|
||||||
blob_hash: &str,
|
blob_hash: &str,
|
||||||
reuse_dedup: bool,
|
reuse_dedup: bool,
|
||||||
) -> Result<(), DomainError> {
|
) -> Result<(), DomainError> {
|
||||||
@@ -199,9 +207,12 @@ async fn index_file(
|
|||||||
// No peer found — fall through and analyze.
|
// No peer found — fall through and analyze.
|
||||||
}
|
}
|
||||||
|
|
||||||
let bytes = tokio::fs::read(blob_path)
|
// CDC-aware, backend-agnostic read: `DedupService` concatenates chunks
|
||||||
.await
|
// for CDC files, delegates straight through for legacy whole-file
|
||||||
.map_err(|e| DomainError::internal_error("Faces", format!("read blob: {e}")))?;
|
// blobs, and inherits the backend wrapper stack (encryption, retry,
|
||||||
|
// cache) transparently. Peak process-heap = image size, already
|
||||||
|
// bounded by `index_semaphore` above.
|
||||||
|
let bytes = dedup.read_blob_bytes(blob_hash).await?;
|
||||||
let detected = analyzer.analyze(&bytes).await?;
|
let detected = analyzer.analyze(&bytes).await?;
|
||||||
|
|
||||||
let faces: Vec<Face> = detected
|
let faces: Vec<Face> = detected
|
||||||
|
|||||||
@@ -32,6 +32,7 @@ use uuid::Uuid;
|
|||||||
use crate::application::ports::file_lifecycle::FileLifecycleHook;
|
use crate::application::ports::file_lifecycle::FileLifecycleHook;
|
||||||
use crate::common::errors::DomainError;
|
use crate::common::errors::DomainError;
|
||||||
use crate::infrastructure::repositories::pg::file_metadata_repository::FileMetadataRepository;
|
use crate::infrastructure::repositories::pg::file_metadata_repository::FileMetadataRepository;
|
||||||
|
use crate::infrastructure::services::dedup_service::DedupService;
|
||||||
use crate::infrastructure::services::exif_service::{ExifMetadata, ExifService};
|
use crate::infrastructure::services::exif_service::{ExifMetadata, ExifService};
|
||||||
|
|
||||||
#[derive(Debug, FromRow)]
|
#[derive(Debug, FromRow)]
|
||||||
@@ -50,12 +51,22 @@ pub struct MetadataExtractionResult {
|
|||||||
|
|
||||||
pub struct MediaMetadataService {
|
pub struct MediaMetadataService {
|
||||||
pool: Arc<PgPool>,
|
pool: Arc<PgPool>,
|
||||||
blob_root: PathBuf,
|
/// CDC-aware blob reader. Same abstraction `thumbnail_service` uses —
|
||||||
|
/// hides both the chunk-manifest concatenation and the underlying
|
||||||
|
/// `BlobStorageBackend` wrapper stack.
|
||||||
|
dedup: Arc<DedupService>,
|
||||||
|
/// Tier-1 scratch directory for `stream_blob_to_tempfile`. Pulled
|
||||||
|
/// from `AppConfig::temp_dir` (env `OXICLOUD_TEMP_DIR`) at DI time.
|
||||||
|
temp_dir: PathBuf,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl MediaMetadataService {
|
impl MediaMetadataService {
|
||||||
pub fn new(pool: Arc<PgPool>, blob_root: PathBuf) -> Self {
|
pub fn new(pool: Arc<PgPool>, dedup: Arc<DedupService>, temp_dir: PathBuf) -> Self {
|
||||||
Self { pool, blob_root }
|
Self {
|
||||||
|
pool,
|
||||||
|
dedup,
|
||||||
|
temp_dir,
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn is_image_file(mime_type: &str) -> bool {
|
pub fn is_image_file(mime_type: &str) -> bool {
|
||||||
@@ -71,15 +82,11 @@ impl MediaMetadataService {
|
|||||||
Self::is_image_file(mime_type) || Self::is_video_file(mime_type)
|
Self::is_image_file(mime_type) || Self::is_video_file(mime_type)
|
||||||
}
|
}
|
||||||
|
|
||||||
fn blob_path(&self, hash: &str) -> PathBuf {
|
|
||||||
let prefix = &hash[0..2];
|
|
||||||
self.blob_root.join(prefix).join(format!("{}.blob", hash))
|
|
||||||
}
|
|
||||||
|
|
||||||
fn arc(&self) -> Arc<Self> {
|
fn arc(&self) -> Arc<Self> {
|
||||||
Arc::new(Self {
|
Arc::new(Self {
|
||||||
pool: self.pool.clone(),
|
pool: self.pool.clone(),
|
||||||
blob_root: self.blob_root.clone(),
|
dedup: self.dedup.clone(),
|
||||||
|
temp_dir: self.temp_dir.clone(),
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -130,13 +137,32 @@ impl MediaMetadataService {
|
|||||||
|
|
||||||
/// Extract metadata for one file and persist it (no-op when nothing useful
|
/// Extract metadata for one file and persist it (no-op when nothing useful
|
||||||
/// could be extracted).
|
/// could be extracted).
|
||||||
|
///
|
||||||
|
/// Streams the blob (CDC-aware — chunks concatenated on the fly for
|
||||||
|
/// chunked files; wrapper stack handles encryption + retry + cache)
|
||||||
|
/// to a tempfile in the configured tier-1 temp dir, then hands its
|
||||||
|
/// `.path()` to the sync extractors (kamadak-exif via
|
||||||
|
/// `std::fs::read(path)` for images, `nom-exif` video track reader
|
||||||
|
/// for videos — both are path-based). Peak process-heap = one chunk
|
||||||
|
/// (~1 MiB) regardless of media size.
|
||||||
pub async fn extract_and_save(
|
pub async fn extract_and_save(
|
||||||
&self,
|
&self,
|
||||||
file_id: &Uuid,
|
file_id: &Uuid,
|
||||||
file_path: &Path,
|
blob_hash: &str,
|
||||||
mime_type: &str,
|
mime_type: &str,
|
||||||
) -> Result<(), DomainError> {
|
) -> Result<(), DomainError> {
|
||||||
let path = file_path.to_path_buf();
|
// File-extension hint for the tempfile suffix — helps
|
||||||
|
// `nom-exif`'s content sniffer land the right parser branch.
|
||||||
|
let suffix = if Self::is_video_file(mime_type) {
|
||||||
|
".mp4"
|
||||||
|
} else {
|
||||||
|
".jpg"
|
||||||
|
};
|
||||||
|
let named = self
|
||||||
|
.dedup
|
||||||
|
.stream_blob_to_tempfile(blob_hash, &self.temp_dir, suffix)
|
||||||
|
.await?;
|
||||||
|
let path = named.path().to_path_buf();
|
||||||
let mime = mime_type.to_string();
|
let mime = mime_type.to_string();
|
||||||
let meta = tokio::task::spawn_blocking(move || Self::extract_blocking(&path, &mime))
|
let meta = tokio::task::spawn_blocking(move || Self::extract_blocking(&path, &mime))
|
||||||
.await
|
.await
|
||||||
@@ -146,6 +172,9 @@ impl MediaMetadataService {
|
|||||||
format!("spawn_blocking join error: {e}"),
|
format!("spawn_blocking join error: {e}"),
|
||||||
)
|
)
|
||||||
})?;
|
})?;
|
||||||
|
// Keep the tempfile alive until after extraction — the
|
||||||
|
// spawn_blocking closure only borrowed the raw path.
|
||||||
|
drop(named);
|
||||||
|
|
||||||
let Some(meta) = meta else {
|
let Some(meta) = meta else {
|
||||||
return Ok(());
|
return Ok(());
|
||||||
@@ -175,13 +204,13 @@ impl MediaMetadataService {
|
|||||||
pub fn spawn_extraction_background(
|
pub fn spawn_extraction_background(
|
||||||
service: Arc<Self>,
|
service: Arc<Self>,
|
||||||
file_id: Uuid,
|
file_id: Uuid,
|
||||||
file_path: PathBuf,
|
blob_hash: String,
|
||||||
mime_type: String,
|
mime_type: String,
|
||||||
) {
|
) {
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
tracing::info!("📷 Extracting capture metadata for: {}", file_id);
|
tracing::info!("📷 Extracting capture metadata for: {}", file_id);
|
||||||
if let Err(e) = service
|
if let Err(e) = service
|
||||||
.extract_and_save(&file_id, &file_path, &mime_type)
|
.extract_and_save(&file_id, &blob_hash, &mime_type)
|
||||||
.await
|
.await
|
||||||
{
|
{
|
||||||
tracing::warn!("Failed to extract capture metadata: {}", e);
|
tracing::warn!("Failed to extract capture metadata: {}", e);
|
||||||
@@ -192,14 +221,14 @@ impl MediaMetadataService {
|
|||||||
pub fn spawn_extraction_with_delete_background(
|
pub fn spawn_extraction_with_delete_background(
|
||||||
service: Arc<Self>,
|
service: Arc<Self>,
|
||||||
file_id: Uuid,
|
file_id: Uuid,
|
||||||
file_path: PathBuf,
|
blob_hash: String,
|
||||||
mime_type: String,
|
mime_type: String,
|
||||||
) {
|
) {
|
||||||
tokio::spawn(async move {
|
tokio::spawn(async move {
|
||||||
tracing::info!("📷 Updating capture metadata for: {}", file_id);
|
tracing::info!("📷 Updating capture metadata for: {}", file_id);
|
||||||
let _ = service.delete_metadata(&file_id).await;
|
let _ = service.delete_metadata(&file_id).await;
|
||||||
if let Err(e) = service
|
if let Err(e) = service
|
||||||
.extract_and_save(&file_id, &file_path, &mime_type)
|
.extract_and_save(&file_id, &blob_hash, &mime_type)
|
||||||
.await
|
.await
|
||||||
{
|
{
|
||||||
tracing::warn!("Failed to update capture metadata: {}", e);
|
tracing::warn!("Failed to update capture metadata: {}", e);
|
||||||
@@ -287,9 +316,8 @@ impl MediaMetadataService {
|
|||||||
info!("Cloned capture metadata for file {}", new_file_id);
|
info!("Cloned capture metadata for file {}", new_file_id);
|
||||||
}
|
}
|
||||||
Ok(_) => {
|
Ok(_) => {
|
||||||
let file_path = service.blob_path(&blob_hash);
|
|
||||||
if let Err(e) = service
|
if let Err(e) = service
|
||||||
.extract_and_save(&new_file_id, &file_path, &mime_type)
|
.extract_and_save(&new_file_id, &blob_hash, &mime_type)
|
||||||
.await
|
.await
|
||||||
{
|
{
|
||||||
warn!(
|
warn!(
|
||||||
@@ -335,9 +363,8 @@ impl MediaMetadataService {
|
|||||||
let media = row.map_err(|e| {
|
let media = row.map_err(|e| {
|
||||||
DomainError::database_error(format!("Failed to fetch media file row: {}", e))
|
DomainError::database_error(format!("Failed to fetch media file row: {}", e))
|
||||||
})?;
|
})?;
|
||||||
let file_path = self.blob_path(&media.blob_hash);
|
|
||||||
match self
|
match self
|
||||||
.extract_and_save(&media.file_id, &file_path, &media.mime_type)
|
.extract_and_save(&media.file_id, &media.blob_hash, &media.mime_type)
|
||||||
.await
|
.await
|
||||||
{
|
{
|
||||||
Ok(()) => processed += 1,
|
Ok(()) => processed += 1,
|
||||||
@@ -530,7 +557,7 @@ impl FileLifecycleHook for MediaMetadataService {
|
|||||||
Self::spawn_extraction_background(
|
Self::spawn_extraction_background(
|
||||||
service,
|
service,
|
||||||
uuid,
|
uuid,
|
||||||
self.blob_path(blob_hash),
|
blob_hash.to_string(),
|
||||||
content_type.to_string(),
|
content_type.to_string(),
|
||||||
);
|
);
|
||||||
} else {
|
} else {
|
||||||
@@ -584,7 +611,7 @@ impl FileLifecycleHook for MediaMetadataService {
|
|||||||
Self::spawn_extraction_with_delete_background(
|
Self::spawn_extraction_with_delete_background(
|
||||||
self.arc(),
|
self.arc(),
|
||||||
uuid,
|
uuid,
|
||||||
self.blob_path(blob_hash),
|
blob_hash.to_string(),
|
||||||
content_type.to_string(),
|
content_type.to_string(),
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user