perf: keyset/LATERAL SQL shapes, auth+blob-cache single-flight, spool buffers, DTO interning
Round 3 of benchmark-gated optimizations (benches/ROUND3.md; every change gated by a before/after benchmark — an AFTER that did not beat its BEFORE was to be rolled back; none needed it. Equivalence gates assert identical row sequences / byte-identical output on every behavior-preserving rewrite): DB hot paths (local PG16, EXPLAIN-verified): - Web-UI listing (list_resources_paged): cursor pushed INSIDE the folders/files UNION-ALL branches as sargable row-value comparisons with per-branch ORDER/LIMIT + two partial expression indexes (folder_id, LOWER(name), id). 20k-entry folder: 26.6 -> 1.3 ms/page (19.5x); other sort modes at parity or better. New migration 20260918000000. [benches/LISTING-KEYSET.md section in ROUND3] - Photos timeline (list_media_files): per-drive CROSS JOIN LATERAL top-N on the timeline index, joins moved above the top-N. 50k-photo library: 97.4 -> 1.6 ms/page (55.7x). The old "LIMIT stops the scan early" comment was refuted by EXPLAIN. - PROPFIND sub-folders (both DAV surfaces): keyset list_folders_batch off idx_folders_unique_name replaces COUNT(*) OVER() + LIMIT/OFFSET (5k dirs: 79.7 -> 17.9 ms full walk, 4.5x). Concurrency: - Basic-auth cache single-flight (moka try_get_with): 8 concurrent DAV connections at TTL expiry paid 8 Argon2id runs (2.6 s CPU + 8x64 MiB); now 1 (300 ms). Failed verifications remain uncached. - CachedBlobBackend per-hash single-flight + unique tmp names: 16 concurrent cold readers = 16 full remote downloads racing truncating writes on ONE deterministic .tmp (corruptible cache); now 1 download (16x less egress, 2.8x wall on a shared link) and torn files can never be renamed into the cache. I/O and allocations: - Chunk-assembly reads 64K -> 512K buffers (2.3x, 8x fewer syscalls); chunk-spool writes via BufWriter 512K (5.6x, 32x fewer syscalls). - S3/Azure put_blob_from_bytes_unsynced overrides: dedup settle no longer pays a HEAD probe per new chunk (2 RTT -> 1, 1.8x); Azure stops copying every chunk (Bytes -> Body, -0.44 ms - 4 MiB alloc per 4 MiB chunk). - Entity->DTO mapping: Arc<str> interning of closed-set display fields + common MIMEs, 1-alloc etag/size formatting, FolderDto moves instead of clones. File row: 11 -> 4 allocs; folder row: 11.8 -> 1 (2.1x faster). - CardDAV REPORT: deleted dead per-contact vCard pre-generation and the O(N^2) uid scan whose result was discarded (5k contacts: 55.7 -> 5.7 ms, 9.8x); byte-identical XML asserted. - Search-results cache: byte weigher + 32 MiB budget (OXICLOUD_SEARCH_CACHE_MAX_BYTES) replaces the 1000-ENTRY cap that let ~300 MiB of enriched rows sit in RSS; read latency parity. - Dropped aws-config + aws-smithy-types (zero references; -82 dep-graph nodes, three SDK stacks gone from every build). tokio "process" is now an explicit feature (was enabled transitively by aws-config). Frontend: - Cached Intl.DateTimeFormat keyed by (locale, options) in formatDate and 4 sibling callsites: 20k dates 2612 -> 51 ms (51.6x); vitest gate asserts output identity across locales and a 3x floor. Validation: cargo fmt + clippy --all-features --all-targets -D warnings clean; 518 unit + 548 integration-cfg tests green; new-shape endpoints smoke-tested end-to-end over HTTP (all 5 listing sort modes with cursor walks, WebDAV PROPFIND Depth-1, photos timeline, Basic-auth DAV login); frontend npm run check clean, new vitest gates green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EBsU2qEzny3A8WQUEuMNCr
This commit is contained in:
Claude
@@ -488,13 +488,21 @@ impl FileBlobReadRepository {
|
||||
/// `sort_date` epoch for each file (used as pagination cursor).
|
||||
///
|
||||
/// Uses the denormalised `media_sort_date` column (synced from
|
||||
/// `file_metadata.captured_at` by trigger) so no JOIN with
|
||||
/// `file_metadata` is needed. The partial covering index
|
||||
/// `idx_files_media_timeline_by_drive` (migration 20260901000001)
|
||||
/// keys on `(drive_id, media_sort_date DESC)` filtered on non-trashed
|
||||
/// image/video rows — Postgres does one IndexScan per in-scope
|
||||
/// drive_id already ordered by capture date, so LIMIT stops the scan
|
||||
/// early. Same O(LIMIT) shape as the pre-D7 `user_id`-keyed hot path.
|
||||
/// `file_metadata.captured_at` by trigger). The accessible drive ids
|
||||
/// are materialised once, then a `CROSS JOIN LATERAL (… ORDER BY
|
||||
/// media_sort_date DESC LIMIT k)` per drive turns the partial covering
|
||||
/// index `idx_files_media_timeline_by_drive` (migration 20260901000001,
|
||||
/// `(drive_id, media_sort_date DESC)` filtered on non-trashed
|
||||
/// image/video rows) into one BOUNDED index scan per drive; the outer
|
||||
/// merge sorts `drives × k` rows. The folders / file_metadata joins sit
|
||||
/// outside the top-N so only the k emitted rows pay them.
|
||||
///
|
||||
/// The previous shape put the joins and the global `ORDER BY … LIMIT`
|
||||
/// above a `drive_id IN (…)` nested loop — Postgres fed EVERY media row
|
||||
/// through the join into a top-N heapsort, scanning the timeline index
|
||||
/// to exhaustion on every page: O(library) per page, 97 ms on a
|
||||
/// 50k-photo library vs 1.6 ms for this shape (55.7x,
|
||||
/// benches/PHOTOS-TIMELINE.md).
|
||||
///
|
||||
/// Scope (`docs/plan/drive.md` §15): drives with
|
||||
/// `policies.include_in_photo_index = true` where the caller has a
|
||||
@@ -537,37 +545,47 @@ impl FileBlobReadRepository {
|
||||
};
|
||||
let sql = format!(
|
||||
r#"
|
||||
SELECT fi.id::text, fi.name, fi.folder_id::text, fo.path,
|
||||
fi.size, fi.mime_type,
|
||||
EXTRACT(EPOCH FROM fi.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM fi.updated_at)::bigint,
|
||||
fi.blob_hash,
|
||||
|
||||
fi.created_by, fi.updated_by,
|
||||
EXTRACT(EPOCH FROM fi.media_sort_date)::bigint AS sort_date,
|
||||
WITH accessible AS MATERIALIZED (
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
ON g.resource_type = 'drive'
|
||||
AND g.resource_id = d.id
|
||||
WHERE (
|
||||
(g.subject_type = 'user' AND g.subject_id = $1)
|
||||
OR (g.subject_type = 'group' AND g.subject_id IN
|
||||
(SELECT storage.caller_group_ids($1)))
|
||||
)
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
)
|
||||
SELECT top.id::text, top.name, top.folder_id::text, fo.path,
|
||||
top.size, top.mime_type,
|
||||
EXTRACT(EPOCH FROM top.created_at)::bigint,
|
||||
EXTRACT(EPOCH FROM top.updated_at)::bigint,
|
||||
top.blob_hash,
|
||||
top.created_by, top.updated_by,
|
||||
EXTRACT(EPOCH FROM top.media_sort_date)::bigint AS sort_date,
|
||||
fm.width, fm.height
|
||||
FROM storage.files fi
|
||||
LEFT JOIN storage.folders fo ON fo.id = fi.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = fi.id
|
||||
WHERE fi.drive_id IN (
|
||||
SELECT d.id
|
||||
FROM storage.drives d
|
||||
JOIN storage.role_grants g
|
||||
ON g.resource_type = 'drive'
|
||||
AND g.resource_id = d.id
|
||||
WHERE (
|
||||
(g.subject_type = 'user' AND g.subject_id = $1)
|
||||
OR (g.subject_type = 'group' AND g.subject_id IN
|
||||
(SELECT storage.caller_group_ids($1)))
|
||||
)
|
||||
AND (g.expires_at IS NULL OR g.expires_at > NOW())
|
||||
AND (d.policies->>'include_in_photo_index')::boolean = true
|
||||
)
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
FROM (
|
||||
SELECT fi.*
|
||||
FROM accessible a
|
||||
CROSS JOIN LATERAL (
|
||||
SELECT fi.*
|
||||
FROM storage.files fi
|
||||
WHERE fi.drive_id = a.id
|
||||
AND NOT fi.is_trashed
|
||||
AND (fi.mime_type LIKE 'image/%' OR fi.mime_type LIKE 'video/%')
|
||||
{cursor_pred}
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) fi
|
||||
ORDER BY fi.media_sort_date DESC
|
||||
LIMIT $3
|
||||
) top
|
||||
LEFT JOIN storage.folders fo ON fo.id = top.folder_id
|
||||
LEFT JOIN storage.file_metadata fm ON fm.file_id = top.id
|
||||
ORDER BY top.media_sort_date DESC
|
||||
"#,
|
||||
);
|
||||
let rows: Vec<MediaFileRow> = sqlx::query_as(&sql)
|
||||
|
||||
@@ -501,6 +501,61 @@ impl FolderRepository for FolderDbRepository {
|
||||
Ok((folders?, total))
|
||||
}
|
||||
|
||||
/// Keyset sub-folder page: `name > $after ORDER BY name LIMIT $limit`,
|
||||
/// one bounded index-range read off `idx_folders_unique_name` — the
|
||||
/// cursor predicate is only emitted when a cursor exists (a bound
|
||||
/// disjunction would block the index condition under generic plans,
|
||||
/// same rule as `list_files_batch`). Root scope (`parent_id = None`)
|
||||
/// keeps the trait's in-memory default: roots are one-per-drive, a
|
||||
/// handful of rows.
|
||||
async fn list_folders_batch(
|
||||
&self,
|
||||
parent_id: Option<&str>,
|
||||
after_name: Option<&str>,
|
||||
limit: usize,
|
||||
) -> Result<Vec<Folder>, DomainError> {
|
||||
let Some(pid) = parent_id else {
|
||||
let mut all = self.list_folders(None).await?;
|
||||
all.sort_by(|a, b| a.name().cmp(b.name()));
|
||||
return Ok(all
|
||||
.into_iter()
|
||||
.filter(|f| after_name.is_none_or(|a| f.name() > a))
|
||||
.take(limit)
|
||||
.collect());
|
||||
};
|
||||
|
||||
let cursor_pred = if after_name.is_some() {
|
||||
"AND name > $3"
|
||||
} else {
|
||||
"AND $3::text IS NULL"
|
||||
};
|
||||
let sql = format!(
|
||||
"SELECT id::text, name, path, parent_id::text, drive_id, \
|
||||
EXTRACT(EPOCH FROM created_at)::bigint, \
|
||||
EXTRACT(EPOCH FROM updated_at)::bigint, \
|
||||
EXTRACT(EPOCH FROM tree_modified_at)::bigint, \
|
||||
created_by, updated_by \
|
||||
FROM storage.folders \
|
||||
WHERE parent_id = $1::uuid AND NOT is_trashed \
|
||||
{cursor_pred} \
|
||||
ORDER BY name \
|
||||
LIMIT $2"
|
||||
);
|
||||
let rows: Vec<FolderRow> = sqlx::query_as(&sql)
|
||||
.bind(pid)
|
||||
.bind(limit as i64)
|
||||
.bind(after_name)
|
||||
.fetch_all(self.pool())
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("FolderDb", format!("batch: {e}")))?;
|
||||
|
||||
rows.into_iter()
|
||||
.map(|(id, name, path, pid, did, ca, ma, tma, cb, ub)| {
|
||||
Self::row_to_folder(id, name, path, pid, did, ca, ma, tma, cb, ub)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Paginated companion to `list_root_folders_for_caller` — same
|
||||
/// drive-membership predicate, adds LIMIT/OFFSET and an optional
|
||||
/// window-function COUNT so total pages can be surfaced without a
|
||||
@@ -1391,13 +1446,6 @@ impl FolderDbRepository {
|
||||
WHERE fm.folder_id = $1::uuid AND NOT fm.is_trashed
|
||||
"#;
|
||||
|
||||
let cte_inner = match (include_folders, include_files) {
|
||||
(true, true) => format!("{folder_branch} UNION ALL {file_branch}"),
|
||||
(true, false) => folder_branch.to_owned(),
|
||||
(false, true) => file_branch.to_owned(),
|
||||
(false, false) => unreachable!(),
|
||||
};
|
||||
|
||||
// ── Cursor binds ─────────────────────────────────────────────────────
|
||||
// $1 = parent_id $2 = cursor_str $3 = cursor_int
|
||||
// $4 = cursor_ts $5 = cursor_id $6 = limit
|
||||
@@ -1406,112 +1454,184 @@ impl FolderDbRepository {
|
||||
let cursor_ts = cursor.and_then(|c| c.sort_ts);
|
||||
let cursor_id = cursor.map(|c| c.resource_id);
|
||||
|
||||
// ── Sort-specific WHERE + ORDER BY ───────────────────────────────────
|
||||
// Each arm produces two variants based on `reverse`.
|
||||
// For "name": folder_first stays ASC in both directions (folders always
|
||||
// precede files); only the alpha order within each group flips.
|
||||
let (where_clause, order_clause) = match order_by {
|
||||
// ── Per-branch cursor pushdown ───────────────────────────────────────
|
||||
// The cursor is applied INSIDE each UNION-ALL branch as a sargable
|
||||
// row-value comparison on base columns — not on the CTE's computed
|
||||
// columns — and every branch pre-sorts and pre-limits, so Postgres
|
||||
// reads O(limit) rows per branch instead of rescanning and
|
||||
// top-N-sorting the entire folder on every page (19.5x on a
|
||||
// 20k-entry folder, benches/LISTING-KEYSET.md). The "name" sort is
|
||||
// served by the expression indexes idx_files_folder_lname /
|
||||
// idx_folders_parent_lname (migration 20260918000000).
|
||||
//
|
||||
// Sort-key columns that are CONSTANT within a branch (folder_first,
|
||||
// the folder branch's type_order = 0 and size = -1) are folded in
|
||||
// Rust: depending on which group the cursor points into, the branch
|
||||
// predicate shortens to a row-value over the remaining keys, the
|
||||
// branch keeps all its rows, or the branch drops out entirely.
|
||||
enum BranchCursor {
|
||||
/// The cursor has moved past every row this branch can produce.
|
||||
Drop,
|
||||
/// Every row in this branch sorts after the cursor.
|
||||
All,
|
||||
/// Row-value comparison over the branch's non-constant sort keys.
|
||||
Pred(String),
|
||||
}
|
||||
use BranchCursor::{All, Drop, Pred};
|
||||
|
||||
let has_cursor = cursor.is_some();
|
||||
// (folder-branch cursor, file-branch cursor, per-branch ORDER BY on
|
||||
// the branch's output aliases, outer merge ORDER BY)
|
||||
let (folder_cur, file_cur, branch_order, outer_order) = match order_by {
|
||||
"type" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (type_order < $3)
|
||||
OR (type_order = $3 AND sort_str < $2)
|
||||
OR (type_order = $3 AND sort_str = $2 AND id < $5::uuid)"#,
|
||||
"ORDER BY type_order DESC, sort_str DESC, id DESC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
("<", "ORDER BY type_order DESC, sort_str DESC, id DESC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (type_order > $3)
|
||||
OR (type_order = $3 AND sort_str > $2)
|
||||
OR (type_order = $3 AND sort_str = $2 AND id > $5::uuid)"#,
|
||||
"ORDER BY type_order ASC, sort_str ASC, id ASC",
|
||||
)
|
||||
}
|
||||
(">", "ORDER BY type_order ASC, sort_str ASC, id ASC")
|
||||
};
|
||||
let folder_cur = match cursor_int {
|
||||
None => All,
|
||||
// Folder rows have type_order = 0; a cursor sitting on a
|
||||
// file (type_order > 0) either exhausts the folder group
|
||||
// (ASC) or precedes all of it (DESC).
|
||||
Some(c_to) if c_to > 0 => {
|
||||
if reverse {
|
||||
All
|
||||
} else {
|
||||
Drop
|
||||
}
|
||||
}
|
||||
Some(_) => Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
|
||||
};
|
||||
let file_cur = if has_cursor {
|
||||
Pred(format!(
|
||||
"(fm.category_order::bigint, LOWER(fm.name), fm.id) {op} ($3, $2, $5::uuid)"
|
||||
))
|
||||
} else {
|
||||
All
|
||||
};
|
||||
(folder_cur, file_cur, ord, ord)
|
||||
}
|
||||
"modified_at" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (modified_at > $4)
|
||||
OR (modified_at = $4 AND id > $5::uuid)"#,
|
||||
"ORDER BY modified_at ASC, id ASC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
(">", "ORDER BY modified_at ASC, id ASC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (modified_at < $4)
|
||||
OR (modified_at = $4 AND id < $5::uuid)"#,
|
||||
"ORDER BY modified_at DESC, id DESC",
|
||||
)
|
||||
}
|
||||
("<", "ORDER BY modified_at DESC, id DESC")
|
||||
};
|
||||
let mk = |col: &str| {
|
||||
if has_cursor {
|
||||
Pred(format!("({col}.updated_at, {col}.id) {op} ($4, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
}
|
||||
};
|
||||
(mk("f"), mk("fm"), ord, ord)
|
||||
}
|
||||
"created_at" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (created_at > $4)
|
||||
OR (created_at = $4 AND id > $5::uuid)"#,
|
||||
"ORDER BY created_at ASC, id ASC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
(">", "ORDER BY created_at ASC, id ASC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($4::timestamptz IS NULL)
|
||||
OR (created_at < $4)
|
||||
OR (created_at = $4 AND id < $5::uuid)"#,
|
||||
"ORDER BY created_at DESC, id DESC",
|
||||
)
|
||||
}
|
||||
("<", "ORDER BY created_at DESC, id DESC")
|
||||
};
|
||||
let mk = |col: &str| {
|
||||
if has_cursor {
|
||||
Pred(format!("({col}.created_at, {col}.id) {op} ($4, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
}
|
||||
};
|
||||
(mk("f"), mk("fm"), ord, ord)
|
||||
}
|
||||
"size" => {
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (size < $3)
|
||||
OR (size = $3 AND id < $5::uuid)"#,
|
||||
"ORDER BY size DESC, id DESC",
|
||||
)
|
||||
let (op, ord) = if reverse {
|
||||
("<", "ORDER BY size DESC, id DESC")
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (size > $3)
|
||||
OR (size = $3 AND id > $5::uuid)"#,
|
||||
"ORDER BY size ASC, id ASC",
|
||||
)
|
||||
}
|
||||
(">", "ORDER BY size ASC, id ASC")
|
||||
};
|
||||
let folder_cur = match cursor_int {
|
||||
None => All,
|
||||
// Folder rows have size = -1; a cursor sitting on a file
|
||||
// (size >= 0) exhausts the folder group (ASC) or precedes
|
||||
// all of it (DESC).
|
||||
Some(c_sz) if c_sz > -1 => {
|
||||
if reverse {
|
||||
All
|
||||
} else {
|
||||
Drop
|
||||
}
|
||||
}
|
||||
Some(_) => Pred(format!("f.id {op} $5::uuid")),
|
||||
};
|
||||
let file_cur = if has_cursor {
|
||||
Pred(format!("(fm.size::bigint, fm.id) {op} ($3, $5::uuid)"))
|
||||
} else {
|
||||
All
|
||||
};
|
||||
(folder_cur, file_cur, ord, ord)
|
||||
}
|
||||
_ => {
|
||||
// "name" (default): folder_first stays ASC so folders always precede
|
||||
// files; only the alpha order within each group flips when reversed.
|
||||
if reverse {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (folder_first::bigint > $3)
|
||||
OR (folder_first::bigint = $3 AND sort_str < $2)
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id < $5::uuid)"#,
|
||||
"ORDER BY folder_first ASC, sort_str DESC, id DESC",
|
||||
)
|
||||
// "name" (default): folder_first stays ASC so folders always
|
||||
// precede files; only the alpha order within each group flips
|
||||
// when reversed. cursor_int carries folder_first (0|1).
|
||||
let op = if reverse { "<" } else { ">" };
|
||||
let branch_ord = if reverse {
|
||||
"ORDER BY sort_str DESC, id DESC"
|
||||
} else {
|
||||
(
|
||||
r#"WHERE ($3::bigint IS NULL)
|
||||
OR (folder_first::bigint > $3)
|
||||
OR (folder_first::bigint = $3 AND sort_str > $2)
|
||||
OR (folder_first::bigint = $3 AND sort_str = $2 AND id > $5::uuid)"#,
|
||||
"ORDER BY folder_first ASC, sort_str ASC, id ASC",
|
||||
)
|
||||
}
|
||||
"ORDER BY sort_str ASC, id ASC"
|
||||
};
|
||||
let outer_ord = if reverse {
|
||||
"ORDER BY folder_first ASC, sort_str DESC, id DESC"
|
||||
} else {
|
||||
"ORDER BY folder_first ASC, sort_str ASC, id ASC"
|
||||
};
|
||||
let (folder_cur, file_cur) = match cursor_int {
|
||||
None => (All, All),
|
||||
// Cursor inside the folder group: folders continue after
|
||||
// the row-value cursor; every file still follows.
|
||||
Some(0) => (
|
||||
Pred(format!("(LOWER(f.name), f.id) {op} ($2, $5::uuid)")),
|
||||
All,
|
||||
),
|
||||
// Cursor inside the file group: the folder group is done.
|
||||
Some(_) => (
|
||||
Drop,
|
||||
Pred(format!("(LOWER(fm.name), fm.id) {op} ($2, $5::uuid)")),
|
||||
),
|
||||
};
|
||||
(folder_cur, file_cur, branch_ord, outer_ord)
|
||||
}
|
||||
};
|
||||
|
||||
let wrap = |branch: &str, cur: &BranchCursor| -> Option<String> {
|
||||
let extra = match cur {
|
||||
Drop => return None,
|
||||
All => String::new(),
|
||||
Pred(p) => format!(" AND {p}"),
|
||||
};
|
||||
Some(format!(
|
||||
"(SELECT * FROM ({branch}{extra}) b {branch_order} LIMIT $6)"
|
||||
))
|
||||
};
|
||||
let mut branches = Vec::with_capacity(2);
|
||||
if include_folders && let Some(b) = wrap(folder_branch, &folder_cur) {
|
||||
branches.push(b);
|
||||
}
|
||||
if include_files && let Some(b) = wrap(file_branch, &file_cur) {
|
||||
branches.push(b);
|
||||
}
|
||||
// Every requested branch dropped out (e.g. folders-only listing with
|
||||
// the cursor already past the folder group).
|
||||
if branches.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
let inner = branches.join(" UNION ALL ");
|
||||
|
||||
let sql = format!(
|
||||
"WITH resources AS ({cte_inner}) \
|
||||
SELECT resource_type, id, name, folder_id, mime_type, size, \
|
||||
"SELECT resource_type, id, name, folder_id, mime_type, size, \
|
||||
created_at, modified_at, drive_id, blob_hash, \
|
||||
sort_str, type_order, folder_first \
|
||||
FROM resources \
|
||||
{where_clause} \
|
||||
{order_clause} \
|
||||
FROM ({inner}) r \
|
||||
{outer_order} \
|
||||
LIMIT $6"
|
||||
);
|
||||
|
||||
|
||||
@@ -130,7 +130,9 @@ impl BlobStorageBackend for AzureBlobBackend {
|
||||
return Ok(size);
|
||||
}
|
||||
|
||||
client.put_block_blob(data.to_vec()).await.map_err(|e| {
|
||||
// `Bytes` converts into `azure_core::Body` by reference count —
|
||||
// the old `data.to_vec()` copied every chunk once more.
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
|
||||
@@ -138,6 +140,26 @@ impl BlobStorageBackend for AzureBlobBackend {
|
||||
})
|
||||
}
|
||||
|
||||
/// Dedup settle path: PUT unconditionally. Content-addressed keys make
|
||||
/// re-PUTs idempotent, so the `get_properties` probe
|
||||
/// `put_blob_from_bytes` pays is a pure extra round-trip on every NEW
|
||||
/// chunk (2 RTTs -> 1, benches/S3-PUT.md — same shape as S3).
|
||||
fn put_blob_from_bytes_unsynced(
|
||||
&self,
|
||||
hash: &str,
|
||||
data: Bytes,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let client = self.blob_client(&hash);
|
||||
let size = data.len() as u64;
|
||||
client.put_block_blob(data).await.map_err(|e| {
|
||||
DomainError::internal_error("Azure", format!("Failed to upload blob {hash}: {e}"))
|
||||
})?;
|
||||
Ok(size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
|
||||
@@ -13,12 +13,14 @@ use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use bytes::Bytes;
|
||||
use dashmap::DashMap;
|
||||
use lru::LruCache;
|
||||
use std::num::NonZeroUsize;
|
||||
use tokio::fs;
|
||||
use tokio::io::{AsyncReadExt, AsyncSeekExt, AsyncWriteExt};
|
||||
use tokio::sync::Mutex;
|
||||
use tokio_util::io::ReaderStream;
|
||||
use uuid::Uuid;
|
||||
|
||||
use crate::application::ports::blob_storage_ports::{
|
||||
BlobStorageBackend, BlobStream, StorageHealthStatus,
|
||||
@@ -56,6 +58,13 @@ pub struct CachedBlobBackend {
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
/// Per-hash single-flight gates for cache misses. K concurrent cold
|
||||
/// readers of one blob (e.g. a video player's parallel Range probes)
|
||||
/// used to each download the FULL blob from the remote backend — and
|
||||
/// race their writes on one shared `.tmp` path. The gate coalesces
|
||||
/// them onto one fetch; waiters re-check the cache and serve locally
|
||||
/// (16 fetches -> 1, benches/BLOB-CACHE.md).
|
||||
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
|
||||
}
|
||||
|
||||
impl CachedBlobBackend {
|
||||
@@ -70,6 +79,7 @@ impl CachedBlobBackend {
|
||||
NonZeroUsize::new(1_000_000).unwrap(),
|
||||
))),
|
||||
current_size: Arc::new(AtomicU64::new(0)),
|
||||
inflight: Arc::new(DashMap::new()),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -150,6 +160,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
inflight: self.inflight.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
// Write to inner backend
|
||||
@@ -172,6 +183,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
max_cache_bytes: self.max_cache_bytes,
|
||||
index: self.index.clone(),
|
||||
current_size: self.current_size.clone(),
|
||||
inflight: self.inflight.clone(),
|
||||
};
|
||||
Box::pin(async move {
|
||||
let size = inner.put_blob_from_bytes(&hash, data.clone()).await?;
|
||||
@@ -203,6 +215,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
let inflight = self.inflight.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache presence (and bump LRU recency) under a brief lock,
|
||||
// then release it BEFORE touching the filesystem so concurrent
|
||||
@@ -219,14 +232,17 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch from inner, spool to cache
|
||||
// Cache miss — fetch from inner (single-flight), spool to cache
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
inflight,
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let dest = self_ref
|
||||
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
|
||||
.await?;
|
||||
let file = fs::File::open(&dest).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("re-open cached: {e}"))
|
||||
})?;
|
||||
@@ -249,6 +265,7 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
let cache_dir = self.cache_dir.clone();
|
||||
let max_cache_bytes = self.max_cache_bytes;
|
||||
let current_size = self.current_size.clone();
|
||||
let inflight = self.inflight.clone();
|
||||
Box::pin(async move {
|
||||
// Check cache presence (and bump LRU recency) under a brief lock,
|
||||
// then release it BEFORE the open()/seek() syscalls so concurrent
|
||||
@@ -271,14 +288,19 @@ impl BlobStorageBackend for CachedBlobBackend {
|
||||
}
|
||||
}
|
||||
|
||||
// Cache miss — fetch full blob into cache, then serve range
|
||||
// Cache miss — fetch full blob into cache (single-flight: a
|
||||
// player's parallel cold Range probes coalesce onto ONE remote
|
||||
// download), then serve the range locally.
|
||||
let self_ref = CachedRef {
|
||||
cache_dir,
|
||||
max_cache_bytes,
|
||||
index: index.clone(),
|
||||
current_size: current_size.clone(),
|
||||
inflight,
|
||||
};
|
||||
let dest = self_ref.fetch_and_cache_static(&hash, &*inner).await?;
|
||||
let dest = self_ref
|
||||
.fetch_and_cache_singleflight(&hash, &*inner, &cached)
|
||||
.await?;
|
||||
let mut file = fs::File::open(&dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("re-open: {e}")))?;
|
||||
@@ -406,6 +428,7 @@ struct CachedRef {
|
||||
max_cache_bytes: u64,
|
||||
index: Arc<Mutex<LruCache<String, CacheEntry>>>,
|
||||
current_size: Arc<AtomicU64>,
|
||||
inflight: Arc<DashMap<String, Arc<Mutex<()>>>>,
|
||||
}
|
||||
|
||||
impl CachedRef {
|
||||
@@ -414,6 +437,37 @@ impl CachedRef {
|
||||
self.cache_dir.join(prefix).join(format!("{hash}.blob"))
|
||||
}
|
||||
|
||||
/// Single-flight wrapper around [`Self::fetch_and_cache_static`]: the
|
||||
/// first caller for a hash becomes the leader and downloads; concurrent
|
||||
/// callers queue on the per-hash gate, then re-check the cache and serve
|
||||
/// the leader's file without touching the remote backend. Errors are not
|
||||
/// cached — the gate entry is dropped, so the next caller retries.
|
||||
async fn fetch_and_cache_singleflight(
|
||||
&self,
|
||||
hash: &str,
|
||||
inner: &dyn BlobStorageBackend,
|
||||
cached: &Path,
|
||||
) -> Result<PathBuf, DomainError> {
|
||||
let gate = self
|
||||
.inflight
|
||||
.entry(hash.to_string())
|
||||
.or_insert_with(|| Arc::new(Mutex::new(())))
|
||||
.clone();
|
||||
let _guard = gate.lock().await;
|
||||
|
||||
// Re-check under the gate: if we queued behind the leader, the blob
|
||||
// is on disk now and this turns into a local open.
|
||||
if self.index.lock().await.get(hash).is_some() && fs::metadata(cached).await.is_ok() {
|
||||
return Ok(cached.to_path_buf());
|
||||
}
|
||||
|
||||
let result = self.fetch_and_cache_static(hash, inner).await;
|
||||
// Drop the gate whether we succeeded or failed; a late-arriving
|
||||
// caller after an error creates a fresh gate and retries the fetch.
|
||||
self.inflight.remove(hash);
|
||||
result
|
||||
}
|
||||
|
||||
/// Pop LRU entries until the cache is back within its byte budget,
|
||||
/// returning the on-disk paths of the evicted blobs.
|
||||
///
|
||||
@@ -486,31 +540,51 @@ impl CachedRef {
|
||||
})?;
|
||||
}
|
||||
|
||||
let tmp = dest.with_extension("tmp");
|
||||
let mut file = fs::File::create(&tmp)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("create tmp: {e}")))?;
|
||||
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
let mut total = 0u64;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
|
||||
// Unique temp name: even if two fetches for one hash ever race
|
||||
// (e.g. across processes sharing a cache dir), each writes its own
|
||||
// inode and the rename is atomic — a torn/interleaved file can
|
||||
// never land at the final path.
|
||||
let tmp = dest.with_extension(format!("{}.tmp", Uuid::new_v4()));
|
||||
let write_result: Result<u64, DomainError> = async {
|
||||
let mut file = fs::File::create(&tmp).await.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("create tmp: {e}"))
|
||||
})?;
|
||||
total += bytes.len() as u64;
|
||||
file.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
|
||||
}
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||
drop(file);
|
||||
|
||||
fs::rename(&tmp, &dest)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("rename: {e}")))?;
|
||||
use futures::StreamExt;
|
||||
let mut stream = stream;
|
||||
let mut total = 0u64;
|
||||
while let Some(chunk) = stream.next().await {
|
||||
let bytes = chunk.map_err(|e| {
|
||||
DomainError::internal_error("BlobCache", format!("stream read: {e}"))
|
||||
})?;
|
||||
total += bytes.len() as u64;
|
||||
file.write_all(&bytes)
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("write: {e}")))?;
|
||||
}
|
||||
file.flush()
|
||||
.await
|
||||
.map_err(|e| DomainError::internal_error("BlobCache", format!("flush: {e}")))?;
|
||||
Ok(total)
|
||||
}
|
||||
.await;
|
||||
let total = match write_result {
|
||||
Ok(total) => total,
|
||||
Err(e) => {
|
||||
// Unique tmp names never get overwritten by a later fetch —
|
||||
// reap the partial file instead of leaking it.
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(e);
|
||||
}
|
||||
};
|
||||
|
||||
if let Err(e) = fs::rename(&tmp, &dest).await {
|
||||
let _ = fs::remove_file(&tmp).await;
|
||||
return Err(DomainError::internal_error(
|
||||
"BlobCache",
|
||||
format!("rename: {e}"),
|
||||
));
|
||||
}
|
||||
|
||||
let to_evict = {
|
||||
let mut idx = self.index.lock().await;
|
||||
|
||||
@@ -200,6 +200,38 @@ impl BlobStorageBackend for S3BlobBackend {
|
||||
})
|
||||
}
|
||||
|
||||
/// Dedup settle path: PUT unconditionally. Keys are content-addressed
|
||||
/// (BLAKE3), so a re-PUT writes identical bytes — overwrite-safe
|
||||
/// idempotency without the HEAD probe `put_blob_from_bytes` pays. The
|
||||
/// dedup layer already filtered out chunks the database knows about,
|
||||
/// so the probe was a pure extra round-trip on every NEW chunk of
|
||||
/// every upload (2 RTTs -> 1, benches/S3-PUT.md).
|
||||
fn put_blob_from_bytes_unsynced(
|
||||
&self,
|
||||
hash: &str,
|
||||
data: Bytes,
|
||||
) -> Pin<Box<dyn std::future::Future<Output = Result<u64, DomainError>> + Send + '_>> {
|
||||
let hash = hash.to_owned();
|
||||
Box::pin(async move {
|
||||
let key = Self::object_key(&hash);
|
||||
let size = data.len() as u64;
|
||||
self.client
|
||||
.put_object()
|
||||
.bucket(&self.bucket)
|
||||
.key(&key)
|
||||
.body(ByteStream::from(data))
|
||||
.send()
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::internal_error(
|
||||
"S3",
|
||||
format!("Failed to upload blob {}: {}", hash, e),
|
||||
)
|
||||
})?;
|
||||
Ok(size)
|
||||
})
|
||||
}
|
||||
|
||||
fn get_blob_stream(
|
||||
&self,
|
||||
hash: &str,
|
||||
|
||||
Reference in New Issue
Block a user