fix(name duplicate): fixed via NFC normalisation

TL;DR:

    fix duplicate filename via:

    ```
    docker exec <container> migrate-nfc-filenames --dry-run    # preview
    docker exec <container> migrate-nfc-filenames              # execute
    ```

 == issue ==

  Last week I uploaded Capture d'écran 2026-06-03 à 20.04.24.png from the web. It synced down to Nextcloud on my Mac. Two minutes later, the Web UI was showing the file twice.

  Both rows had:
  - the same name
  - the same size
  - the same content hash

  So why two rows? Because to PostgreSQL, the names weren't the same.

  Web upload (browser → Postgres):
    "é" stored as 1 codepoint  (U+00E9)        bytes: c3 a9     ← NFC

  NiextCloud client (macOS → Postgres):
    "é" stored as 2 codepoints (e + U+0301)    bytes: 65 cc 81  ← NFD

  macOS's APFS keeps filenames in NFD (decomposed); browsers send NFC (composed). Visually é and é are identical. To WHERE name = $1 they're two different keys. Our UNIQUE index on (folder_id, name, user_id) never fired — and the row count quietly drifted every time a Mac user touched an accented
  filename.

 == The fix is two halves ==

  1. No new duplicates — every name-receiving boundary (file upload, NC PUT, rename, MOVE, path lookup) now NFC-normalizes before touching the database. The storage invariant becomes "every stored name is NFC".
  2. Clean up existing data — one-shot migrate-nfc-filenames binary walks storage.files, NFC-normalizes any non-NFC row, and resolves the collisions we've accumulated. Same-content duplicates go to trash (recoverable); different-content collisions get renamed with a .duplicate suffix.

 == use of the clean up ==

    example of use (do not forget to define env **DATABASE_URL**)

    either
        `cargo run --bin migrate-nfc-filenames -- --dry-run`
    or
        `cargo build --bin migrate-nfc-filenames`
        `./target/debug/migrate-nfc-filenames --dry-run`

    example:
```
    % ./target/debug/migrate-nfc-filenames --dry-run
    === NFC filename migration (DRY RUN — no writes) ===

    Loaded 543 non-trashed file rows

    NORMALIZE  163451b5-5e6c-404b-9b1e-f4b01a2b7269  user=42433185-4717-416d-9a15-4580fff171ec  'Capture d’écran 2026-03-20 à 14.44.50.png' → 'Capture d’écran 2026-03-20 à 14.44.50.png'
    NORMALIZE  827dddec-4dd5-48c2-a120-dec5289f7d29  user=969deca6-7935-4f12-a430-4d636b62fa3e  'Capture d’écran 2026-04-03 à 15.43.38.png' → 'Capture d’écran 2026-04-03 à 15.43.38.png'
    NORMALIZE  09559934-a620-472d-9ba8-fc3cfeb6dc6f  user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5  'Capture d’écran 2026-06-03 à 20.05.38.png' → 'Capture d’écran 2026-06-03 à 20.05.38.png'
    NORMALIZE  5ce6dbf9-0562-4758-8783-671aa9069590  user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5  'Capture d’écran 2026-06-05 à 11.07.25.png' → 'Capture d’écran 2026-06-05 à 11.07.25.png'
    DEDUP      newer=26bcf82b-99cc-45c8-9d69-dd7e5c4484ff (trash, same blob)  older=df3adc67-a778-424d-a817-b930c75f3b06  user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5  hash=0d2cc7b0ffce2850

    === Summary ===
      scanned                            : 543
      already in NFC                     : 538
      normalized in place (no collision) : 4
      dedup-trashed (same content)       : 1
      renamed to .duplicate              : 0

    DRY RUN — no rows were written. Re-run without --dry-run to apply.
```

    once valid remove --dry-run
This commit is contained in:
Edouard Vanbelle
2026-06-06 17:47:59 +02:00
parent 5a18ec4bac
commit 7db547a669
8 changed files with 433 additions and 8 deletions
Generated
+1
View File
@@ -3856,6 +3856,7 @@ dependencies = [
"tower-http",
"tracing",
"tracing-subscriber",
"unicode-normalization",
"urlencoding",
"utoipa",
"uuid",
+5
View File
@@ -44,6 +44,7 @@ mp3-duration = "0.1"
kamadak-exif = "0.6.1"
md-5 = "0.11.0"
sha2 = "0.11.0"
unicode-normalization = "0.1.24"
blake3 = { version = "1.8.4", features = ["rayon", "mmap"] }
hex = "0.4.3"
http-body-util = "0.1.3"
@@ -87,6 +88,10 @@ unexpected_cfgs = { level = "warn", check-cfg = ['cfg(integration_tests)'] }
name = "generate-openapi"
path = "src/bin/generate-openapi.rs"
[[bin]]
name = "migrate-nfc-filenames"
path = "src/bin/migrate-nfc-filenames.rs"
[build-dependencies]
oxc_allocator = "0.125.0"
oxc_parser = "0.125.0"
+7
View File
@@ -14,6 +14,7 @@ COPY static static
RUN mkdir -p src/bin && \
echo 'fn main() { println!("Dummy build for caching dependencies"); }' > src/main.rs && \
echo 'fn main() {}' > src/bin/generate-openapi.rs && \
echo 'fn main() {}' > src/bin/migrate-nfc-filenames.rs && \
cargo build --release && \
rm -rf src static-dist target/release/deps/oxicloud* target/release/build/oxicloud-*
@@ -56,6 +57,12 @@ RUN apk --no-cache upgrade && \
# Copy the compiled binary and entrypoint (--chmod avoids extra RUN chmod layers)
COPY --from=builder --chmod=755 /app/target/release/oxicloud /usr/local/bin/
# Ship the NFC filename migration binary alongside the server so
# operators can run it inside the container without a separate Rust
# toolchain — `docker exec <container> migrate-nfc-filenames --dry-run`
# to preview, drop `--dry-run` to execute. One-shot tool, safe to
# ship; it only mutates `storage.files` rows whose name ≠ NFC(name).
COPY --from=builder --chmod=755 /app/target/release/migrate-nfc-filenames /usr/local/bin/
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN sed -i 's/\r//' /usr/local/bin/entrypoint.sh && \
chmod 755 /usr/local/bin/entrypoint.sh
+326
View File
@@ -0,0 +1,326 @@
//! `migrate-nfc-filenames` — one-shot CLI to NFC-normalize
//! `storage.files.name` across an OxiCloud instance.
//!
//! Why: PostgreSQL compares bytes literally and the `UNIQUE`
//! index on `(folder_id, name, user_id) WHERE NOT is_trashed`
//! does not catch Unicode normalization differences. macOS APFS
//! stores filenames in NFD; browsers post NFC. A file uploaded
//! from the web ("café.txt", NFC) and the same name re-uploaded
//! from a NextCloud desktop client on macOS (round-tripped to
//! NFD: `e` + combining acute) lands as two distinct rows, both
//! visible in the listing, both pointing at the same blob.
//!
//! What this does:
//!
//! 1. Scans every non-trashed file row.
//! 2. For each row whose name ≠ NFC(name):
//! - If no other row in the same `(folder_id, user_id)` already
//! holds the NFC form → UPDATE the row's name to NFC.
//! - If a collision exists with **same blob_hash**: trash the
//! newer of the two (`is_trashed = true`, `trashed_at = NOW()`).
//! User can restore from the trash UI if needed.
//! - If a collision exists with **different blob_hash**: rename
//! the newer row to `{nfc_name}.duplicate`, incrementing the
//! suffix (`.duplicate-1`, `.duplicate-2`, …) until a free name
//! is found. Preserves both files; user can inspect and resolve.
//! - In both collision cases, the surviving (older) row's name
//! is also normalized to NFC.
//!
//! Run:
//! `cargo run --bin migrate-nfc-filenames -- --dry-run`
//! `cargo run --bin migrate-nfc-filenames`
//!
//! Folder rows are NOT touched in this pass — trashing a folder
//! affects descendants; that pass is deferred to a follow-up.
use chrono::{DateTime, Utc};
use sqlx::{PgPool, Row};
use std::env;
use uuid::Uuid;
use oxicloud::domain::services::path_service::normalize_storage_name;
#[derive(Debug, Clone)]
struct FileRow {
id: Uuid,
folder_id: Option<Uuid>,
user_id: Uuid,
name: String,
blob_hash: String,
created_at: DateTime<Utc>,
}
#[derive(Default)]
struct Stats {
scanned: u64,
already_nfc: u64,
normalized_in_place: u64,
deduped_same_content: u64,
renamed_duplicate: u64,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let args: Vec<String> = env::args().collect();
let dry_run = args.iter().any(|a| a == "--dry-run");
let database_url =
env::var("DATABASE_URL").expect("DATABASE_URL must be set in the environment");
let pool = PgPool::connect(&database_url).await?;
println!(
"=== NFC filename migration ({}) ===",
if dry_run {
"DRY RUN — no writes"
} else {
"EXECUTING"
}
);
println!();
let rows = load_non_trashed_files(&pool).await?;
println!("Loaded {} non-trashed file rows", rows.len());
println!();
let mut stats = Stats {
scanned: rows.len() as u64,
..Default::default()
};
for row in &rows {
let nfc_name = normalize_storage_name(&row.name);
if nfc_name == row.name {
stats.already_nfc += 1;
continue;
}
// Row is in non-NFC form. Look for a collision in the same
// (folder_id, user_id) scope, including rows that may also
// be non-NFC but happen to normalize to the same NFC value.
let collision = find_collision(&pool, row, &nfc_name).await?;
match collision {
None => {
println!(
"NORMALIZE {} user={} '{}' → '{}'",
row.id, row.user_id, row.name, nfc_name
);
if !dry_run {
sqlx::query("UPDATE storage.files SET name = $1 WHERE id = $2")
.bind(&nfc_name)
.bind(row.id)
.execute(&pool)
.await?;
}
stats.normalized_in_place += 1;
}
Some(other) => {
// Pick winner/loser by `created_at` — older wins.
let (older, newer) = if row.created_at <= other.created_at {
(row, &other)
} else {
(&other, row)
};
if older.blob_hash == newer.blob_hash {
// Same content → trash the newer; promote older's
// name to NFC if it isn't already.
println!(
"DEDUP newer={} (trash, same blob) older={} user={} hash={}",
newer.id,
older.id,
older.user_id,
&older.blob_hash[..16.min(older.blob_hash.len())]
);
if !dry_run {
sqlx::query(
"UPDATE storage.files
SET is_trashed = TRUE,
trashed_at = NOW()
WHERE id = $1",
)
.bind(newer.id)
.execute(&pool)
.await?;
normalize_survivor_name(&pool, older, &nfc_name).await?;
}
stats.deduped_same_content += 1;
} else {
// Different content → rename newer to a free
// `{nfc_name}.duplicate[-N]`; promote older to NFC.
let disambiguated = find_free_duplicate_name(&pool, newer, &nfc_name).await?;
println!(
"RENAME newer={} (different blob) older={} '{}' → '{}'",
newer.id, older.id, newer.name, disambiguated
);
if !dry_run {
sqlx::query("UPDATE storage.files SET name = $1 WHERE id = $2")
.bind(&disambiguated)
.bind(newer.id)
.execute(&pool)
.await?;
normalize_survivor_name(&pool, older, &nfc_name).await?;
}
stats.renamed_duplicate += 1;
}
}
}
}
println!();
println!("=== Summary ===");
println!(" scanned : {}", stats.scanned);
println!(
" already in NFC : {}",
stats.already_nfc
);
println!(
" normalized in place (no collision) : {}",
stats.normalized_in_place
);
println!(
" dedup-trashed (same content) : {}",
stats.deduped_same_content
);
println!(
" renamed to .duplicate : {}",
stats.renamed_duplicate
);
if dry_run {
println!();
println!("DRY RUN — no rows were written. Re-run without --dry-run to apply.");
}
Ok(())
}
async fn load_non_trashed_files(pool: &PgPool) -> Result<Vec<FileRow>, Box<dyn std::error::Error>> {
let raw = sqlx::query(
"SELECT id, folder_id, user_id, name, blob_hash, created_at
FROM storage.files
WHERE NOT is_trashed
ORDER BY created_at",
)
.fetch_all(pool)
.await?;
let mut out = Vec::with_capacity(raw.len());
for r in raw {
out.push(FileRow {
id: r.try_get("id")?,
folder_id: r.try_get("folder_id")?,
user_id: r.try_get("user_id")?,
name: r.try_get("name")?,
blob_hash: r.try_get("blob_hash")?,
created_at: r.try_get("created_at")?,
});
}
Ok(out)
}
/// Looks for a row in the same `(folder_id, user_id)` scope whose
/// CURRENT name equals `nfc_name`, excluding the row being processed.
/// The other row may itself be in non-NFC form whose normalized
/// representation happens to differ from `nfc_name`; the collision
/// check is intentionally based on stored bytes (matching the
/// UNIQUE-index semantics that this migration is repairing).
async fn find_collision(
pool: &PgPool,
row: &FileRow,
nfc_name: &str,
) -> Result<Option<FileRow>, Box<dyn std::error::Error>> {
let result = sqlx::query(
"SELECT id, folder_id, user_id, name, blob_hash, created_at
FROM storage.files
WHERE name = $1
AND user_id = $2
AND ($3::uuid IS NULL AND folder_id IS NULL
OR folder_id = $3::uuid)
AND id <> $4
AND NOT is_trashed
LIMIT 1",
)
.bind(nfc_name)
.bind(row.user_id)
.bind(row.folder_id)
.bind(row.id)
.fetch_optional(pool)
.await?;
Ok(result.map(|r| FileRow {
id: r.get("id"),
folder_id: r.get("folder_id"),
user_id: r.get("user_id"),
name: r.get("name"),
blob_hash: r.get("blob_hash"),
created_at: r.get("created_at"),
}))
}
/// Finds a free name in the form `{nfc_name}.duplicate` or
/// `{nfc_name}.duplicate-N` for `N >= 1`, scoped to the row's
/// `(folder_id, user_id)`. Returns the first candidate that does
/// not currently exist as a non-trashed row.
async fn find_free_duplicate_name(
pool: &PgPool,
row: &FileRow,
nfc_name: &str,
) -> Result<String, Box<dyn std::error::Error>> {
let mut suffix: u32 = 0;
loop {
let candidate = if suffix == 0 {
format!("{}.duplicate", nfc_name)
} else {
format!("{}.duplicate-{}", nfc_name, suffix)
};
let taken: bool = sqlx::query_scalar(
"SELECT EXISTS(
SELECT 1 FROM storage.files
WHERE name = $1
AND user_id = $2
AND ($3::uuid IS NULL AND folder_id IS NULL
OR folder_id = $3::uuid)
AND id <> $4
AND NOT is_trashed)",
)
.bind(&candidate)
.bind(row.user_id)
.bind(row.folder_id)
.bind(row.id)
.fetch_one(pool)
.await?;
if !taken {
return Ok(candidate);
}
suffix = suffix.saturating_add(1);
// Safety bound — should never trigger under realistic data.
if suffix > 10_000 {
return Err(format!(
"Exhausted .duplicate-N suffixes for '{}' in scope (user={}, folder_id={:?})",
nfc_name, row.user_id, row.folder_id
)
.into());
}
}
}
/// If the surviving (older) row's stored name is not yet in NFC,
/// UPDATE it now that the collision has been resolved.
async fn normalize_survivor_name(
pool: &PgPool,
survivor: &FileRow,
nfc_name: &str,
) -> Result<(), Box<dyn std::error::Error>> {
if survivor.name == nfc_name {
return Ok(());
}
sqlx::query("UPDATE storage.files SET name = $1 WHERE id = $2")
.bind(nfc_name)
.bind(survivor.id)
.execute(pool)
.await?;
Ok(())
}
+12 -2
View File
@@ -1,6 +1,8 @@
use uuid::Uuid;
use crate::domain::services::path_service::{StoragePath, validate_storage_name};
use crate::domain::services::path_service::{
StoragePath, normalize_storage_name, validate_storage_name,
};
// Re-export entity errors from the centralized module
pub use super::entity_errors::{FileError, FileResult};
@@ -107,6 +109,7 @@ impl File {
mime_type: String,
folder_id: Option<String>,
) -> FileResult<Self> {
let name = normalize_storage_name(&name);
if let Err(reason) = validate_storage_name(&name) {
return Err(FileError::InvalidFileName(format!("{name}: {reason}")));
}
@@ -143,6 +146,7 @@ impl File {
created_at: u64,
modified_at: u64,
) -> FileResult<Self> {
let name = normalize_storage_name(&name);
if let Err(reason) = validate_storage_name(&name) {
return Err(FileError::InvalidFileName(format!("{name}: {reason}")));
}
@@ -204,6 +208,7 @@ impl File {
owner_id: Option<Uuid>,
blob_hash: String,
) -> FileResult<Self> {
let name = normalize_storage_name(&name);
if let Err(reason) = validate_storage_name(&name) {
return Err(FileError::InvalidFileName(format!("{name}: {reason}")));
}
@@ -345,7 +350,11 @@ impl File {
// Create storage_path from string
let storage_path = StoragePath::from_string(&path);
// Create directly without validation to avoid errors in DTO conversions
// Create directly without validation to avoid errors in DTO
// conversions. Still NFC-normalize so even DTO-reconstructed
// entities maintain the storage invariant.
let name = normalize_storage_name(&name);
Self {
id,
name,
@@ -365,6 +374,7 @@ impl File {
/// Creates a new version of the file with updated name
pub fn with_name(&self, new_name: String) -> FileResult<Self> {
let new_name = normalize_storage_name(&new_name);
if let Err(reason) = validate_storage_name(&new_name) {
return Err(FileError::InvalidFileName(format!("{new_name}: {reason}")));
}
+12 -4
View File
@@ -1,6 +1,8 @@
use uuid::Uuid;
use crate::domain::services::path_service::{StoragePath, validate_storage_name};
use crate::domain::services::path_service::{
StoragePath, normalize_storage_name, validate_storage_name,
};
// Re-export entity errors from the centralized module
pub use super::entity_errors::{FolderError, FolderResult};
@@ -80,6 +82,7 @@ impl Folder {
parent_id: Option<String>,
owner_id: Option<Uuid>,
) -> FolderResult<Self> {
let name = normalize_storage_name(&name);
// Validate folder name
if let Err(reason) = validate_storage_name(&name) {
return Err(FolderError::InvalidFolderName(format!("{name}: {reason}")));
@@ -171,6 +174,7 @@ impl Folder {
modified_at: u64,
tree_modified_at: u64,
) -> FolderResult<Self> {
let name = normalize_storage_name(&name);
if let Err(reason) = validate_storage_name(&name) {
return Err(FolderError::InvalidFolderName(format!("{name}: {reason}")));
}
@@ -272,10 +276,13 @@ impl Folder {
let storage_path = StoragePath::from_string(&path);
// Create directly without validation to avoid errors in DTO
// conversions. `tree_modified_at` defaults to `modified_at`:
// DTO round-trips lose the real rollup signal, so callers
// that need a freshly-rolled-up etag must reload from the
// conversions. Still NFC-normalize so DTO-reconstructed
// entities maintain the storage invariant.
// `tree_modified_at` defaults to `modified_at`: DTO
// round-trips lose the real rollup signal, so callers that
// need a freshly-rolled-up etag must reload from the
// repository.
let name = normalize_storage_name(&name);
Self {
id,
name,
@@ -293,6 +300,7 @@ impl Folder {
/// Creates a new version of the folder with updated name
pub fn with_name(&self, new_name: String) -> FolderResult<Self> {
let new_name = normalize_storage_name(&new_name);
if let Err(reason) = validate_storage_name(&new_name) {
return Err(FolderError::InvalidFolderName(format!(
"{new_name}: {reason}"
+62
View File
@@ -5,6 +5,29 @@
//! infrastructure/services/path_service.rs because it has file system dependencies.
use std::path::PathBuf;
use unicode_normalization::UnicodeNormalization;
/// NFC-normalize a single file or folder name component.
///
/// The storage layer (PostgreSQL `storage.files.name` and
/// `storage.folders.name`) compares bytes literally — there is no
/// Unicode-aware collation in either UNIQUE index. macOS APFS stores
/// filenames in NFD (decomposed: `é` = `e` + U+0301), while browsers
/// and most other clients post NFC (`é` = U+00E9). Without
/// normalization, the same logical filename can land as two distinct
/// rows: one from a web upload, one from a NextCloud desktop client
/// re-upload of the round-tripped name. The UNIQUE index does not
/// catch it because the bytes differ.
///
/// This function is called at every name-receiving boundary (entity
/// constructors, repository path lookups) so the database invariant
/// becomes "every stored name is NFC". A one-shot migration
/// (`migrate-nfc-filenames`) cleans up rows that pre-date this rule.
///
/// Pure function — no I/O, allocates one `String`.
pub fn normalize_storage_name(name: &str) -> String {
name.nfc().collect()
}
/// Validates a single file or folder name component.
///
@@ -251,4 +274,43 @@ mod tests {
assert!(!path.segments().contains(&"..".to_string()));
assert!(!path.segments().contains(&".".to_string()));
}
// ── NFC normalization tests ─────────────────────────────────
/// Plain ASCII names must round-trip identical bytes.
#[test]
fn test_normalize_ascii_unchanged() {
assert_eq!(normalize_storage_name("file.txt"), "file.txt");
assert_eq!(normalize_storage_name("My Documents"), "My Documents");
}
/// The macOS APFS / NextCloud-desktop pathological case: `é`
/// decomposed as `e` + combining acute (U+0301). Stored bytes
/// `65 cc 81` collapse to NFC `c3 a9`.
#[test]
fn test_normalize_nfd_to_nfc() {
let nfd = "caf\u{0065}\u{0301}";
let nfc = "caf\u{00E9}";
assert_ne!(nfd.as_bytes(), nfc.as_bytes());
assert_eq!(normalize_storage_name(nfd), nfc);
}
/// Already-NFC input must round-trip unchanged. This is the
/// idempotence property the boundary normalization relies on.
#[test]
fn test_normalize_nfc_idempotent() {
let nfc = "Capture d\u{2019}\u{00E9}cran.png";
assert_eq!(normalize_storage_name(nfc), nfc);
// And applying twice is the same as once.
assert_eq!(normalize_storage_name(&normalize_storage_name(nfc)), nfc);
}
/// Multi-codepoint NFD sequences (combining acute + grave +
/// typographic apostrophe) all converge to a single NFC form.
#[test]
fn test_normalize_mixed_accents() {
let nfd = "Capture d\u{2019}\u{0065}\u{0301}cran a\u{0300}.png";
let nfc = "Capture d\u{2019}\u{00E9}cran \u{00E0}.png";
assert_eq!(normalize_storage_name(nfd), nfc);
}
}
@@ -707,8 +707,14 @@ impl FileReadPort for FileBlobReadRepository {
return Ok(None);
}
// Last segment is the filename, preceding segments are the folder path
let filename = segments[segments.len() - 1];
// Last segment is the filename, preceding segments are the
// folder path. NFC-normalize the filename so a NextCloud
// client's NFD-encoded path still hits the NFC row stored
// by a web upload — see `normalize_storage_name` for the
// full rationale.
let filename = crate::domain::services::path_service::normalize_storage_name(
segments[segments.len() - 1],
);
let folder_path = segments[..segments.len() - 1].join("/");
let row = if folder_path.is_empty() {