Edouard Vanbelle
7db547a669
TL;DR:
fix duplicate filename via:
```
docker exec <container> migrate-nfc-filenames --dry-run # preview
docker exec <container> migrate-nfc-filenames # execute
```
== issue ==
Last week I uploaded Capture d'écran 2026-06-03 à 20.04.24.png from the web. It synced down to Nextcloud on my Mac. Two minutes later, the Web UI was showing the file twice.
Both rows had:
- the same name
- the same size
- the same content hash
So why two rows? Because to PostgreSQL, the names weren't the same.
Web upload (browser → Postgres):
"é" stored as 1 codepoint (U+00E9) bytes: c3 a9 ← NFC
NiextCloud client (macOS → Postgres):
"é" stored as 2 codepoints (e + U+0301) bytes: 65 cc 81 ← NFD
macOS's APFS keeps filenames in NFD (decomposed); browsers send NFC (composed). Visually é and é are identical. To WHERE name = $1 they're two different keys. Our UNIQUE index on (folder_id, name, user_id) never fired — and the row count quietly drifted every time a Mac user touched an accented
filename.
== The fix is two halves ==
1. No new duplicates — every name-receiving boundary (file upload, NC PUT, rename, MOVE, path lookup) now NFC-normalizes before touching the database. The storage invariant becomes "every stored name is NFC".
2. Clean up existing data — one-shot migrate-nfc-filenames binary walks storage.files, NFC-normalizes any non-NFC row, and resolves the collisions we've accumulated. Same-content duplicates go to trash (recoverable); different-content collisions get renamed with a .duplicate suffix.
== use of the clean up ==
example of use (do not forget to define env **DATABASE_URL**)
either
`cargo run --bin migrate-nfc-filenames -- --dry-run`
or
`cargo build --bin migrate-nfc-filenames`
`./target/debug/migrate-nfc-filenames --dry-run`
example:
```
% ./target/debug/migrate-nfc-filenames --dry-run
=== NFC filename migration (DRY RUN — no writes) ===
Loaded 543 non-trashed file rows
NORMALIZE 163451b5-5e6c-404b-9b1e-f4b01a2b7269 user=42433185-4717-416d-9a15-4580fff171ec 'Capture d’écran 2026-03-20 à 14.44.50.png' → 'Capture d’écran 2026-03-20 à 14.44.50.png'
NORMALIZE 827dddec-4dd5-48c2-a120-dec5289f7d29 user=969deca6-7935-4f12-a430-4d636b62fa3e 'Capture d’écran 2026-04-03 à 15.43.38.png' → 'Capture d’écran 2026-04-03 à 15.43.38.png'
NORMALIZE 09559934-a620-472d-9ba8-fc3cfeb6dc6f user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-03 à 20.05.38.png' → 'Capture d’écran 2026-06-03 à 20.05.38.png'
NORMALIZE 5ce6dbf9-0562-4758-8783-671aa9069590 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 'Capture d’écran 2026-06-05 à 11.07.25.png' → 'Capture d’écran 2026-06-05 à 11.07.25.png'
DEDUP newer=26bcf82b-99cc-45c8-9d69-dd7e5c4484ff (trash, same blob) older=df3adc67-a778-424d-a817-b930c75f3b06 user=a0643a21-0092-4a84-9dde-7ac4e76bc1a5 hash=0d2cc7b0ffce2850
=== Summary ===
scanned : 543
already in NFC : 538
normalized in place (no collision) : 4
dedup-trashed (same content) : 1
renamed to .duplicate : 0
DRY RUN — no rows were written. Re-run without --dry-run to apply.
```
once valid remove --dry-run
91 lines
4.5 KiB
Docker
91 lines
4.5 KiB
Docker
# ─── Stage 1: Shared build base (avoids duplicate apk install) ────────────────
|
|
FROM rust:1.94.1-alpine3.23 AS base
|
|
RUN apk --no-cache upgrade && \
|
|
apk add --no-cache musl-dev pkgconfig postgresql-dev gcc perl make
|
|
|
|
# ─── Stage 2: Cache dependencies ─────────────────────────────────────────────
|
|
FROM base AS cacher
|
|
WORKDIR /app
|
|
COPY Cargo.toml Cargo.lock ./
|
|
# build.rs + static/ are needed so the build script can run and set OUT_DIR
|
|
COPY build.rs ./
|
|
COPY static static
|
|
# Create a minimal project to download and cache dependencies
|
|
RUN mkdir -p src/bin && \
|
|
echo 'fn main() { println!("Dummy build for caching dependencies"); }' > src/main.rs && \
|
|
echo 'fn main() {}' > src/bin/generate-openapi.rs && \
|
|
echo 'fn main() {}' > src/bin/migrate-nfc-filenames.rs && \
|
|
cargo build --release && \
|
|
rm -rf src static-dist target/release/deps/oxicloud* target/release/build/oxicloud-*
|
|
|
|
# ─── Stage 3: Build the application ──────────────────────────────────────────
|
|
FROM base AS builder
|
|
WORKDIR /app
|
|
# Copy cached dependencies (only target dir and cargo registry)
|
|
COPY --from=cacher /app/target target
|
|
COPY --from=cacher /usr/local/cargo/registry /usr/local/cargo/registry
|
|
# Copy source, build script, and static assets
|
|
COPY Cargo.toml Cargo.lock build.rs ./
|
|
COPY src src
|
|
COPY static static
|
|
COPY migrations migrations
|
|
# askama templates — read at *compile time* by the derive macro, so
|
|
# they must be present in the build stage even though they're embedded
|
|
# into the final binary and never read from disk at runtime.
|
|
COPY templates templates
|
|
# Build with all optimizations (DATABASE_URL only needed at compile-time for sqlx)
|
|
ARG DATABASE_URL="postgres://postgres:postgres@localhost/oxicloud"
|
|
RUN DATABASE_URL="${DATABASE_URL}" cargo build --release
|
|
|
|
# ─── Stage 4: Minimal runtime image ──────────────────────────────────────────
|
|
FROM alpine:3.23.3
|
|
|
|
# OCI image metadata
|
|
LABEL org.opencontainers.image.title="OxiCloud" \
|
|
org.opencontainers.image.description="Ultra-fast, secure & lightweight self-hosted cloud storage built in Rust" \
|
|
org.opencontainers.image.url="https://github.com/DioCrafts/OxiCloud" \
|
|
org.opencontainers.image.source="https://github.com/DioCrafts/OxiCloud" \
|
|
org.opencontainers.image.vendor="DioCrafts" \
|
|
org.opencontainers.image.licenses="MIT"
|
|
|
|
# Install only necessary runtime dependencies and update packages
|
|
# su-exec is needed by the entrypoint to drop privileges after fixing volume permissions
|
|
RUN apk --no-cache upgrade && \
|
|
apk add --no-cache libgcc ca-certificates libpq tzdata su-exec && \
|
|
addgroup -g 1001 -S oxicloud && \
|
|
adduser -u 1001 -S oxicloud -G oxicloud
|
|
|
|
# Copy the compiled binary and entrypoint (--chmod avoids extra RUN chmod layers)
|
|
COPY --from=builder --chmod=755 /app/target/release/oxicloud /usr/local/bin/
|
|
# Ship the NFC filename migration binary alongside the server so
|
|
# operators can run it inside the container without a separate Rust
|
|
# toolchain — `docker exec <container> migrate-nfc-filenames --dry-run`
|
|
# to preview, drop `--dry-run` to execute. One-shot tool, safe to
|
|
# ship; it only mutates `storage.files` rows whose name ≠ NFC(name).
|
|
COPY --from=builder --chmod=755 /app/target/release/migrate-nfc-filenames /usr/local/bin/
|
|
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
|
|
RUN sed -i 's/\r//' /usr/local/bin/entrypoint.sh && \
|
|
chmod 755 /usr/local/bin/entrypoint.sh
|
|
|
|
# Copy processed static files (bundled/minified by build.rs in release)
|
|
COPY --from=builder --chown=oxicloud:oxicloud /app/static-dist /app/static
|
|
# Create storage directory with proper permissions
|
|
RUN mkdir -p /app/storage && chown -R oxicloud:oxicloud /app/storage
|
|
|
|
# Set working directory
|
|
WORKDIR /app
|
|
|
|
# Expose application port
|
|
EXPOSE 8086
|
|
|
|
# Liveness probe — verifies the HTTP server is up (no DB check, fast).
|
|
# Docker / Compose / Swarm will mark the container unhealthy after 3 failures.
|
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=10s --retries=3 \
|
|
CMD wget -qO- http://localhost:8086/health || exit 1
|
|
|
|
# Entrypoint fixes volume permissions then drops to oxicloud user.
|
|
# The container starts as root so it can chown mounted volumes,
|
|
# then su-exec drops privileges before running the application.
|
|
ENTRYPOINT ["entrypoint.sh"]
|
|
CMD ["oxicloud"]
|