Commit Graph

1797 Commits

Author SHA1 Message Date
Edouard Vanbelle 836c7a57c1 audit(RUSTSEC-2026-0222): inhibit alert, wasmtime plugin are not used per today 2026-08-01 20:30:33 +02:00
Edouard Vanbelle 88921c975a fix(hurl test): add new job 2026-08-01 20:08:17 +02:00
Edouard Vanbelle c31b8b814d fix(oidc): change the test 2026-08-01 19:58:38 +02:00
Edouard Vanbelle bbfb106a32 feat(maintenance): add a maintenance notification during backend migration 2026-08-01 18:46:17 +02:00
Edouard Vanbelle 142afecbbf feat(storage): add a guide on storage 2026-08-01 17:50:35 +02:00
Edouard Vanbelle f409658c96 feat(storage): hot swap backend on migrate 2026-08-01 17:11:07 +02:00
Edouard Vanbelle 8329b4aa56 feat(storage): improve admin panel 2026-08-01 17:10:06 +02:00
Edouard Vanbelle cc439aaff9 fix(dedup): fix informations 2026-08-01 16:20:49 +02:00
Edouard Vanbelle f949939508 feat(storage): add cmd option --select-storage 2026-08-01 14:36:24 +02:00
Edouard Vanbelle d7c19570a5 feat(storage): wire choice of storage 2026-08-01 14:19:14 +02:00
Edouard Vanbelle 2de71b6d9a feat(storage): add readonly during storage migration 2026-08-01 13:43:04 +02:00
Edouard Vanbelle 6b7bb67500 feat(storage): jobs can choose storage to migrate/scan 2026-08-01 13:06:09 +02:00
Edouard Vanbelle 354e058114 feat(storage): add multi entry in config 2026-08-01 12:58:43 +02:00
Edouard Vanbelle 7534427dc2 fix(oidc): apply clippy recos on PR 652 2026-08-01 12:27:20 +02:00
Edouard Vanbelle 2de5abc6ca plan(storage-multi-entry): simplify the storage migration
Two chronic problems fall out:

1. **Split-brain config.** Admin edits DB via the panel; app boot ignores DB.
   Migration completes; live backend hasn't moved. Admin has to remember to
   copy env vars into `.env` and restart. Two sources of truth for the same
   setting. Cutover is a manual multi-step flow; users routinely get it wrong.

2. **Migration data-loss window on concurrent writes.** The copy walks
   `storage.blobs` in hash order. A blob whose hash is lex-lower than the
   current cursor, written to source AFTER migration passed it, is never
   copied to target. `passed=true, findings=0` completion does NOT guarantee
   target has every blob. Silent.

3. **Migration target selection is fragile.** DTO passes the whole S3 config
   at trigger time; secrets sit plaintext in `admin_settings`. Any future
   pluggable-storage story compounds this (Azure, GCS, WebDAV-as-source, …).

This plan replaces the split-brain model with a single-source-of-truth
architecture:

- `.env` declares **N named storage entries** (immutable per-deploy).
- `admin_settings.storage.active_backend_name` holds ONE row — which named
  entry the app currently runs on. That's the whole runtime config.
- Migration is the atomic transition from one active entry to another. Server
  is put in read-only mode for the copy window; on completion, the active
  pointer flips; a restart cuts over.
2026-08-01 12:05:01 +02:00
Edouard Vanbelle 82a50936d4 feat(storage-migration): move storage mig. to recoverable job 2026-08-01 11:53:51 +02:00
Dionisio Pozo 0c6a33c721 Merge pull request #652 from DragonDuck24/oidc-check-require-email-verification
fix: Make oidc check if email verification is required
2026-08-01 00:12:33 +02:00
Drake 1801150ad4 make oidc check if email verification is required 2026-07-31 11:09:33 -05:00
Dionisio Pozo 8ba2d81297 Merge pull request #651 from EdouardVanbelle/feat/recoverable-jobs
feat(recoverable jobs): add engine + consistency jobs
2026-07-30 10:30:39 +02:00
Dionisio Pozo 53a80cf181 Merge pull request #650 from EdouardVanbelle/chore/db-migration-check
ci: add protection on merge and db migration
2026-07-30 10:30:03 +02:00
Edouard Vanbelle d3c2fc3e94 fix(job): correct amount of jobs + remove cound due to grace window 2026-07-30 02:00:39 +02:00
Edouard Vanbelle 5a87999949 fix(integration test): fix drive test used bytes with grace period 2026-07-30 01:29:09 +02:00
Edouard Vanbelle 48cbec8fae feat(admin ui): too many tabs, change display 2026-07-30 01:14:35 +02:00
Edouard Vanbelle 9fe47a53ea feat(jobs): add admin call to purge jubs result 2026-07-30 00:39:11 +02:00
Edouard Vanbelle 507bc2e98d feat(recoverable-job): add backend_consistency (storage) 2026-07-30 00:15:20 +02:00
Edouard Vanbelle 07beb461d6 feat(jobs/ui): hide consistency_batch, the main button is here 2026-07-29 23:22:36 +02:00
Edouard Vanbelle 8de50ec404 feat(jobs/ui): blobs_consistency add b3sum check 2026-07-29 23:15:45 +02:00
Edouard Vanbelle e822714b50 feat(jobs/ui): reorder job names + fix(blobs_consistency) 2026-07-29 23:01:20 +02:00
Edouard Vanbelle 5527d09618 feat(recoverable-job): check if old blob (no cdc) still remains: notice only 2026-07-29 22:47:07 +02:00
Edouard Vanbelle 61b8571a48 feat(recoverable-job): add grace period for drives, files, folders consistency 2026-07-29 22:29:20 +02:00
Edouard Vanbelle 44356b14f2 feat(recoverable-job): add blobs_consistency job 2026-07-29 22:25:05 +02:00
Edouard Vanbelle 0f12399a48 feat(recoverable-job): fix files_consistency to check blob chunk consistency 2026-07-29 22:17:15 +02:00
Edouard Vanbelle 5881968f50 feat(recoverable-job): add progress view 2026-07-29 08:41:19 +02:00
Edouard Vanbelle 4336eca4d1 feat(recoverable-job): add admin page 2026-07-29 08:03:43 +02:00
Edouard Vanbelle e1556e3d36 feat(recoverable-job): add findings 2026-07-29 07:57:06 +02:00
Edouard Vanbelle 41d83b3053 feat(recoverable-job): add consistency_batch (runs all consistency check) 2026-07-29 01:44:52 +02:00
Edouard Vanbelle 782a5c99bd feat(recoverable-job): add folder_consistency 2026-07-29 01:39:58 +02:00
Edouard Vanbelle 394708c9a1 feat(recov. job): add drive consistency service 2026-07-29 01:25:15 +02:00
Edouard Vanbelle b343ab5e0e plan(job): clarify way to split consistency job 2026-07-29 01:22:12 +02:00
Edouard Vanbelle 8177b04786 feat(recoverable-job): wire API (cancel, view run, ...) 2026-07-28 23:25:37 +02:00
Edouard Vanbelle 302d2ff80f feat(recoverable-job): add core engine 2026-07-28 22:39:07 +02:00
Edouard Vanbelle 2315a54154 ci: add protection on merge and db migration
this is a protection to prevent the merge of any branch adding a database
migration with an timestamp earlier than the current targetted branch
2026-07-28 22:07:45 +02:00
Edouard Vanbelle 10ae6c3e44 doc(job-registry): add SQL entry for recoverable jobs 2026-07-28 22:01:11 +02:00
Edouard Vanbelle 0b7618d858 clarify naming conventions 2026-07-28 22:00:28 +02:00
Edouard Vanbelle 996cb98a6d doc(job-registry): add doc for implementors 2026-07-28 21:24:09 +02:00
Edouard Vanbelle 0e8b1fbbeb refactor(job-registry): simplify the job registering* 2026-07-28 21:13:09 +02:00
Dionisio Pozo 4d64603233 Merge pull request #573 from EdouardVanbelle/plan/consistency-check+job-registry
feat(job registry): normalize engine to handle periodic or event's triggered job
2026-07-28 17:53:25 +02:00
Edouard Vanbelle 4ab57d0d8e test(job-registry): additional hurl coverage 2026-07-28 00:35:34 +02:00
Edouard Vanbelle 24c514fe59 fix(job-registry): fix error handling* 2026-07-28 00:35:31 +02:00
Edouard Vanbelle f66f7fa31f feat(job-registry): remplace /api/admin/internal/trigger-*
remplace /api/admin/internal/trigger-* to /api/admin/jobs/{...}/trigger
remove OXICLOUD_ENABLE_ADMIN_INTERNAL_ENDPOINTS
2026-07-27 23:54:43 +02:00