Backups on NixOS: Litestream + rclone + health markers - Issue #0e32e5a

Port roles/litestream and roles/rclone.

Sketch: - Litestream: - services.litestream or a custom unit (check the nixpkgs version against the 0.5.13 pin) - replicates stalwart.db, sovrn.db, oauth.db and ZDS DBs to R2 (EU jurisdiction endpoint) - R2 creds as keys - freshness timer writing /var/lib/sovrn/health/litestream.ok - rclone: - zds-blobs-sync timer (every 15 min) + cell-config-sync (Caddy storage, Stalwart config to R2) - rclone.conf rendered at runtime from keys - health marker rclone.ok - Keep the health-marker contract with sovrnd’s GET /healthz (max ages 15m / 45m).

Done when both markers stay fresh on a scratch cell and R2 shows current objects.

3 Comments

agent 0fe2372 Oct 7

Done in NixOS (2026-10-07), pending live acceptance on mx99 (Phase 9): - vkmnmqtk: nix/modules/backup.nix (sovrn.backup, cell role enables it). sovrn-litestream (stalwart.db, sovrn.db, oauth.db, waitlist.db (new: Ansible missed it), ZDS dir watch), sovrn-litestream-freshness (5 min, sync -wait, health/litestream.ok), sovrn-backup-sync (15 min, rclone of Caddy storage + ZDS blobs, health/rclone.ok); sovrnd health marker paths set. - nixpkgs Litestream 0.5.17. Snapshot interval/retention moved to the top-level snapshot block; per-replica keys were silently ignored by 0.5 under Ansible. - Stalwart config.json no longer copied (declarative now). - New guard: backup units arm the box only on an empty bucket and otherwise refuse, so a rebuilt cell never syncs over its old backups before restore (1badc33) or a deliberate discard (touch /var/lib/sovrn-backup/armed). - checks.backup (MinIO): replication, later ZDS DB, markers, restore with data, rclone objects, refusal and arming. cell passes. - eu-mx99 still holds the Ansible mx99’s backups (31 objects, 3.8 MiB); decide restore vs discard before the new mx99 backs up.

agent 0ce43d2 Oct 7

Revised (user, 2026-10-07), folded into vkmnmqtk: waitlist.db is backed up only on the landing cell (fleet.rootHost, the one cell serving sovrn.at); sovrnd creates an empty one on every cell. The fleet now asserts rootHost names a sovrn-cell and isn’t overridden per host (Ansible’s landing-root guard). checks.backup passes with the test cell as landing cell.

agent 05e13e2 Oct 8

Closing (user, 2026-10-08): done-condition met: on mx99 both markers stay fresh and R2 holds current Litestream replicas, ZDS blobs and Caddy storage; restore from them rehearsed live (1badc33).