3 files changed,
+48,
-286
+3,
-1
1@@ -5,6 +5,8 @@ The fleet: every NixOS host, deployed with [Colmena](https://github.com/zhaofeng
2 | Host | Provider | Roles | Notes |
3 | --- | --- | --- | --- |
4 | `infra.rtw.run` | Hetzner | `forge` | Git hosting (soft-serve, kilimanjaro.io). Legacy layout (installed with nixos-infect), nixos-26.05 |
5+| `infra.mymood.at` | netcup | `moods`, `sovrn-metrics`, `sovrn-relay` | The shared box: moods, sovrn's metrics stack (`infra.sovrn.at`) and backup MX (`mxb.eu.sovrn.at`). Inbound ports also need netcup's firewall policy |
6+| `mx99.eu.sovrn.at` | Hetzner (CAX11, arm64) | `sovrn-cell` | sovrn's staging cell; also a Nix remote builder for aarch64 |
7
8 ## Layout
9
10@@ -57,4 +59,4 @@ A service reads `config.servers.secrets.<name>.path` (under `/var/lib/servers-ke
11
12 Standard hosts are installed with `just new-host` over the provider's stock image (nixos-anywhere, disko layout). Hetzner images take `FLEET_SSH_KEY` (default `~/.ssh/hetzner`) for root; netcup images only allow a password, so `new-host` asks for one of `keys/admins.pub` and copies it over first. The SSH host key lives at `servers/hosts/<fqdn>/ssh_host_ed25519_key` and is reused on reinstalls (for infra.mymood.at it is a link to the key moods generated). New hosts get `stateVersion` 26.05.
13
14-The project repos' host recipes (`just deploy`, `new-host`, … in sovrn's `Justfile.nix` and moods' `Justfile`) forward here.
15+The project repos' host recipes (`just deploy`, `new-host`, … in sovrn's and moods' `Justfile`) forward here. Neither project keeps host configuration of its own.
+34,
-2
1@@ -1,6 +1,6 @@
2 # Fleet migration: sovrn, moods and servers → one Colmena fleet
3
4-Status: Phases 0–10 done except Phase 9's CI step (2026-10-08): infra.mymood.at runs moods, the metrics stack (infra.sovrn.at) and the backup relay (mxb.eu.sovrn.at); mx99.eu.sovrn.at is an arm64 staging cell, accepted live and rebuilt from its backups in a restore rehearsal. Next: Phase 11 (decommission Ansible).
5+Status: Migration complete (2026-10-08). Every host is deployed by this repository from the operator's machine: infra.rtw.run (forge), infra.mymood.at (moods, sovrn's metrics stack infra.sovrn.at and backup relay mxb.eu.sovrn.at), mx99.eu.sovrn.at (arm64 staging cell). sovrn's Ansible is gone; recovery is automatic and rehearsed.
6
7 ## Goals
8
9@@ -1468,6 +1468,37 @@ without a restore path while there is no production data.
10 - Optional: one small library flake for the three near-identical secrets
11 modules.
12
13+**Phase 11 record (done 2026-10-08).**
14+
15+- **sovrn** (`tsxnqxwz`): `deployment/` deleted (Ansible roles, playbooks,
16+ inventory and the encrypted vaults; 77 files). Also gone: `nix/hosts.nix`,
17+ `nix/hosts.json`, `nix/modules/hetzner.nix`, `nix/keys/`, the
18+ `colmenaHive`/`nixosConfigurations` outputs and the `colmena` and `disko`
19+ inputs, colmena and nixos-anywhere from devenv. `nix/modules/base.nix`
20+ became `nix/tests/node.nix` (the VM test nodes' base only).
21+ `Justfile.nix` folded into `Justfile` (dev loop, gen, tests, the fleet
22+ stubs; `ci-staging` dropped). Docs rewritten for the fleet:
23+ `docs/deployment.md`, runbooks `provision-cell.md`,
24+ `recovery-primary-ip.md` (as rehearsed in Phase 10),
25+ `pds-cell-recovery.md`, `metrics-box.md`, `relay-drain-verify.md`,
26+ `nix/README.md`; secret-provisioning wording in the design docs. All VM
27+ checks pass.
28+- **moods** (`ltsvqxsy`): `nix/hosts.nix`, `nix/hosts.json`,
29+ `nix/modules/{base,vm,hetzner,netcup}.nix`, `nix/keys/`, the host outputs
30+ and inputs, colmena and nixos-anywhere from devenv. Its VM test passes.
31+- **CI: none for now** (user, 2026-10-08): tests and deploys run from this
32+ machine; staging (mx99) first, production cells after. Phase 9's CI step
33+ and open decision 1 are dropped until that changes.
34+- **Bugs closed** (sovrn): `b45a76f`, `d43eebe`, `63eb0d9`, `0e32e5a`,
35+ `a6edca3`, `1badc33`, `ae42c89`, `6cae3a0`, `ebce532`, `3418287`
36+ (superseded), `c3b0d04`, and the epic `3beadb2`. Still open by design:
37+ `dfb37b1` (sovrnd lacks the backup-marker and TLS-expiry series, so
38+ `BackupStale`/`TlsExpiring` can't fire), `af17761` (nixos-26.11),
39+ `edbcac6`, `dc80d76`, and the follow-ups filed during Phases 9–10
40+ (`9cbbce8`, `2b0ddfc`, `0444db1`, `d4988da`, `6844d4f`, `a6c4223`).
41+- The old Ansible vaults (with the keys rotated in Phase 4) remain in
42+ sovrn's history; rewrite it if that matters.
43+
44 ## Rollback summary
45
46 | Step | Rollback |
47@@ -1517,7 +1548,8 @@ without a restore path while there is no production data.
48
49 ## Open decisions
50
51-1. Where CI runs and which remote URLs the fleet uses for sovrn and moods.
52+1. ~~Where CI runs~~: none for now (2026-10-08); everything runs from the
53+ operator's machine.
54 2. `buildOnTarget` for infra.rtw.run.
55 3. Whether and when to rebuild infra.rtw.run on the standard layout (2.8),
56 e.g. together with adding the CI runner.
+11,
-283
1@@ -25,58 +25,6 @@
2 "type": "github"
3 }
4 },
5- "colmena_2": {
6- "inputs": {
7- "flake-compat": "flake-compat_2",
8- "flake-utils": "flake-utils_2",
9- "nix-github-actions": "nix-github-actions_2",
10- "nixpkgs": [
11- "moods",
12- "nixpkgs"
13- ],
14- "stable": "stable_2"
15- },
16- "locked": {
17- "lastModified": 1790091756,
18- "narHash": "sha256-YkaWZQV/OO4ZEmin+RHZ/6qFfBah/qSSXduTQMZTNR0=",
19- "owner": "zhaofengli",
20- "repo": "colmena",
21- "rev": "948491ae031dc0e2e61f0982943dd9964a724bdb",
22- "type": "github"
23- },
24- "original": {
25- "owner": "zhaofengli",
26- "ref": "v0.5.0",
27- "repo": "colmena",
28- "type": "github"
29- }
30- },
31- "colmena_3": {
32- "inputs": {
33- "flake-compat": "flake-compat_3",
34- "flake-utils": "flake-utils_5",
35- "nix-github-actions": "nix-github-actions_3",
36- "nixpkgs": [
37- "sovrn",
38- "nixpkgs"
39- ],
40- "stable": "stable_3"
41- },
42- "locked": {
43- "lastModified": 1790091756,
44- "narHash": "sha256-YkaWZQV/OO4ZEmin+RHZ/6qFfBah/qSSXduTQMZTNR0=",
45- "owner": "zhaofengli",
46- "repo": "colmena",
47- "rev": "948491ae031dc0e2e61f0982943dd9964a724bdb",
48- "type": "github"
49- },
50- "original": {
51- "owner": "zhaofengli",
52- "ref": "v0.5.0",
53- "repo": "colmena",
54- "type": "github"
55- }
56- },
57 "disko": {
58 "inputs": {
59 "nixpkgs": [
60@@ -97,48 +45,6 @@
61 "type": "github"
62 }
63 },
64- "disko_2": {
65- "inputs": {
66- "nixpkgs": [
67- "moods",
68- "nixpkgs"
69- ]
70- },
71- "locked": {
72- "lastModified": 1789770686,
73- "narHash": "sha256-uZkBR7yHdIKUFB5SZdfgh1qkGfI3XmYmI/lTiquxbck=",
74- "owner": "nix-community",
75- "repo": "disko",
76- "rev": "725ea35e410ad83be4931d1bff7e090eacaf3563",
77- "type": "github"
78- },
79- "original": {
80- "owner": "nix-community",
81- "repo": "disko",
82- "type": "github"
83- }
84- },
85- "disko_3": {
86- "inputs": {
87- "nixpkgs": [
88- "sovrn",
89- "nixpkgs"
90- ]
91- },
92- "locked": {
93- "lastModified": 1789770686,
94- "narHash": "sha256-uZkBR7yHdIKUFB5SZdfgh1qkGfI3XmYmI/lTiquxbck=",
95- "owner": "nix-community",
96- "repo": "disko",
97- "rev": "725ea35e410ad83be4931d1bff7e090eacaf3563",
98- "type": "github"
99- },
100- "original": {
101- "owner": "nix-community",
102- "repo": "disko",
103- "type": "github"
104- }
105- },
106 "flake-compat": {
107 "flake": false,
108 "locked": {
109@@ -155,38 +61,6 @@
110 "type": "github"
111 }
112 },
113- "flake-compat_2": {
114- "flake": false,
115- "locked": {
116- "lastModified": 1767039857,
117- "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
118- "owner": "edolstra",
119- "repo": "flake-compat",
120- "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
121- "type": "github"
122- },
123- "original": {
124- "owner": "edolstra",
125- "repo": "flake-compat",
126- "type": "github"
127- }
128- },
129- "flake-compat_3": {
130- "flake": false,
131- "locked": {
132- "lastModified": 1767039857,
133- "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=",
134- "owner": "edolstra",
135- "repo": "flake-compat",
136- "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab",
137- "type": "github"
138- },
139- "original": {
140- "owner": "edolstra",
141- "repo": "flake-compat",
142- "type": "github"
143- }
144- },
145 "flake-utils": {
146 "inputs": {
147 "systems": "systems"
148@@ -259,55 +133,17 @@
149 "type": "github"
150 }
151 },
152- "flake-utils_5": {
153- "inputs": {
154- "systems": "systems_5"
155- },
156- "locked": {
157- "lastModified": 1731533236,
158- "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
159- "owner": "numtide",
160- "repo": "flake-utils",
161- "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
162- "type": "github"
163- },
164- "original": {
165- "owner": "numtide",
166- "repo": "flake-utils",
167- "type": "github"
168- }
169- },
170- "flake-utils_6": {
171- "inputs": {
172- "systems": "systems_6"
173- },
174- "locked": {
175- "lastModified": 1731533236,
176- "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
177- "owner": "numtide",
178- "repo": "flake-utils",
179- "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
180- "type": "github"
181- },
182- "original": {
183- "owner": "numtide",
184- "repo": "flake-utils",
185- "type": "github"
186- }
187- },
188 "moods": {
189 "inputs": {
190- "colmena": "colmena_2",
191- "disko": "disko_2",
192- "flake-utils": "flake-utils_3",
193+ "flake-utils": "flake-utils_2",
194 "nixpkgs": "nixpkgs"
195 },
196 "locked": {
197- "lastModified": 1791341458,
198- "narHash": "sha256-zFUR4tMa+Eyq8wLemdP8ZUEoocOzgJM9FH+1cRt/zDk=",
199+ "lastModified": 1791440086,
200+ "narHash": "sha256-IAzMJfvPCrPBVdp+kDy9AFmKJrW4Hq2l0iebAqdsD/s=",
201 "ref": "HEAD",
202- "rev": "ee8d86d7e12544e051cbdb7fe7eb147abf7b7c34",
203- "revCount": 32,
204+ "rev": "cfdb59c3fb5536f6a0c20876dfda8588931ab504",
205+ "revCount": 33,
206 "type": "git",
207 "url": "file:///home/btburke/projects/moods"
208 },
209@@ -338,50 +174,6 @@
210 "type": "github"
211 }
212 },
213- "nix-github-actions_2": {
214- "inputs": {
215- "nixpkgs": [
216- "moods",
217- "colmena",
218- "nixpkgs"
219- ]
220- },
221- "locked": {
222- "lastModified": 1737420293,
223- "narHash": "sha256-F1G5ifvqTpJq7fdkT34e/Jy9VCyzd5XfJ9TO8fHhJWE=",
224- "owner": "nix-community",
225- "repo": "nix-github-actions",
226- "rev": "f4158fa080ef4503c8f4c820967d946c2af31ec9",
227- "type": "github"
228- },
229- "original": {
230- "owner": "nix-community",
231- "repo": "nix-github-actions",
232- "type": "github"
233- }
234- },
235- "nix-github-actions_3": {
236- "inputs": {
237- "nixpkgs": [
238- "sovrn",
239- "colmena",
240- "nixpkgs"
241- ]
242- },
243- "locked": {
244- "lastModified": 1737420293,
245- "narHash": "sha256-F1G5ifvqTpJq7fdkT34e/Jy9VCyzd5XfJ9TO8fHhJWE=",
246- "owner": "nix-community",
247- "repo": "nix-github-actions",
248- "rev": "f4158fa080ef4503c8f4c820967d946c2af31ec9",
249- "type": "github"
250- },
251- "original": {
252- "owner": "nix-community",
253- "repo": "nix-github-actions",
254- "type": "github"
255- }
256- },
257 "nixpkgs": {
258 "locked": {
259 "lastModified": 1790578696,
260@@ -464,7 +256,7 @@
261 },
262 "pgit": {
263 "inputs": {
264- "flake-utils": "flake-utils_4",
265+ "flake-utils": "flake-utils_3",
266 "nixpkgs": "nixpkgs_3"
267 },
268 "locked": {
269@@ -494,17 +286,15 @@
270 },
271 "sovrn": {
272 "inputs": {
273- "colmena": "colmena_3",
274- "disko": "disko_3",
275- "flake-utils": "flake-utils_6",
276+ "flake-utils": "flake-utils_4",
277 "nixpkgs": "nixpkgs_4"
278 },
279 "locked": {
280- "lastModified": 1791438254,
281- "narHash": "sha256-vlPNtylfbMt7W+Y+4+lAuZKrN07LmiUiEYkerLTCeHo=",
282+ "lastModified": 1791441619,
283+ "narHash": "sha256-lAx0QdfJmK5TP3dzbVWYBTcQv2PYiEqEulr6uebf9js=",
284 "ref": "HEAD",
285- "rev": "9c9ee66a594533acfa944dc8dcf6e37aace1d99b",
286- "revCount": 195,
287+ "rev": "6f3bdcf7a9eb698c4b34dcf50b099f337f656c68",
288+ "revCount": 196,
289 "type": "git",
290 "url": "file:///home/btburke/projects/sovrn"
291 },
292@@ -530,38 +320,6 @@
293 "type": "github"
294 }
295 },
296- "stable_2": {
297- "locked": {
298- "lastModified": 1783625654,
299- "narHash": "sha256-pI1244/PJfTyKhlAr2QYQC55vR6UQdnGA0rJUgtO2IQ=",
300- "owner": "NixOS",
301- "repo": "nixpkgs",
302- "rev": "a0230bd8d5cbd13893b2263918d396a2c7dd0407",
303- "type": "github"
304- },
305- "original": {
306- "owner": "NixOS",
307- "ref": "release-26.05",
308- "repo": "nixpkgs",
309- "type": "github"
310- }
311- },
312- "stable_3": {
313- "locked": {
314- "lastModified": 1783625654,
315- "narHash": "sha256-pI1244/PJfTyKhlAr2QYQC55vR6UQdnGA0rJUgtO2IQ=",
316- "owner": "NixOS",
317- "repo": "nixpkgs",
318- "rev": "a0230bd8d5cbd13893b2263918d396a2c7dd0407",
319- "type": "github"
320- },
321- "original": {
322- "owner": "NixOS",
323- "ref": "release-26.05",
324- "repo": "nixpkgs",
325- "type": "github"
326- }
327- },
328 "systems": {
329 "locked": {
330 "lastModified": 1681028828,
331@@ -621,36 +379,6 @@
332 "repo": "default",
333 "type": "github"
334 }
335- },
336- "systems_5": {
337- "locked": {
338- "lastModified": 1681028828,
339- "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
340- "owner": "nix-systems",
341- "repo": "default",
342- "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
343- "type": "github"
344- },
345- "original": {
346- "owner": "nix-systems",
347- "repo": "default",
348- "type": "github"
349- }
350- },
351- "systems_6": {
352- "locked": {
353- "lastModified": 1681028828,
354- "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
355- "owner": "nix-systems",
356- "repo": "default",
357- "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
358- "type": "github"
359- },
360- "original": {
361- "owner": "nix-systems",
362- "repo": "default",
363- "type": "github"
364- }
365 }
366 },
367 "root": "root",