servers

 1@@ -1,6 +1,6 @@
 2 # Fleet migration: sovrn, moods and servers → one Colmena fleet
 3 
 4-Status: Phases 0–8 done (2026-10-07): infra.mymood.at runs moods, sovrn's metrics stack (infra.sovrn.at) with its telemetry edge, and the backup relay (mxb.eu.sovrn.at), reachable on port 25. Next: Phase 9 (mx99 on the fleet).
 5+Status: Phases 0–9 done except Phase 9's CI step (2026-10-08): infra.mymood.at runs moods, the metrics stack (infra.sovrn.at) and the backup relay (mxb.eu.sovrn.at); mx99.eu.sovrn.at is an arm64 staging cell on the fleet, accepted live (signup, mail, alerts, backups, relay drain). Next: Phase 10 (recovery).
 6 
 7 ## Goals
 8 
 9@@ -1361,6 +1361,55 @@ one change each; moods was never restarted.
10    bump the sovrn input, run sovrn's checks, `colmena apply --on
11    mx99.eu.sovrn.at`. Needs the CI remote URLs (open decision 1).
12 
13+**Phase 9 record (2026-10-07/08; CI step open).**
14+
15+- **mx99 rebuilt** as an arm64 staging cell (Hetzner CAX11: 2 vCPU, 4 GB) by
16+  `just new-host 2.29.20.216 mx99.eu.sovrn.at hetzner sovrn-cell`
17+  (`pdsOriginSuffix = ".pds1.eu.sovrn.at"` pre-set). New host key; R2 and ZDS
18+  secrets reused from the store, `sovrnd-stalwart-password` generated. The
19+  first `just deploy` uploads the Colmena keys nixos-anywhere doesn't. DNS:
20+  A/AAAA, PTR, `sovrn.at` (landing), wildcard `*.pds1.eu.sovrn.at`.
21+- **Building for arm64:** this machine builds aarch64 under qemu (user
22+  enabled `extra-platforms`): Go commands 729 s, ZDS 2105 s. mx99 is also a
23+  Nix remote builder (`/etc/nix/machines`, with its host key): Go commands
24+  219 s cold / 146 s warm, and production cells reuse the same store paths.
25+  The user is fine with the staging cell spending resources on builds.
26+- Live: Stalwart converged (17 created), LE certificate; backup guard armed on
27+  the emptied `eu-mx99`, Litestream and rclone markers fresh; `/healthz` ok;
28+  metrics and logs at infra.sovrn.at under `cell="mx99.eu.sovrn.at"`; all
29+  four scrape jobs up (sovrnd's `/metrics` exists).
30+- **Alert drill:** `CellErrorBurst` fired for mx99 and was emailed. (The
31+  first drill didn't run: `systemd-run` units have no `seq`/`sleep` on PATH;
32+  rerun with `--setenv=PATH=/run/current-system/sw/bin`.) `CellDiskFullSoon`
33+  fired falsely on mx99 from install/build bursts: staging cells
34+  (`fleet.stagingCells`) now judge the trend over 6 h held 1 h (sovrn
35+  `afba45d`, infra.mymood.at generation 18).
36+- **Relay covers mx99** (infra.mymood.at generation 17, route-sync started);
37+  Stalwart's web UI off on the relay (generation 16); `backupMx =
38+  "mxb.eu.sovrn.at"` deployed to mx99.
39+- **sovrnd fixes found live:** a first login was refused when the user had no
40+  service record (`c2db915`); domain activation rolled back because the PDS
41+  health gate didn't wait for ZDS to listen (`6efaf8b`); the freshness probe
42+  raced Litestream's socket on first boot (`2883fe7`). Each has a test.
43+- **Domain signup** (`test.kilimanjaro.io`, user-created through sovrnd):
44+  ZDS instance active, DNS checks passed, mailbox `test@`.
45+- **Mail:** outbound via SMTP2GO passes DKIM (s931828), SPF (em931828
46+  return path) and DMARC at Fastmail; inbound from Gmail delivered (Fastmail
47+  can't test it: it hosts `kilimanjaro.io` and keeps subdomains internal).
48+- **Relay queue-and-drain passed:** Stalwart stopped on mx99; Gmail fell back
49+  to `mxb.eu.sovrn.at`; queued; first attempt to mx99:2525 refused; Stalwart
50+  restarted; delivered on 2525 with STARTTLS at the next retry (~5 min);
51+  mailbox ingested it.
52+- Follow-ups filed: `9cbbce8` (cells must not claim `sovrn.at`; internal
53+  domain for service accounts), `2b0ddfc` (Apple configuration profile),
54+  `0444db1` (guided device setup). iOS needed SMTP credentials entered
55+  separately and a stale account re-added.
56+- Noticed: Stalwart's inbound spam filter takes ~30–60 s per message on both
57+  mx99 and the relay (likely DNS-based list lookups timing out); doesn't
58+  block delivery but worth a look. Stalwart logs DANE disabled (resolvers
59+  don't validate DNSSEC).
60+- **Open:** CI (sovrn checks, then `colmena apply --on mx99`, open decision 1).
61+
62 ### Phase 10: Recovery (`1badc33`)
63 
64 `sovrn-restore.target` and `sovrn-freshen-backups` as in the bug, rehearsed