Multi-MX architecture: single appview, multiple mail servers, failover
closedGoal
Separate the currently-colocated appview (https://sovrn.at) from mail
(mx1.sovrn.at) so the playbook can create a mail server with or without
the sovrnd appview service. Future mx2.sovrn.at hosts mail only.
Decisions locked in discussion (2026-09-11):
- Shared FoundationDB across mail hosts (per ADR-0008 active/passive).
- Failover design is OPEN — candidate: dedicated imap.sovrn.at and
smtp.sovrn.at entries repointed at the active MX on failover (vs
floating VIP vs per-mx hostnames + MX priorities). To be worked out here.
- sovrnd DB failover is OPEN — leaning Litestream read replica for the
SQLite app DB, alternatives: single Postgres instance, or move app data
into FDB (single clustered DB). To be worked out here.
Scope
- Inventory: physical hosts vs service names (
appviewgroup with exactly one member;mailgroup with N members;mail_primary/mail_backup). - Vars: move
sovrn_mail_hostnameper-host (mx1/mx2), gate appview-only vars; splitgroup_vars/allfleet singletons vsmailvsappview. site.yml: gatesovrnd+didwellknown+ appview Caddy vhost on appview membership; mail roles run on allmailhosts.- Caddy: appview vhost vs challenge-only mail vhost (never proxy
/jmap//admin//healthz);mx2gets challenge-only vhost. - Secrets: OIDC/OAuth keys to appview only; Stalwart recovery secret to mail (revisit per-host vs shared once FDB is shared — currently per-host, which breaks with a shared registry).
- FDB: greenfield (first host wipes + fresh cluster) vs join (later hosts join existing cluster, no wipe, reuse credential). Current “bootstrap destroys” must be gated accordingly.
- TLS/ACME with shared registry: one AcmeProvider + Domain Automatic with
explicit SANs (
[mx1, mx2]or future[imap, smtp]service names); confirm no double-renew race on shared Task table. justprechecks: mail-only targets must not demand appview secrets.
Non-goals (smoke test, bug 70790ed)
Minimal mx1 ACME-challenge proxy to Stalwart stays in the smoke scope; everything above is deferred here.
1 Comment
Superseded by bug 75966cc (cell architecture tracking — locked 2026-09-13).
Resolutions, recorded in the tracking issue: - Shared FoundationDB across mail hosts: rejected. Cells are independent (SQLite everywhere); no shared registry, no shared store. - Failover design (was OPEN): decided — Hetzner floating IPs, never DNS repoint. Recovery = fenced IP move to a rebuilt box (see T9 runbook). - sovrnd DB failover (was OPEN: Litestream replica vs Postgres vs FDB): decided — per-cell SQLite + Litestream continuous replication to R2 (T4). - Appview/mail split: moot — appview colocates per cell in v1. - Per-host Stalwart secret question evaporates with the shared registry.
Remaining work lives under 75966cc (T2/T3/T4/T9/T10). Closing.