Multi-MX architecture: single appview, multiple mail servers, failover

closed
#6ca5959 opened by agent Sep 11

Goal

Separate the currently-colocated appview (https://sovrn.at) from mail (mx1.sovrn.at) so the playbook can create a mail server with or without the sovrnd appview service. Future mx2.sovrn.at hosts mail only.

Decisions locked in discussion (2026-09-11): - Shared FoundationDB across mail hosts (per ADR-0008 active/passive). - Failover design is OPEN — candidate: dedicated imap.sovrn.at and smtp.sovrn.at entries repointed at the active MX on failover (vs floating VIP vs per-mx hostnames + MX priorities). To be worked out here. - sovrnd DB failover is OPEN — leaning Litestream read replica for the SQLite app DB, alternatives: single Postgres instance, or move app data into FDB (single clustered DB). To be worked out here.

Scope

  • Inventory: physical hosts vs service names (appview group with exactly one member; mail group with N members; mail_primary/mail_backup).
  • Vars: move sovrn_mail_hostname per-host (mx1/mx2), gate appview-only vars; split group_vars/all fleet singletons vs mail vs appview.
  • site.yml: gate sovrnd + didwellknown + appview Caddy vhost on appview membership; mail roles run on all mail hosts.
  • Caddy: appview vhost vs challenge-only mail vhost (never proxy /jmap//admin//healthz); mx2 gets challenge-only vhost.
  • Secrets: OIDC/OAuth keys to appview only; Stalwart recovery secret to mail (revisit per-host vs shared once FDB is shared — currently per-host, which breaks with a shared registry).
  • FDB: greenfield (first host wipes + fresh cluster) vs join (later hosts join existing cluster, no wipe, reuse credential). Current “bootstrap destroys” must be gated accordingly.
  • TLS/ACME with shared registry: one AcmeProvider + Domain Automatic with explicit SANs ([mx1, mx2] or future [imap, smtp] service names); confirm no double-renew race on shared Task table.
  • just prechecks: mail-only targets must not demand appview secrets.

Non-goals (smoke test, bug 70790ed)

Minimal mx1 ACME-challenge proxy to Stalwart stays in the smoke scope; everything above is deferred here.

1 Comment

agent 6bc4ad5 Sep 13

Superseded by bug 75966cc (cell architecture tracking — locked 2026-09-13).

Resolutions, recorded in the tracking issue: - Shared FoundationDB across mail hosts: rejected. Cells are independent (SQLite everywhere); no shared registry, no shared store. - Failover design (was OPEN): decided — Hetzner floating IPs, never DNS repoint. Recovery = fenced IP move to a rebuilt box (see T9 runbook). - sovrnd DB failover (was OPEN: Litestream replica vs Postgres vs FDB): decided — per-cell SQLite + Litestream continuous replication to R2 (T4). - Appview/mail split: moot — appview colocates per cell in v1. - Per-host Stalwart secret question evaporates with the shared registry.

Remaining work lives under 75966cc (T2/T3/T4/T9/T10). Closing.