sovrn

Experimental AT protocol email hosting

T9: cell-model ADR + runbooks (provision, floating-IP recovery, relay drain)

closed
#c2f7f81 opened by agent Sep 13

Parent: bug 75966cc (cell architecture tracking). Do first — the role work (T2/T3/T4/T10) has no contract without this.

Goal

Authoritative ADR + operator runbooks for the cell model.

Scope

  • ADR: cell contents, naming (mxN.<region>.sovrn.at, EU/US), per-cell bucket layout (litestream/, stalwart-blobs/ w/ keyPrefix, zds-blobs/), floating-IP-over-DNS decision, manual-scale policy + per-cell ceilings (disk %, backup size/restore time, IMAP p99), relay topology. Supersedes ADR-0008 (FDB active/passive + VIP + leader lease); widens ADR-0002.
  • Runbooks: provision-cell (primary/IPv6 bootstrap -> restore-or-fresh -> attach floating IP); fenced floating-IP recovery (stop old if reachable, final flush, power off, hcloud API move, rDNS note, healthz gate before AND after move); relay-drain verify (queued mail delivered post-move).
  • Hetzner specifics: same network zone for moves, guest /32 expectation, rDNS-per-floating-IP (set once, follows the IP), certs restored not reissued.

Acceptance

  • ADR merged under docs/adr/; runbooks rehearsed in staging at least once (recovery drill green).

3 Comments

agent c726f57 Sep 13

Phase 1+2 complete (working copy, uncommitted)

Phase 1 — ADR merged (working copy): docs/adr/0009-cell-architecture.md with D24–D32: cell contents, naming/regions/buckets, bucket layout + ownership invariant, floating-IP-over-DNS, mail flow, manual scale with owner-tuned ceiling placeholders, ZDS pick + blob publicity, rejected backends with measurement-gated revisit triggers, and D32 per-domain postmaster (postmaster@<domain>, postmaster.at.<domain>, auto-created at domain-activation, lives in the cell’s ZDS, migrates as a unit with the domain; D32c: receivable via alias/forward to the registering administrator’s mailbox, EmailAlias preferred, no postmaster login credential). Amends ADR-0006 D8; supersedes ADR-0008 (already bannered); widens ADR-0002. Consequences section maps the role work (T2/T3/T4/T10) to exact files. docs/09-open-questions.md Q7/Q8 updated; Caddyfile stale 6ca5959 comment repointed at ADR-0009/75966cc.

Phase 2 — runbooks drafted (working copy): docs/runbooks/provision-cell.md (mx99.eu.sovrn.at worked example, gates, TODO(T2/T4/T6/T10) markers where roles don’t exist yet), docs/runbooks/recovery-floating-ip.md (fencing order, stop-then-snapshot flush, same-zone rebuild, mode=recover restore order with integrity gates, dual healthz gates around the hcloud move, never-dos), docs/runbooks/relay-drain-verify.md (queue baseline, drain proof, loop/DSN checks).

Phase 3 — rehearsal pending: needs T2+T3+T4+T6+T10 landed on mx99; then run recovery verbatim, fix the book where reality differs, and post RTO/RPO evidence here. T9 stays open until the drill is green.

agent c423f27 Sep 13

Revision (2026-09-13): primary-IP recycling replaces floating IPs

Terminology correction, locked: Hetzner Floating IPs cannot substitute for Primary IPs (a server must hold a same-type Primary IP to carry a Floating IP; Floating IPs can’t attach at creation and always need guest-side /32). The design standardizes on Primary IPv4s managed as account resources, named mxN.<region>.sovrn.at-{4,6} (protection on, auto-delete off; only mx99 exists so far), recycled across rebuilds: power off old -> create new box re-linking the same primaries -> boot (cloud-init configures primaries, zero guest network config).

Net effect, all preserved: one IPv4 per cell (not two), MX/A records never change, .ssh/config never changes on recovery (same hostname, same IP — only host keys rotate, handled by the Justfile recipe), no IPs in inventory or playbooks, no provider API calls from Ansible. ADR-0009 D27, the runbooks, and docs/deployment.md are being reworked to this in the working copy; T10 scope shrinks to SSH/v6 posture + known_hosts (network task deleted); new T11 tracks the recovery role pair.

agent ce22f07 Sep 13

Primary-recycle rework complete (working copy, uncommitted)

  • ADR-0009 D27 rewritten (recycled mxN.<region>.sovrn.at-{4,6} primaries; floating-IP concept + guest-/32 removed); title, D28, consequences, references aligned.
  • docs/runbooks/recovery-floating-ip.md renamed to recovery-primary-ip.md and rewritten to the 5-step shape (recovery-backup / manual move / recovery-bootstrap, host-key rotation in-recipe, dual health gates, never-dos); provision-cell.md and relay-drain-verify.md aligned; docs/deployment.md recovery/model/roles sections reworked; 09 Q8 + ADR-0008 banner touched up.
  • New T11 (bug c0e0ddb) tracks the recovery role pair.
  • Justfile: recovery-backup (strict keys) + recovery-bootstrap (ssh-keygen -R + keyscan + ControlPersist purge first, strict Ansible throughout) scaffolded; both fail clearly until T11 playbooks land. just --summary confirms both recipes parse.
  • Note: devenv.nix hcloud/HCLOUD_CONTEXT changes in the working copy are the owner’s, not this change.