Follow-up from Task 7 spec review (backup-MX bug d43eebe).

Problem

The backup Stalwart’s domain map (Relay arms + RCPT allowRelaying) is rendered from the Ansible var sovrn_backup_domain_routes, requiring a manual –tags stalwart-backup-bootstrap re-run on every onboard/retire. This reintroduces the control-plane<->backup coupling the at.sovrn.domain.route jetstream design exists to avoid. Safe for v1 (empty map converges CLOSED = reject; stale map bounces, never open-relays) but operator-driven and availability-limiting.

Fix spec

Extend the stalwart_backup role with a new unit (e.g. sovrn-route-sync.service, long-lived consumer or timer): small Go binary (e.g. cmd/route-sync) subscribing to jetstream at.sovrn.domain.route, filtering authorDid == postmaster DID, verifying resolveHandle(postmaster.at.)==author + suffix match (REUSE internal/postmaster.ValidateRoute/VerifyAuthor), cursor store + listRecords backfill + DEGRADED metric, applying adds/removes through the same relay-phase JMAP shapes as the bootstrap script (active arms, retired drops arms; route objects may stay).

No cheaper correct v1 mechanism: a control-plane push would cement the coupling; the manual var is the interim.

Also required before live traffic (Task 8)

Runbook paragraph (relay-drain-verify.md or deployment.md ops): on every onboard/retire, update sovrn_backup_domain_routes + re-run the bootstrap tag; document empty-map-CLOSED and stale-map-bounce semantics. Fold into Task 8 docs work.

2 Comments

agent 0cac0ef Sep 20

Poll-Based Backup Relay Domain Sync — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace the manual sovrn_backup_domain_routes map with a 60-second poll: each cell exposes its hosted-domain list over a bearer-authenticated endpoint, and a small Go service on the pri-20 relay merges the lists and applies domain→cell routing to Stalwart.

Architecture: Cells read their own DB (store.ListDomainsByStatus) and serve GET /backup/domains (bearer token; path deliberately not under /internal/*, which Caddy blocks at the edge). A new cmd/backup-route-sync service polls every configured cell, caches last-known-good per cell on disk (survives the cell outage it exists for), merges with active > degraded > verifying precedence, and applies the FULL relay config (routes, schedule, strategy, RCPT relaying) via internal/stalwart.Client. The existing at.sovrn.domain.route record stays as the on-protocol source of truth but is not consumed by the relay in v1.

Tech Stack: Go (stdlib net/http, crypto/subtle), internal/stalwart JMAP client, internal/store, viper config, Ansible/systemd, SQLite.

Decision log (confirmed with operator)

Poll-only, interval 60s; no push/notification surface on the relay.
Endpoint path /backup/domains.
One static bearer token shared by all cells + the relay (treat domain list as public; simple config).
Endpoint exposes active, degraded, verifying (permissive — pri-20 covers mid-provisioning).
Conflict precedence active > degraded > verifying; ties break by configured cell order; conflicts logged.
Observability: healthz + structured logs only (no /metrics in v1).
Stale cells: keep serving last-known-good indefinitely, with staleness logged (no hard cutoff).
Option A: the Go service owns the entire dynamic relay config; the bash/jq relay phase, sovrn_backup_domain_routes, and the map-hash statefile are removed.

File Structure

Create - internal/appview/backup_domains.go — hosted-domain HTTP handler (+_test.go) - internal/routesync/cellclient.go — fetch one cell’s list (+_test.go) - internal/routesync/merge.go — merge/precedence (+_test.go) - internal/routesync/express.go — Stalwart Expression builders (+_test.go) - internal/routesync/apply.go — full relay JMAP apply (+_test.go) - internal/routesync/state.go — per-cell last-known-good cache (+_test.go) - internal/routesync/poller.go — poll loop + staleness (+_test.go) - cmd/backup-route-sync/main.go — service entrypoint - deployment/roles/stalwart_backup/tasks/routesync.yml - deployment/roles/stalwart_backup/templates/sovrn-route-sync.service.j2 - deployment/roles/stalwart_backup/templates/route-sync.toml.j2

Modify - config.go, config_test.go — [backupdomains] tokenfile - router.go — mount GET /backup/domains - deployment/roles/sovrnd/templates/sovrn.toml.j2 - deployment/inventory/group_vars/all/sovrn.yml — drop sovrn_backup_domain_routes; add token + sync vars - deployment/inventory/host_vars/mxb.eu.sovrn.at/vars.yml - deployment/roles/stalwart_backup/tasks/bootstrap.yml, relay.yml - deployment/roles/stalwart/files/bootstrap-stalwart.sh — remove relay phase (option A) - deployment/scripts/provision-shared-secrets, ensure-bootstrap-secrets - deployment/playbooks/site.yml - docs/runbooks/relay-drain-verify.md, docs/deployment.md, docs/adr/0009-cell-architecture.md, docs/08-security-compliance.md

Phases

A = control plane + sync service (locally testable). B = deployment + docs.

Task 1: `[backupdomains]` config + shared token secret

Files: Modify config.go, config_test.go; deployment/roles/sovrnd/templates/sovrn.toml.j2; deployment/scripts/provision-shared-secrets; deployment/scripts/ensure-bootstrap-secrets; deployment/inventory/group_vars/all/sovrn.yml; deployment/roles/sovrn_secrets/tasks/main.yml.

[ ] Step 1: Failing test TestBackupDomainsTokenFileBinds (t.Setenv SOVRN_BACKUPDOMAINS_TOKENFILE; assert cfg.BackupDomains.TokenFile).
[ ] Step 2: Verify fail — go test . -run TestBackupDomainsTokenFileBinds -v.
[ ] Step 3: Implement BackupDomainsConfig{TokenFile string mapstructure:"tokenfile"}, add BackupDomains to Config, append "backupdomains.tokenfile" to bindEnv. Empty disables the endpoint.
[ ] Step 4: Verify pass.
[ ] Step 5: Ansible secret plumbing: [backupdomains] tokenfile in sovrn.toml.j2; sovrn_backup_domains_token in provision-shared-secrets + required_shared_keys; materialize ${sovrn_secrets_dir}/backup-domains-token 0600 in sovrn_secrets; var in group_vars/all/sovrn.yml.
[ ] Step 6: Commit feat: backupdomains token config and shared secret.

Task 2: `GET /backup/domains` handler

Files: Create internal/appview/backup_domains.go, backup_domains_test.go; modify router.go.

[ ] Step 1: Failing tests: no token → 401; wrong token → 401; good token → 200 with cell and only active/degraded/verifying; BackupDomainsPath not prefixed /internal/.
[ ] Step 2: Verify fail.
[ ] Step 3: Implement constant-time bearer check; ListDomainsByStatus(active, degraded, verifying); JSON {cell,updatedAt,domains:[{name,status}]}.
[ ] Step 4: Mount on top mux only when cfg.BackupDomains.TokenFile != ""; fatal if set but unreadable; cell = cfg.CellSelfHost() fallback cfg.Mail.Hostname.
[ ] Step 5: Verify pass + go test ./....
[ ] Step 6: Commit feat: cell hosted-domains endpoint for backup relay.

Task 3: Cell client + merge

Files: Create internal/routesync/cellclient.go, merge.go (+tests).

[ ] Step 1: Failing tests: bearer header sent; cell mismatch rejected; non-200 rejected; merge precedence active > degraded > verifying; tie-break by cell order; conflict returned.
[ ] Step 2: Verify fail.
[ ] Step 3: Implement Fetch(ctx, httpClient, cell, token) and Merge(cells, per).
[ ] Step 4: Verify pass.
[ ] Step 5: Commit feat: backup relay cell client and domain merge.

Task 4: Stalwart relay apply (option A)

Files: Create internal/routesync/express.go, apply.go (+tests).

[ ] Step 1: Failing tests pinning exact JSON: strategy arm if rcpt_domain == 'example.com' / then 'cell-mx1-eu-sovrn-at' / else first route; relaying else "false"; apply call order routes→schedule→strategy→relaying→reload; route create tolerates primaryKeyViolation.
[ ] Step 2: Verify fail.
[ ] Step 3: Implement routeName, StrategyExpr, RelayingExpr, Apply(ctx, *stalwart.Client, RelayMap) (routes Relay no-auth, remote schedule TTL + 5m/15m/1h/3h/8h/24h/48h/72h ladder, singleton patches, ReloadSettings).
[ ] Step 4: Verify pass.
[ ] Step 5: Commit feat: Go backup relay config apply.

Task 5: Poller, cache, service entrypoint

Files: Create internal/routesync/state.go, poller.go (+tests), cmd/backup-route-sync/main.go.

[ ] Step 1: Failing tests: cache atomic round-trip; failed fetch retains last-known-good and does not apply; apply only when merged map changes; per-cell staleness tracked.
[ ] Step 2: Verify fail.
[ ] Step 3: Implement State (Load/Save temp+rename 0640), Poller (60s ticker, immediate apply on start, SIGHUP force-pull, retain-on-failure indefinitely), and main.go (flags/env CELLS, POLL_INTERVAL=60s, TOKEN_FILE, STALWART_URL, STALWART_USER, ADMIN_CREDS_FILE, STATE_PATH, LISTEN=127.0.0.1:8091; GET /healthz 200 when every cell seen within 3×interval else 503; structured slog only — no /metrics).
[ ] Step 4: Verify pass — go test ./internal/routesync/... ./cmd/....
[ ] Step 5: Commit feat: backup route sync poller service.

Task 6: Ansible — ship service, drop the manual map

Files: Create deployment/roles/stalwart_backup/tasks/routesync.yml, templates/sovrn-route-sync.service.j2, templates/route-sync.toml.j2. Modify tasks/bootstrap.yml, tasks/relay.yml, files/bootstrap-stalwart.sh, group_vars/all/sovrn.yml, host_vars/mxb.eu.sovrn.at/vars.yml, playbooks/site.yml.

[ ] Step 1: Drop sovrn_backup_domain_routes; add token/sync vars.
[ ] Step 2: Build+ship binary on mail_backup hosts (mirror sovrnd role go build + GLIBC assert + copy + patchelf); install config + unit (After=stalwart.service, Restart=on-failure), enable+start.
[ ] Step 3: Remove the strategy/relaying/route/schedule sections and relay phase from bootstrap-stalwart.sh; remove relay invocation from bootstrap.yml/site.yml; delete relay.yml + map-hash statefile.
[ ] Step 4: routesync.yml asserts cells non-empty; renders route-sync.toml; no_log on token-adjacent tasks.
[ ] Step 5: python3 deployment/scripts/test-deploy and ansible --check on mxb.
[ ] Step 6: Commit deploy: backup route sync service replaces manual domain map.

Task 7: Docs, runbook, ADR, bug close-out

Files: Modify docs/runbooks/relay-drain-verify.md, docs/deployment.md, docs/adr/0009-cell-architecture.md, docs/08-security-compliance.md.

[ ] Step 1: Rewrite manual map-sync paragraph: automatic poll ≤60s; endpoint/token; permissive status policy; conflict precedence; retire grace ≥ 2 poll intervals.
[ ] Step 2: Document secret + rotation.
[ ] Step 3: ADR-0009 note: relay consumes the authenticated domain-list endpoint (pull); at.sovrn.domain.route remains the public on-protocol record.
[ ] Step 4: Update bug 0a0fb42.
[ ] Step 5: Commit docs: poll-based backup relay domain sync.

agent 08a502f Sep 20

Implemented: poll-based backup relay domain sync (supersedes the jetstream watcher)

Design (operator-approved): cells serve their hosted-domain list over an authenticated endpoint; the relay polls it and owns the full Stalwart relay config. No ATProto/jetstream dependency, no control-plane push, no manual domain map.

Commits (base `bf9cc8e9`)

493169bc feat: backupdomains token config and shared secret
115c5c62 feat: cell hosted-domains endpoint for backup relay
e73d843d feat: backup relay cell client and domain merge
3bd6b255 feat: Go backup relay config apply
857bda23 feat: backup route sync poller service
cfece1c0 deploy: backup route sync service replaces manual domain map
280932e4 fix: route-sync uses the recovery admin secret
15d3fb46 docs: poll-based backup relay domain sync
d6f5aa71 docs: correct route-sync auth comment in group_vars
31dc2f41 fix: identify backup-domains endpoint by cell mail hostname

What shipped

GET /backup/domains on sovrnd (bearer auth; active|degraded|verifying; not under /internal/; identity = cell mail hostname; mounted only when backupdomains.tokenfile set).
Single shared static token sovrn_backup_domains_token → /etc/sovrn/secrets/backup-domains-token (0600).
cmd/backup-route-sync: 60s poll, durable per-cell last-known-good (/var/lib/sovrn-route-sync/state.json), retain-forever on cell failure, merge active>degraded>verifying (cell-order tie-break, conflicts logged), applies only on canonical change. Owns per-cell Relay routes, remote schedule TTL/backoff, outbound-strategy arms, and MtaStageRcpt.allowRelaying else=false. Loopback /healthz + JSON logs; no /metrics.
Ansible ships the binary/unit/env on mail_backup hosts; auth = recovery admin (sovrn_stalwart_user + sovrn_stalwart_secretfile).
Removed: sovrn_backup_domain_routes, sovrn_backup_relay_state, stalwart_backup/tasks/relay.yml, and the shell relay phase.
at.sovrn.domain.route records are still written (on-protocol source of truth) but not consumed by the relay in v1.

Verification (all green)

SOVRN_INTEGRATION=0 go test ./...; go vet ./...; go test -race (routesync + cmd); python3 deployment/scripts/test-deploy (23 tests); bash -n bootstrap script; ansible-playbook --syntax-check; no stale references.

Residual risk — MUST verify live (Task 8)

Cross-host GET https://<cell>/backup/domains via Caddy with the shared token; sovrn_backup_cells entries are cell mail/MX hostnames and must serve the app vhost over HTTPS (invariant documented).
Recovery-admin Basic auth to Stalwart JMAP over loopback.
Exact JMAP payloads against real Stalwart: route /get+/set, schedule name query+set, strategy/relaying singleton updates, ReloadSettings; no notUpdated/notCreated; confirm the inferred primaryKeyViolation type string.
Freshly bootstrapped relay rejects a non-served RCPT before the first successful poll (Stalwart default allowRelaying = !is_empty(authenticated_as) ⇒ unauthenticated port 25 denied).
Total cell outage leaves the relay serving only cached domains; retired domains drop on the next successful poll; state survives a Stalwart wipe.
GLIBC_2.41 ceiling + patchelf interpreter fix on the trixie relay host.

Migration: existing vaults must run just provision-shared-secrets once (adds sovrn_backup_domains_token) before just update.

Backup route sync: jetstream watcher replacing manual domain-map var