TRACKING: smoke-server integration tests (deferred live verification) - Issue #28c7d1f

Consolidates every integration test blocked on the new smoke-test server (fresh host + hosted domain with full DNS credentials). Blocked until that server is provisioned. Add future deferred tests as comments on this issue.

Gate (applies to all items below)

[ ] Smoke-test server provisioned with full DNS (MX + SPF + relay CNAMEs + Resend domain verification for notify.sovrn.at).
[ ] Operator secrets in place: sovrn_pds_resend_api_key in vault-shared.yml (ciphertext commit), relay secrets, per-cell ZDS vault.

1. ZDS confirmation emails via Resend (from bug 781ef28, Task 4)

[ ] just update <cell> --tags zds,sovrnd converges green; rerun no-change.
[ ] Per-domain env proof: ~/.config/zds/<slug>.env shows ZDS_MAIL_PROVIDER=resend, ZDS_RESEND_API_KEY=re_..., [email protected], no ZDS_COMAIL_*; file is 0600; unit active; describeServer healthy behind Caddy.
[ ] Mail-path proof: create test account via sovrnd loopback path, trigger verification email, confirm delivery in Resend dashboard/logs; no missing_resend_api_key / missing_email_from in ZDS logs.

2. Stalwart local delivery + relay split (full scope in bug 9bec37e)

[ ] Config dump: MtaOutboundStrategy route is IF is_local_domain(rcpt_domain) THEN 'local' ELSE 'sovrn-relay'; relay object correct; no duplicates after restart.
[ ] Local alice@<hosted> -> bob@<hosted> delivered locally with zero relay hits.
[ ] External mail goes via sovrn-relay only (no direct-MX fallback); relay DKIM/SPF/DMARC pass.
[ ] Mixed-recipient split verified; unknown-local-recipient behavior decided.
[ ] No Gateway not found / id-not-found in logs; stale NOTE in internal/relay/stalwart.go:35-36 removed; docs updated.

How to add tests

Post a comment with ## N. <name>, preconditions, numbered steps with exact commands, and acceptance checkboxes. Keep unit-testable logic in go test; this issue is live-host verification only.

1 Comment

agent 2d85c97 Sep 16

3. SQLite + ZDS-blob backup to R2 (from bug 2559f24, T4+T11)

Preconditions: - R2 bucket <cell-hostname> created manually, jurisdiction matching cell region; bucket-scoped API token recorded. - host_vars/<cell>/vars.yml bucket override (if not the inventory_hostname default) + vaulted sovrn_r2_access_key / sovrn_r2_secret_key committed; fleet-wide endpoint set once in group_vars/all/sovrn.yml. - just update <cell> converges green (litestream + rclone roles applied).

Steps: 1. ssh <cell> 'sudo litestream snapshots -config /etc/litestream.yml' — expect one non-empty snapshot line per static DB (stalwart.db, sovrn.db, oauth.db) plus one per <reversed>.db under litestream-zds/. 2. Onboard a test domain, wait ~1 min, re-run step 1 — expect the new <reversed>.db present with NO Ansible run (dir-watch auto-pickup). 3. ssh <cell> 'sudo /usr/local/sbin/zds-blobs-sync.sh && sudo rclone check /var/lib/zds/blobs/ r2:<cell>/zds-blobs/ --config /etc/rclone.conf' — expect exit 0. 4. just recovery-backup <cell> — expect services stop, WAL checkpoints, final syncs, age report; relay queue covers the gap. 5. Manual provider move (power off old, new box re-linking same mxN.<region>.sovrn.at-{4,6} primaries), then just recovery-bootstrap <cell> — expect restore + integrity_check ok per DB + blob sync-down + services start in order. 6. ssh <cell> 'for db in /var/lib/stalwart/stalwart.db /var/lib/sovrn/sovrn.db /var/lib/sovrn/oauth.db /var/lib/zds/dbs/*.db; do echo "== $db"; sudo sqlite3 "$db" "PRAGMA integrity_check;"; done' — expect every line ok. 7. Relay-drain verify per docs/runbooks/relay-drain-verify.md.

Acceptance: - [ ] Snapshots non-empty for every DB; new ZDS domain picked up without Ansible. - [ ] rclone check clean; blob sync timer active (systemctl is-active zds-blobs-sync.timer). - [ ] Recovery drill green: healthz, JMAP login, ZDS describeServer, litestream snapshots replicating again. - [ ] RTO/RPO posted on T9; runbook deviations fixed in the book. - [ ] Open live-host questions closed: bare litestream snapshot -config actually snapshots (else scope to per-DB); stalwart-cli NDJSON story settled (currently guarded no-op); R2 generations sane after install-then-stop window; restored file ownership correct per service user. - [ ] Cert/config restore verified: cert files present under /var/lib/caddy post-restore with no mass reissue in Caddy logs; config.json present and Stalwart NOT in bootstrap mode; Stalwart ACME cert storage location confirmed (in stalwart.db vs datadir files — if files outside the .db, extend the backup set).