T2 follow-up: comail/crawler provisioning for per-domain ZDS env

closed
#781ef28 opened by agent Sep 15

Follow-up discovered during T2 implementation (bug 2f8a863, verified in final review).

Sovrnd reads SOVRN_PDS_COMAIL_API_KEY / SOVRN_PDS_COMAIL_DID / SOVRN_PDS_CRAWLERS via the new PDSConfig bindings (config.go), but nothing sets them: sovrn.toml.j2 has no [pds] section and no sovrn_comail_* vars exist anywhere in deployment/ (fleet vault generator mints only OIDC + OAuth keys). email_from / operator_handle resolve via viper defaults; plc_directory is intentionally optional.

Current behavior (by design, tested): empty comail/crawlers fail per-record RenderEnv validation loudly and Reconcile skips that unit’s env — degraded, not silent. So no cell can serve a domain until this lands.

Scope: - Storage decision: fleet-shared (vault-shared.yml) vs per-cell (host_vars/<cell>/vault.yml) for comail key/DID; crawler list is non-secret (group vars). - Recipe support alongside provision-shared-secrets (fleet) or provision-zds-secrets (per-cell) for generation + structural asserts. - Group-vars (sovrn_comail_*-shaped) + [pds] block in sovrn.toml.j2 (or env-file equivalent for sovrnd). - First-cell converge proving Reconcile renders env end-to-end with real values.

Acceptance: just update <cell> --tags zds,sovrnd on a real cell renders a complete per-domain env (no validation skips) and the domain’s ZDS starts healthy behind Caddy.

3 Comments

agent 748919e Sep 16

PDS mail via Resend (replacing comail) — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Replace comail with one fleet-shared Resend API key so just update <cell> --tags zds,sovrnd renders complete per-domain ZDS envs with ZDS_MAIL_PROVIDER=resend.

Architecture: Hardcode resend in RenderEnv (same as comail was hardcoded — no provider knob, YAGNI). PDSConfig swaps comail_api_key/comail_did for resend_api_key. Secret travels vault-shared.yml → sovrn.toml [pds] value (0640 root:sovrn) → SupervisorConfig → per-instance <slug>.env (0600). Non-secrets via group vars.

Tech Stack: Go (viper config, pdslifecycle), Ansible (sovrnd/sovrn_secrets roles, vault), ZDS 0.3.1, Resend API.

Review verdict (why Resend works): Pinned ZDS 0.3.1 supports --mail-provider comail|resend + ZDS_RESEND_API_KEY (verified via zds --help, binary strings, upstream docs/operations.md). Resend needs only key + verified sender domain — strictly simpler than comail (key + DID). Decisions: replace comail entirely (no dual support); fleet-shared key in vault-shared.yml; sender [email protected]; crawlers stay bsky.network,vsky.network (vsky proxies requestCrawl to all known public relays — no need to enumerate atproto.at/relays).


Task 0: Operator pre-req (no code, out-of-band, blocks Task 4)

  • [ ] Step 1: Verify notify.sovrn.at in Resend — Resend dashboard → Domains → add notify.sovrn.at → publish TXT/SPF/DKIM → wait for Verified.
  • [ ] Step 2: Mint + store the key — dashboard → API Keys → create (sending scope) → secrets add resend_pds_api_key. Never paste into chat or group vars.

Task 1: Go — replace comail with Resend (TDD)

Files: - Modify: config.go (PDSConfig, setDefaults, bindEnv) - Modify: pds_supervision.go (wire ResendAPIKey) - Modify: internal/pdslifecycle/env.go (EnvParams, RenderEnv, validateEnv, secretKeys) - Modify: internal/pdslifecycle/supervisor.go (SupervisorConfig, ActivationParams, envParamsForRecord, Activate) - Modify: pds_supervision_test.go, internal/pdslifecycle/env_test.go, supervisor_test.go, reconcile_env_test.go, testdata/env-basic.golden

  • [ ] Step 1: Write the failing test — update the static env values subtest in pds_supervision_test.go to set SOVRN_PDS_RESEND_API_KEY=re_test123 and expect cfg.PDS.ResendAPIKey + EmailFrom == "[email protected]".
  • [ ] Step 2: Run to verify it fails — go test -run 'TestPDSSupervisionEnvGate/static' . -v, expect FAIL.
  • [ ] Step 3: Implement config.go — PDSConfig{Supervision, ResendAPIKey, EmailFrom, Crawlers, OperatorHandle, PLCDirectory}; default [email protected]; bindEnv pds.resend_api_key (drop comail keys).
  • [ ] Step 4: Implement env.go/supervisor.go/pds_supervision.go — RenderEnv emits ZDS_MAIL_PROVIDER=resend + ZDS_RESEND_API_KEY, drops ZDS_COMAIL_*; validator resend api key is required; secretKeys swap; supervisor config/activation params swap.
  • [ ] Step 5: Update golden + remaining tests; run SOVRN_INTEGRATION=0 go test ./... — expect PASS.
  • [ ] Step 6: Commit — jj commit -m "feat(pds): send PDS mail via fleet-shared Resend key" <files>.

Task 2: Deployment — [pds] block + fleet-shared vault + asserts

Files: - Modify: deployment/roles/sovrnd/templates/sovrn.toml.j2 (append [pds]) - Modify: deployment/inventory/group_vars/all/sovrn.yml (sovrn_pds_* vars) - Modify: deployment/roles/sovrn_secrets/tasks/main.yml (assert, no_log) - Operator: deployment/inventory/group_vars/all/vault-shared.yml (ciphertext only)

  • [ ] Step 1: [pds] block — supervision/resend_api_key/email_from/crawlers/operator_handle/plc_directory.
  • [ ] Step 2: Group vars — sovrn_pds_supervision: true, sovrn_pds_email_from: "[email protected]", sovrn_pds_crawlers: "https://bsky.network,https://vsky.network", operator/plc defaults, vault-var indirection for the key.
  • [ ] Step 3: sovrn_secrets assert — sovrn_pds_resend_api_key defined, non-empty, ^re_[A-Za-z0-9_-]+$, no_log, gated on supervision.
  • [ ] Step 4: Operator vault step — ansible-vault edit vault-shared.yml, add key, re-encrypt, commit ciphertext (separate operator commit).
  • [ ] Step 5: Commit — jj commit -m "feat(deploy): wire fleet-shared Resend key into sovrnd [pds]" <files>.

Task 3: Docs touch-ups

  • [ ] Step 1: Confirm _precheck needs nothing (shared-vault-encrypted check already covers the fleet file).
  • [ ] Step 2: Note in provision-shared-secrets docstring that Resend keys are pasted manually (relay-style), not generated.
  • [ ] Step 3: Commit docs.

Task 4: First-cell converge — DEFERRED until smoke-test host is provisioned

  • [ ] Step 0 (gate): smoke-test host provisioned — BLOCKED, skip for now.
  • [ ] Steps 1–4 (converge, env-file proof, mail-path proof, record on this bug) run once the host exists.
agent 75821ae Sep 16

Implementation complete (Tasks 1-3). Task 4 deferred pending smoke-test host.

  • Commit 1 (Go): PDSConfig comail_api_key/comail_did -> resend_api_key; email_from default [email protected]; RenderEnv emits ZDS_MAIL_PROVIDER=resend + ZDS_RESEND_API_KEY; validator/secretKeys/supervisor wiring swapped; golden + all tests updated. SOVRN_INTEGRATION=0 go test ./… green.
  • Commit 2 (deploy): [pds] block in sovrn.toml.j2; sovrnpds* group vars (crawlers bsky.network,vsky.network); sovrnsecrets assert on ^re shape gated by supervision; provision-shared-secrets docstring notes manual paste. Verified: Jinja render of full sovrn.toml through real LoadConfig yields correct PDSConfig (temp test, since removed).
  • Blocked on operator: verify notify.sovrn.at in Resend, paste re_ key into vault-shared.yml (ciphertext commit), then Task 4 first-cell converge.
agent 798114e Sep 16

Deferred Task 4 (first-cell converge + Resend mail-path proof) moved to tracking issue 28c7d1f, consolidated with the relay split verification from 9bec37e. Both blocked on smoke-test server provisioning.