S2 complete — Stalwart-on-FDB de-risked (2026-09-08)

Stage 2 is done and verified live. The FDB seam is no longer an unknown.

What changed (repo)

• nix/pkgs/stalwart.nix: added foundationdb build feature (OSS, not enterprise-gated) + postPatch pinning the bindings fdb-7_4 → fdb-7_3 to match nixpkgs’ foundationdb 7.3.68 (a 7.4-api client rejects the 7.3 libfdb_c at runtime). pkgs.foundationdb.lib added as link input.
• nix/images/stalwart.nix: image now carries fdbserver/fdbcli + libfdb_c.so.
• nix/scripts/serve-stalwart.sh: SOVRN_DATASTORE=fdb mode — starts a local single-process fdbserver, configures the DB once (single ssd, sentinel-gated), bootstraps Stalwart against FoundationDb. Restart path reuses the persisted cluster. sqlite mode unchanged (/data/config.json).
• devenv.nix: passes SOVRN_DATASTORE through (default sqlite).

Verification (all against the live FDB harness)

• Full integration suite passes on FDB: TestIntegrationProvisionUserStory (C2 lifecycle), TestIntegrationAppPassword (IMAP/SMTP app-password round-trip), TestIntegrationOIDCBearerAuth, TestIntegrationAppViewXRPCEndToEnd, TestIntegrationAppViewCustomDomainDevGate, TestIntegrationHappyPathLexicon — all green, identical behavior to SQLite.
• Restart/failover: stop container → start new one on the same data dir → fdbserver rejoins the persisted cluster → Stalwart reconnects → suite passes again. No data loss, no reconstitution.
• Single-node FDB requires no coordinator (confirmed live; Coordinator::Disabled is the default — crates/registry/src/schema/enums.rs:264).
• DKIM teardown on FDB verified: objectIsLinked guard fires with linkedObjects=[DkimSignature…]; destroy-signatures → delete-domain leaves zero orphaned signatures. See cf71374.

Findings

1. The fdb-7_3 pin is a one-line Stalwart fork. Per docs/05 §5 it is AGPL and must be published; tracked in nix/pkgs/stalwart.nix comment + ADR-0008.
2. Test hygiene (pre-existing, not FDB-specific): internal/integration/appview_test.go cleanup uses t.Context(), which is already canceled when t.Cleanup runs, so domains/tenants/DkimSignatures leak after each run. Story/oidc/apppassword tests use a fresh background context and clean up correctly. Worth a follow-up bug; not a blocker.
3. 9d84518 (pagination) remains open and unchanged — PoC-scale is unaffected; address before prod scale.

Unblocks S3 (auth + app-password wiring) and S4 (Ansible deploy) — no FDB-specific work remains in the critical path.

S3 complete — real auth + app-password wiring (2026-09-08)

The auth seam is no longer a stub. All changes are green against the live harness.

What changed (repo)

• internal/svcverifier (new) — wraps indigo’s ServiceAuthValidator: bearer getServiceAuth JWTs verified against the caller’s DID-doc signing key (key-refresh on failure), aud == service DID, exp/iat enforced, optional lxm (route NSID) match. Unit-tested (valid, wrong-aud, lxm-mismatch, expired, unknown-signer).
• internal/appview/auth.go — Auth{DevDID, Verifier}: bearer service-auth is verified and the subject DID injected; a request with no bearer falls back to DevDID (development) or is rejected 401. DevDID is no longer an unconditional default.
• App-password XRPC — new lexicons at.sovrn.mail.createAppPassword / listAppPasswords / removeAppPassword (+ defs#appPasswordView), regenerated types, and handlers in internal/appview issuing/listing/revoking via internal/appwd.
• router.go — top-level dispatcher: /xrpc/* behind auth; /.well-known/openid- configuration + /jwks (tokenissuer, unauthenticated); /oauth/* (authbroker client-metadata/login/callback, unauthenticated, enabled only when oauth.clientid set). internal/authbroker/handler.go added.
• config.go — oidc.keyfile, service.did, oauth.{clientid,callbackurl,keyfile}.

Verification

• TestIntegrationAppPasswordXRPC (live): tenant→domain→account → createAppPassword → IMAP LOGIN succeeds (the read-mail proof) → listAppPasswords → removeAppPassword → IMAP rejects the revoked secret. Passes.
• TestRouterOIDCAndOAuthEndpoints: discovery doc (issuer correct), JWKS, OAuth client-metadata, and 401-without-credentials. Passes.
• TestRouterDevAuthAllowsXRPC: DevDID path still serves XRPC. Passes.
• Full non-integration suite + full integration suite: green.

Notes / remaining

• The browser OAuth flow (authbroker) is wired but not exercised end-to-end here — a full flow needs a publicly reachable client-metadata URL + live PDS. The confidential client is constructed when oauth.clientid is set; attestation key persists to oauth-attestation.pem (dev-file provider; KMS in prod).
• svcverifier uses indigo’s identity.DefaultDirectory() (PLC + did:web + caching) — production may want a bounded resolver; flagged for later, not PoC-blocking.
• The “OAuth login → DNS gate → activate → read mailbox” full E2E against a live PDS is the one remaining S3 item for a public server; it needs the spaces-alpha PDS OAuth support exercised, which is a follow-up (the read-mail seam is proven via app-password).

Unblocks S4 (Ansible deploy): the binary now exposes authenticated XRPC + OIDC metadata + OAuth front-of-house + app-password issuance.

S4 complete — app-view UI skeleton (templ + Tailwind v4) — 2026-09-09

The browser app-view exists as a minimal server-rendered skeleton covering the six user-testing steps. All green against the live harness.

What changed (repo)

• internal/appview/ui/ (new): templ components (layout/login/home/domain/ account), HTTP handlers, a browser-session auth middleware, and embedded static assets. Tailwind v4 (CSS-first: @import "tailwindcss"; + @source "../*.templ";), compiled by the standalone CLI into static/tailwind.css; static/app.js is ~10 lines of copy-to-clipboard JS.
• Routes (on the top-level mux): GET /login (open), /static/* (open), and the session-authed catch-all / (home, domains, accounts, app passwords). Session auth reads the sovrn_session cookie → broker.ResumeSession → DID; falls back to DevDID in development.
• internal/store: tenants.owner_did column (+ in-place ALTER TABLE migration), GetTenantByOwnerDID; the UI ensures one tenant per DID and lists that tenant’s domains.
• internal/appview/provision.go: extracted ProvisionDomain/ProvisionAccount so the XRPC handlers and the UI share one provisioning path.
• router.go: retains the *authbroker.Broker; mounts UI routes last so XRPC/OAuth/OIDC/static win over the catch-all.
• Tooling: go.mod adds a-h/templ v0.3.1020; devenv.nix adds pkgs.templ + pkgs.tailwindcss_4; Justfile gen now generates ./api/sovrn ./internal/appview/ui; ui/generate.go carries //go:generate for templ + tailwind.

Verification

• TestIntegrationAppViewUI (live harness): home → add domain (custom) → DNS page shows MX/SPF/DKIM → add account → IMAP guide → issue app password → IMAP login succeeds with the UI-issued secret.
• Unit tests (ui_test.go): buildDNS, home/domain rendering via the real route+mux, session DevDID fallback, 303→/login without a session.
• Full unit + integration suites green; go vet + gofmt clean.

Notes / remaining

• Login depends on a working OAuth flow: GET /login → POST /oauth/login requires a reachable oauth.clientid + a PDS that completes the flow. The post-login flows are verified via the DevDID session fallback; the real atproto OAuth round-trip is the one remaining S4 item for a public server and stays a documented follow-up.
• DNS prompt is static (MX/SPF/DKIM shown; no live readiness polling in the UI) — the background verifier still activates domains; the page just shows the records + status.
• App-password secret is rendered once server-side; no client-side masking.

Unblocks S5 (Ansible deploy): the binary now serves the full user-testing loop over HTTP.

S5 — Ansible secrets pattern landed (vault file → 0600 host file)

Implements the secrets half of S5 per the revised 2964546 direction: no secret is ever generated at runtime; everything is ansible-vault-encrypted in-repo and materialized as files. This is the pattern all future services follow.

Files (all new, follow docs/deployment.md layout)

• inventory/hosts.yml — primary / backup / staging pool
• group_vars/all/sovrn.yml — non-secret vars: sovrn_secrets_dir (/etc/sovrn/secrets), the three host secret paths
• group_vars/all/vault-shared.yml — fleet-wide OIDC PEM + OAuth attestation key (base64); plaintext placeholder, operator pastes just gen-secrets output then ansible-vault encrypt
• host_vars/primary.sovrn.example/vault.yml — per-host Stalwart recovery-admin password; copy per host
• roles/sovrn_secrets/tasks/main.yml — asserts shared vault vars are real values (fails on placeholders), writes both keys 0600 no_log: true
• roles/stalwart/tasks/secret.yml — per-host credential: mode=bootstrap generates once on the controller + prints a persist-back reminder; otherwise asserts the vault value exists (recover/update)
• playbooks/site.yml — entry point, -e mode=bootstrap|recover|update (default update, matching docs/deployment.md)

Key decisions

• OIDC + OAuth keys are shared fleet-wide (same value behind the LB); Stalwart recovery-admin is per-host (differs per Stalwart DB by construction).
• Bootstrap stays Ansible-only: no generation code in sovrnd. Dev Stalwart credential stays owned by the container entrypoint (nix/scripts/serve-stalwart.sh); just gen-secrets deliberately does not create one (a second credential would mismatch the DB-persisted recovery admin and be rejected).
• Vault files are committed as ciphertext; the operator holds the vault password. Placeholders fail fast via assert, same philosophy as the app.

Verification

• go test ./... green (incl. new fail-fast tests); all 7 Ansible YAML files parse.
• sovrnd with no secrets exits 1: config: stalwart.secretfile is required ....
• go run ./cmd/sovrn-gensecrets writes both keys 0600 and refuses to overwrite.

Remaining S5 (not this change): full common/foundationdb/caddy/goapp/dns roles, packaging, TLS/MX, live smoke test.

S5 single-server smoke (sovrn.at) Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Bring up one ephemeral live server (sovrn.at, MX mx1.sovrn.at) with control plane + Stalwart on single-instance FDB, all identity URLs as paths off https://sovrn.at.

Architecture: Ansible renders a static public did.json (no secret) and a sovrn.toml with locked single-server values; Caddy terminates TLS for sovrn.at and serves /.well-known/did.json as a static file ahead of the sovrnd reverse-proxy. FDB runs single-instance (single ssd, local cluster file); backup role is a no-op for smoke with an rclone-R2 target var reserved for prod.

Tech Stack: Ansible (vault-encrypted secrets already landed), Caddy, FoundationDB single-process, Go sovrnd (viper TOML+env), did:web + atproto service-auth.

Locked decisions (user): service.did=did:web:sovrn.at#svc_mail with fragment; serviceEndpoint=https://sovrn.at (bare host); did.json has NO private key — static public file provisioned by Ansible, served by Caddy; FDB smoke = single instance, no backup; prod backup = R2 via rclone (var only, empty for smoke).

Task 1: Single-server non-secret vars

Files: - Modify: group_vars/all/sovrn.yml

• [ ] Step 1: Append locked single-server vars to group_vars/all/sovrn.yml

# Single-server live smoke (bug 70790ed S5): one origin for everything.
sovrn_appview_url: "https://sovrn.at"
sovrn_oidc_issuer: "https://sovrn.at"
sovrn_oidc_audience: "sovrn-mail"
sovrn_oauth_clientid: "https://sovrn.at/oauth/client-metadata"
sovrn_oauth_callbackurl: "https://sovrn.at/oauth/callback"
sovrn_service_did: "did:web:sovrn.at#svc_mail"
sovrn_mail_hostname: "mx1.sovrn.at"
sovrn_mail_spf: "v=spf1 mx -all"
sovrn_server_host: "127.0.0.1"
sovrn_server_port: 8081
sovrn_datadir: "/var/lib/sovrn"
sovrn_stalwart_url: "http://127.0.0.1:8080"
sovrn_env: "production"
# did.json: static public file (no secret). Caddy serves it directly so
# did:web resolution works even when sovrnd is down. The commented
# verificationMethod slot is for future outgoing service-auth only.
sovrn_wellknown_dir: "/var/lib/sovrn/wellknown"
sovrn_did_document_endpoint: "https://sovrn.at"
# FDB smoke topology: single instance, no backup. Prod target is R2 via
# rclone; leave empty for the ephemeral smoke test.
sovrn_fdb_mode: "single"
sovrn_backup_target: ""

• [ ] Step 2: Validate YAML parses

Run: python3 -c "import yaml,sys; yaml.safe_load(open('group_vars/all/sovrn.yml')); print('YAML OK')" Expected: YAML OK

• [ ] Step 3: Commit

jj commit -m "feat(deploy): single-server sovrn.at non-secret vars" group_vars/all/sovrn.yml

Task 2: Single-server inventory

Files: - Modify: inventory/hosts.yml

• [ ] Step 1: Replace example pool with the single smoke host

# Single-server live smoke (bug 70790ed S5). Ephemeral: no backup/standby.
# Prod restores the primary/backup/staging pool per docs/deployment.md.
all:
  hosts:
    sovrn.at:
  children:
    mail_primary:
      hosts:
        sovrn.at:

• [ ] Step 2: Validate YAML parses

Run: python3 -c "import yaml,sys; yaml.safe_load(open('inventory/hosts.yml')); print('YAML OK')" Expected: YAML OK

• [ ] Step 3: Commit

jj commit -m "feat(deploy): single-server inventory for sovrn.at smoke" inventory/hosts.yml

Task 3: didwellknown role (static did.json, no secret)

Files: - Create: roles/didwellknown/tasks/main.yml - Create: roles/didwellknown/templates/did.json.j2

• [ ] Step 1: Write the failing check — role files must exist

Run: ls roles/didwellknown/tasks/main.yml roles/didwellknown/templates/did.json.j2 Expected: FAIL (No such file) before, PASS after creation.

• [ ] Step 2: Create roles/didwellknown/templates/did.json.j2

{
  "@context": ["https://www.w3.org/ns/did/v1"],
  "id": "{{ sovrn_service_did.split('#')[0] }}",
  "service": [
    {
      "id": "{{ sovrn_service_did }}",
      "type": "SovrnMailService",
      "serviceEndpoint": "{{ sovrn_did_document_endpoint }}"
    }
  ]
}
{# Future outgoing service-auth only: add a verificationMethod here backed by
   a vaulted P-256 private key (same file-0600 pattern as OIDC/OAuth keys).
   NOT needed for smoke: sovrn only verifies inbound tokens today. #}

• [ ] Step 3: Create roles/didwellknown/tasks/main.yml

- name: Create wellknown directory
  ansible.builtin.file:
    path: "{{ sovrn_wellknown_dir }}"
    state: directory
    owner: root
    group: "{{ sovrn_user }}"
    mode: "0755"

- name: Render static did.json (public, no secret)
  ansible.builtin.template:
    src: did.json.j2
    dest: "{{ sovrn_wellknown_dir }}/did.json"
    owner: root
    group: "{{ sovrn_user }}"
    mode: "0644"

• [ ] Step 4: Render the template locally and assert fragment/bare-host shape

Run: python3 - << 'EOF' from jinja2 import Template src = open('roles/didwellknown/templates/did.json.j2').read().split('{#')[0] import json doc = json.loads(Template(src).render(sovrn_service_did='did:web:sovrn.at#svc_mail', sovrn_did_document_endpoint='https://sovrn.at')) assert doc['id'] == 'did:web:sovrn.at', doc assert doc['service'][0]['id'] == 'did:web:sovrn.at#svc_mail', doc assert doc['service'][0]['serviceEndpoint'] == 'https://sovrn.at', doc assert 'verificationMethod' not in doc, doc print('did.json OK:', doc) EOF Expected: did.json OK: ... with bare-host endpoint and #svc_mail fragment, no verificationMethod.

• [ ] Step 5: Commit

jj commit -m "feat(deploy): didwellknown role renders static did.json" roles/didwellknown/

Task 4: Caddy role (single vhost + static did.json)

Files: - Create: roles/caddy/tasks/main.yml - Create: roles/caddy/templates/Caddyfile.j2

• [ ] Step 1: Create roles/caddy/templates/Caddyfile.j2

sovrn.at {
  encode gzip
  # did:web resolution must not depend on sovrnd being up.
  handle /.well-known/did.json {
    root * {{ sovrn_wellknown_dir }}
    file_server
  }
  handle {
    reverse_proxy {{ sovrn_server_host }}:{{ sovrn_server_port }}
  }
}

• [ ] Step 2: Create roles/caddy/tasks/main.yml

- name: Install Caddyfile
  ansible.builtin.template:
    src: Caddyfile.j2
    dest: /etc/caddy/Caddyfile
    owner: root
    group: root
    mode: "0644"
  notify: Reload caddy

- name: Enable and start caddy
  ansible.builtin.systemd:
    name: caddy
    enabled: true
    state: started

• [ ] Step 3: Create roles/caddy/handlers/main.yml

- name: Reload caddy
  ansible.builtin.systemd:
    name: caddy
    state: reloaded

• [ ] Step 4: Validate templates render (jinja2) and YAML parses

Run: python3 -c "import yaml; yaml.safe_load(open('roles/caddy/tasks/main.yml')); yaml.safe_load(open('roles/caddy/handlers/main.yml')); print('YAML OK')" Expected: YAML OK

• [ ] Step 5: Commit

jj commit -m "feat(deploy): caddy single-vhost with static did.json" roles/caddy/

Task 5: goapp sovrn.toml (locked single-server values)

Files: - Create: roles/goapp/tasks/main.yml - Create: roles/goapp/templates/sovrn.toml.j2

• [ ] Step 1: Create roles/goapp/templates/sovrn.toml.j2

[server]
host = "{{ sovrn_server_host }}"
port = {{ sovrn_server_port }}
datadir = "{{ sovrn_datadir }}"
env = "{{ sovrn_env }}"

[stalwart]
url = "{{ sovrn_stalwart_url }}"
username = "{{ sovrn_stalwart_user }}"
secretfile = "{{ sovrn_stalwart_secretfile }}"

[mail]
hostname = "{{ sovrn_mail_hostname }}"
spf = "{{ sovrn_mail_spf }}"
imapport = 993
submitport = 587

[appview]
url = "{{ sovrn_appview_url }}"

[oidc]
issuer = "{{ sovrn_oidc_issuer }}"
audience = "{{ sovrn_oidc_audience }}"
keyfile = "{{ sovrn_oidc_keyfile }}"

[service]
did = "{{ sovrn_service_did }}"

[oauth]
clientid = "{{ sovrn_oauth_clientid }}"
callbackurl = "{{ sovrn_oauth_callbackurl }}"
keyfile = "{{ sovrn_oauth_keyfile }}"

[verification]
interval = "5m"
reapafter = "48h"

Note: oauth.clientid uses the real served path /oauth/client-metadata (not the stale dev /client-metadata.json in Justfile dev). devauth.did and postmaster.did intentionally absent in prod.

• [ ] Step 2: Create roles/goapp/tasks/main.yml

- name: Create sovrn data directory
  ansible.builtin.file:
    path: "{{ sovrn_datadir }}"
    state: directory
    owner: "{{ sovrn_user }}"
    group: "{{ sovrn_user }}"
    mode: "0750"

- name: Render sovrn.toml
  ansible.builtin.template:
    src: sovrn.toml.j2
    dest: /etc/sovrn/sovrn.toml
    owner: root
    group: "{{ sovrn_user }}"
    mode: "0640"
  notify: Restart sovrnd

• [ ] Step 3: Create roles/goapp/handlers/main.yml

- name: Restart sovrnd
  ansible.builtin.systemd:
    name: sovrnd
    state: restarted

• [ ] Step 4: Render locally and assert locked values

Run: python3 - << 'EOF' from jinja2 import Template out = Template(open('roles/goapp/templates/sovrn.toml.j2').read()).render( sovrn_server_host='127.0.0.1', sovrn_server_port=8081, sovrn_datadir='/var/lib/sovrn', sovrn_env='production', sovrn_stalwart_url='http://127.0.0.1:8080', sovrn_stalwart_user='sovrn-admin', sovrn_stalwart_secretfile='/etc/sovrn/secrets/stalwart-secret', sovrn_mail_hostname='mx1.sovrn.at', sovrn_mail_spf='v=spf1 mx -all', sovrn_appview_url='https://sovrn.at', sovrn_oidc_issuer='https://sovrn.at', sovrn_oidc_audience='sovrn-mail', sovrn_oidc_keyfile='/etc/sovrn/secrets/oidc-key.pem', sovrn_service_did='did:web:sovrn.at#svc_mail', sovrn_oauth_clientid='https://sovrn.at/oauth/client-metadata', sovrn_oauth_callbackurl='https://sovrn.at/oauth/callback', sovrn_oauth_keyfile='/etc/sovrn/secrets/oauth-attestation.key') assert 'did = "did:web:sovrn.at#svc_mail"' in out, out assert 'issuer = "https://sovrn.at"' in out, out assert 'clientid = "https://sovrn.at/oauth/client-metadata"' in out, out assert 'hostname = "mx1.sovrn.at"' in out, out assert 'devauth' not in out and 'postmaster' not in out, out print('sovrn.toml OK') EOF Expected: sovrn.toml OK

• [ ] Step 5: Commit

jj commit -m "feat(deploy): goapp sovrn.toml for sovrn.at smoke" roles/goapp/

Task 6: FoundationDB single-instance (+ R2 var for prod)

Files: - Create: roles/foundationdb/tasks/main.yml - Create: roles/foundationdb/defaults/main.yml

• [ ] Step 1: Create roles/foundationdb/defaults/main.yml

# Smoke: single instance, local cluster file, no backup.
# Prod: set sovrn_backup_target to the rclone R2 remote (e.g. "r2:sovrn-fdb-backups")
# and enable the backup tasks (added when the backup role lands).
sovrn_fdb_mode: "single"
sovrn_backup_target: ""
sovrn_fdb_cluster_file: "/etc/foundationdb/fdb.cluster"

• [ ] Step 2: Create roles/foundationdb/tasks/main.yml

- name: Configure single-instance FDB
  ansible.builtin.command: fdbcli --exec "configure new single ssd"
  args:
    creates: "{{ sovrn_fdb_cluster_file }}"
  when: sovrn_fdb_mode == "single"

- name: Assert no backup target for ephemeral smoke
  ansible.builtin.assert:
    that:
      - sovrn_backup_target == ""
    fail_msg: "Smoke host is ephemeral: leave sovrn_backup_target empty. Prod uses the rclone R2 remote."
    when: inventory_hostname == "sovrn.at"

• [ ] Step 3: Validate YAML

Run: python3 -c "import yaml; yaml.safe_load(open('roles/foundationdb/tasks/main.yml')); yaml.safe_load(open('roles/foundationdb/defaults/main.yml')); print('YAML OK')" Expected: YAML OK

• [ ] Step 4: Commit

jj commit -m "feat(deploy): foundationdb single-instance for smoke" roles/foundationdb/

Task 7: Wire playbook + verify

Files: - Modify: playbooks/site.yml

• [ ] Step 1: Append role includes in dependency order

    - name: Render static did.json
      ansible.builtin.include_role:
        name: didwellknown

    - name: Configure single-instance FoundationDB
      ansible.builtin.include_role:
        name: foundationdb

    - name: Install Caddy single vhost
      ansible.builtin.include_role:
        name: caddy

    - name: Render sovrn.toml
      ansible.builtin.include_role:
        name: goapp

(Order: secrets → stalwart secret → didwellknown → foundationdb → caddy → goapp.)

• [ ] Step 2: Run full verification

Run: SOVRN_INTEGRATION=0 go test ./... 2>&1 | tail -5 Expected: all packages PASS (esp. svcverifier fragment-aud tests).

Run: for f in $(git ls-files 'roles/**/*.yml' 'inventory/*.yml' 'group_vars/**/*.yml' 'playbooks/*.yml' 2>/dev/null || jj file list 'roles/**' 'inventory/**' 'group_vars/**' 'playbooks/**'); do python3 -c "import yaml,sys; yaml.safe_load(open('$f'))" || echo "BAD: $f"; done; echo YAML-SWEEP-DONE Expected: YAML-SWEEP-DONE with no BAD lines.

Run: ansible-playbook playbooks/site.yml --syntax-check 2>&1 | tail -3 (if ansible present) Expected: syntax OK (or documented skip when ansible absent).

• [ ] Step 3: Live smoke checklist (operator, on sovrn.at)

curl -s https://sovrn.at/.well-known/did.json  # id=did:web:sovrn.at, service id #svc_mail, endpoint https://sovrn.at
curl -s https://sovrn.at/.well-known/openid-configuration  # issuer=https://sovrn.at
curl -s https://sovrn.at/oauth/client-metadata  # client_id=https://sovrn.at/oauth/client-metadata
# login → add domain (custom) → DNS shows mx1.sovrn.at → add account → app password → IMAP LOGIN

• [ ] Step 4: Commit

jj commit -m "feat(deploy): wire single-server roles into site.yml" playbooks/site.yml

Self-review

• Spec coverage: fragment DID + bare-host endpoint (T3), no-secret static did.json via Ansible file + Caddy serve (T3-T4), single-server values incl. corrected oauth paths (T1/T5), single FDB + empty backup with R2 var reserved (T6), appview/OIDC/OAuth relationship on one origin (T4-T5). Gaps: none — packaging/systemd unit body, TLS cert issuance (Caddy automatic), and MX/DKIM DNS records are operator-side and listed in T7 smoke.
• Placeholder scan: no TBD/TODO; all code blocks complete; exact commands + expected outputs present.
• Type consistency: var names identical across defaults/tasks/templates/playbook; sovrn_service_did always full did:web:sovrn.at#svc_mail, bare DID derived via split('#')[0] in one place.

S5 single-server smoke implemented (sovrn.at): 7 commits — non-secret vars, single-host inventory, didwellknown (static did.json, did:web:sovrn.at#svc_mail, bare-host endpoint, no secret), caddy single-vhost (did.json file_server ahead of reverse_proxy), goapp sovrn.toml (corrected oauth paths), foundationdb single-instance (empty backup_target, R2 var reserved), site.yml wiring. Verified: go tests green (incl. svcverifier), YAML sweep clean, ansible syntax-check OK. Operator smoke checklist: curl did.json / openid-configuration / oauth client-metadata, then login-to-IMAP loop.

Ansible dirs consolidated under deployment/ (inventory, group_vars, host_vars, roles, playbooks) with deployment/ansible.cfg (inventory + roles_path) so operators run from deployment/. docs/deployment.md layout/commands updated; in-file path refs fixed. Verified: YAML sweep clean, syntax-check OK from deployment/, go tests green.

Renamed Ansible role goapp -> sovrnd to match the binary (role dir, site.yml include, deployment.md tree). No stragglers; syntax-check OK.

S5 packaging + systemd Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Every process installed from a package/binary drop with a systemd unit; no role assumes its software is already on the host.

Architecture: New common role owns users/dirs/prereqs and fails fast on non-amd64 (FoundationDB publishes amd64-only .debs). FDB installs from pinned GitHub-release .debs; Stalwart ships as the repo’s own Nix-built FDB-enabled binary copied by Ansible with an ldd-gated rpath fix for libfdb_c; sovrnd is controller-built with the local Go toolchain and copied; Caddy comes from Cloudsmith (2.11.x). Stalwart first-boot bootstrap (the Bootstrap/set JMAP sequence in nix/scripts/serve-stalwart.sh) is ported to uri-module tasks gated on mode==bootstrap plus a host sentinel.

Tech Stack: Ansible (apt, systemd, uri modules), Debian trixie amd64, FoundationDB 7.3.x, Stalwart 0.16.19 (Nix, foundationdb feature), Go 1.26, Caddy 2.11.x (Cloudsmith).

Locked decisions: amd64 server; Nix-built Stalwart copied; controller-built sovrnd + sqlite3 package on host; Cloudsmith Caddy (Debian tracker marks trixie 2.6.2-12 vulnerable to 2026 CVEs incl. a 9.1 FastCGI RCE with no trixie-security backport shown — none hit our minimal Caddyfile directly, but the TLS edge stays current); bootstrap ported to Ansible, mode==bootstrap only, sentinel-gated.

Task 1: common role + shared vars

Files: - Create: deployment/roles/common/tasks/main.yml - Modify: deployment/group_vars/all/sovrn.yml (append block below) - Modify: deployment/playbooks/site.yml (common first)

• [ ] Step 1: Append shared packaging vars to deployment/group_vars/all/sovrn.yml

# Packaging (bug 70790ed S5): amd64-only (FoundationDB publishes amd64 .debs;
# arm64 exists only as GitHub tarballs, unit-tested, no packages).
sovrn_mail_domain: "sovrn.at"
sovrn_fdb_version: "7.3.68"
sovrn_stalwart_bin: "/usr/local/sbin/stalwart"
sovrn_stalwart_config: "/var/lib/stalwart/config.json"
sovrn_stalwart_datadir: "/var/lib/stalwart"
sovrn_stalwart_logdir: "/var/log/stalwart"
sovrn_stalwart_envfile: "/etc/sovrn/stalwart.env"
sovrn_stalwart_sentinel: "/var/lib/stalwart/.bootstrapped"
sovrn_sovrnd_bin: "/usr/local/sbin/sovrnd"

(Confirm sovrn_fdb_version against github.com/apple/foundationdb/releases published .deb assets before first deploy; the role’s dpkg --info assert catches metadata regressions.)

• [ ] Step 2: Create deployment/roles/common/tasks/main.yml

- name: Assert amd64 (FoundationDB .debs are amd64-only)
  ansible.builtin.assert:
    that:
      - ansible_architecture == "x86_64"
    fail_msg: >-
      Host is {{ ansible_architecture }}: FoundationDB publishes amd64-only
      .debs (arm64 exists only as unit-tested GitHub tarballs, no packages).
      Provision an amd64 Debian host.      

- name: Create sovrn group and user
  ansible.builtin.group:
    name: "{{ sovrn_user }}"
    system: true
  ansible.builtin.user:
    name: "{{ sovrn_user }}"
    group: "{{ sovrn_user }}"
    system: true
    shell: /usr/sbin/nologin
    create_home: false

- name: Install base packages
  ansible.builtin.apt:
    name: [ca-certificates, curl, gnupg, jq, ufw]
    state: present
    update_cache: true

- name: Create base directories
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: directory
    owner: "{{ item.owner }}"
    group: "{{ item.group }}"
    mode: "{{ item.mode }}"
  loop:
    - {path: /etc/sovrn, owner: root, group: "{{ sovrn_user }}", mode: "0750"}
    - {path: "{{ sovrn_datadir }}", owner: "{{ sovrn_user }}", group: "{{ sovrn_user }}", mode: "0750"}
    - {path: "{{ sovrn_wellknown_dir }}", owner: root, group: "{{ sovrn_user }}", mode: "0755"}

- name: Allow mail/web/ssh through the firewall
  ansible.builtin.ufw:
    rule: allow
    port: "{{ item }}"
    proto: tcp
  loop: ["22", "25", "80", "443", "143", "465", "587", "993"]

• [ ] Step 3: Put common first in deployment/playbooks/site.yml (insert before sovrn_secrets include):

    - name: Base users, dirs, firewall
      ansible.builtin.include_role:
        name: common

• [ ] Step 4: Validate + commit

Run: python3 -c "import yaml; yaml.safe_load(open('deployment/roles/common/tasks/main.yml')); print('YAML OK')" (from repo root) Expected: YAML OK

jj commit -m "feat(deploy): common role (amd64 assert, sovrn user, dirs, firewall)"

Task 2: FoundationDB install slice

Files: - Modify: deployment/roles/foundationdb/tasks/main.yml (full rewrite below; keeps existing configure + smoke assert)

• [ ] Step 1: Rewrite deployment/roles/foundationdb/tasks/main.yml

- name: Download pinned FDB .debs (clients first: server depends on it)
  ansible.builtin.get_url:
    url: "https://github.com/apple/foundationdb/releases/download/{{ sovrn_fdb_version }}/foundationdb-{{ item }}_{{ sovrn_fdb_version }}-1_amd64.deb"
    dest: "/root/foundationdb-{{ item }}_{{ sovrn_fdb_version }}-1_amd64.deb"
    mode: "0644"
  loop: [clients, server]

- name: Assert .deb architecture metadata is intact
  ansible.builtin.command: "dpkg --info /root/foundationdb-{{ item }}_{{ sovrn_fdb_version }}-1_amd64.deb"
  register: fdb_deb_info
  changed_when: false
  failed_when: "'Architecture: amd64' not in fdb_deb_info.stdout"
  loop: [clients, server]

- name: Install FDB .debs
  ansible.builtin.apt:
    deb: "/root/foundationdb-{{ item }}_{{ sovrn_fdb_version }}-1_amd64.deb"
  loop: [clients, server]

- name: Single-instance foundationdb.conf
  ansible.builtin.template:
    src: foundationdb.conf.j2
    dest: /etc/foundationdb/foundationdb.conf
    owner: root
    group: root
    mode: "0644"
  notify: Restart foundationdb

- name: Enable and start fdbmonitor
  ansible.builtin.systemd:
    name: foundationdb
    enabled: true
    state: started

- name: Configure single-instance FDB (once; cluster file guards reruns)
  ansible.builtin.command: fdbcli --exec "configure new single ssd"
  args:
    creates: "{{ sovrn_fdb_cluster_file }}"
  when: sovrn_fdb_mode == "single"

- name: Assert no backup target for ephemeral smoke
  ansible.builtin.assert:
    that:
      - sovrn_backup_target == ""
    fail_msg: "Smoke host is ephemeral: leave sovrn_backup_target empty. Prod uses the rclone R2 remote."
    when: inventory_hostname == "sovrn.at"

• [ ] Step 2: Create deployment/roles/foundationdb/templates/foundationdb.conf.j2

[fdbmonitor]
user = foundationdb
group = foundationdb

[general]
restart-delay = 60
cluster-file = {{ sovrn_fdb_cluster_file }}

[fdbserver]
command = /usr/sbin/fdbserver
public-address = 127.0.0.1:$ID
listen-address = 127.0.0.1:$ID
datadir = /var/lib/foundationdb/data/$ID
logdir = /var/log/foundationdb
loggroup = default-cluster
locality-machineid = $MACHINE_ID
locality-zoneid = $MACHINE_ID
class =stateless

[fdbserver.4500]

• [ ] Step 3: Create deployment/roles/foundationdb/handlers/main.yml

- name: Restart foundationdb
  ansible.builtin.systemd:
    name: foundationdb
    state: restarted

• [ ] Step 4: Validate + commit

Run: python3 -c "import yaml; yaml.safe_load(open('deployment/roles/foundationdb/tasks/main.yml')); print('YAML OK')" Expected: YAML OK

jj commit -m "feat(deploy): foundationdb install from pinned .debs + single-instance conf"

Task 3: Stalwart install slice (Nix binary + unit + env)

Files: - Create: deployment/roles/stalwart/tasks/install.yml - Create: deployment/roles/stalwart/templates/stalwart.service.j2 - Create: deployment/roles/stalwart/templates/stalwart.env.j2 - Modify: deployment/roles/stalwart/tasks/secret.yml (no change needed — env file renders from the same vault var) - Modify: deployment/playbooks/site.yml (include install.yml after secret.yml)

• [ ] Step 1: Create deployment/roles/stalwart/templates/stalwart.env.j2

CONFIG_PATH={{ sovrn_stalwart_config }}
STALWART_RECOVERY_ADMIN={{ sovrn_stalwart_user }}:{{ sovrn_stalwart_recovery_password }}

• [ ] Step 2: Create deployment/roles/stalwart/templates/stalwart.service.j2

[Unit]
Description=Stalwart mail server (sovrn)
After=network.target foundationdb.service
Wants=foundationdb.service

[Service]
Type=simple
User=stalwart
Group=stalwart
AmbientCapabilities=CAP_NET_BIND_SERVICE
EnvironmentFile={{ sovrn_stalwart_envfile }}
ExecStart={{ sovrn_stalwart_bin }}
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

• [ ] Step 3: Create deployment/roles/stalwart/tasks/install.yml

- name: Create stalwart user
  ansible.builtin.user:
    name: stalwart
    group: stalwart
    system: true
    shell: /usr/sbin/nologin
    create_home: false
  ansible.builtin.group:
    name: stalwart
    system: true

- name: Create stalwart data and log dirs
  ansible.builtin.file:
    path: "{{ item }}"
    state: directory
    owner: stalwart
    group: stalwart
    mode: "0750"
  loop: ["{{ sovrn_stalwart_datadir }}", "{{ sovrn_stalwart_logdir }}"]

- name: Build Stalwart FDB binary on the controller
  ansible.builtin.command: nix build .#stalwart --print-out-paths --no-link
  args:
    chdir: "{{ playbook_dir }}/../.."
  delegate_to: localhost
  become: false
  run_once: true
  register: stalwart_store_path
  changed_when: false

- name: Copy Stalwart binary to the host
  ansible.builtin.copy:
    src: "{{ stalwart_store_path.stdout_lines[0] }}/bin/stalwart"
    dest: "{{ sovrn_stalwart_bin }}"
    owner: root
    group: root
    mode: "0755"
    remote_src: true

- name: Point binary rpath at system libfdb_c (clients .deb provides /usr/lib)
  ansible.builtin.command: "patchelf --set-rpath /usr/lib {{ sovrn_stalwart_bin }}"
  args:
    creates: /usr/local/.stalwart-rpath-fixed
  notify: Mark rpath fixed

- name: Assert libfdb_c resolves
  ansible.builtin.command: "ldd {{ sovrn_stalwart_bin }}"
  register: stalwart_ldd
  changed_when: false
  failed_when: "'libfdb_c' not in stalwart_ldd.stdout or 'not found' in stalwart_ldd.stdout"

- name: Install stalwart env file (0600-equivalent: root:stalwart 0640)
  ansible.builtin.template:
    src: stalwart.env.j2
    dest: "{{ sovrn_stalwart_envfile }}"
    owner: root
    group: stalwart
    mode: "0640"
  no_log: true

- name: Install stalwart.service
  ansible.builtin.template:
    src: stalwart.service.j2
    dest: /etc/systemd/system/stalwart.service
    owner: root
    group: root
    mode: "0644"
  notify: Reload systemd and restart stalwart

- name: Enable stalwart (starts in bootstrap mode when config is absent)
  ansible.builtin.systemd:
    name: stalwart
    enabled: true
    state: started
    daemon_reload: true

• [ ] Step 4: Handlers — append to deployment/roles/stalwart/handlers/main.yml (create file):

- name: Mark rpath fixed
  ansible.builtin.file:
    path: /usr/local/.stalwart-rpath-fixed
    state: touch

- name: Reload systemd and restart stalwart
  ansible.builtin.systemd:
    name: stalwart
    state: restarted
    daemon_reload: true

• [ ] Step 5: Include in deployment/playbooks/site.yml after the secret.yml include:

    - name: Install Stalwart binary, unit, env
      ansible.builtin.include_role:
        name: stalwart
        tasks_from: install.yml

• [ ] Step 6: Validate + commit

Run: python3 -c "import yaml; d=yaml.safe_load(open('deployment/roles/stalwart/tasks/install.yml')); print('YAML OK')" Expected: YAML OK

jj commit -m "feat(deploy): stalwart install (nix binary, unit, env, rpath gate)"

Task 4: Stalwart bootstrap port (mode==bootstrap, sentinel-gated)

Files: - Create: deployment/roles/stalwart/tasks/bootstrap.yml - Modify: deployment/playbooks/site.yml (include after install.yml)

Port of nix/scripts/serve-stalwart.sh FDB path, production differences: privileged mail ports (systemd + CAP_NET_BIND_SERVICE, no ¹¹⁴³⁄₁₅₈₇), allowPlainTextAuth stays default false (TLS enforced), tracer to {{ sovrn_stalwart_logdir }}, requestTlsCertificate: false (Caddy terminates), generateDkimKeys: false (control plane provisions DKIM).

• [ ] Step 1: Create deployment/roles/stalwart/tasks/bootstrap.yml

- name: Skip bootstrap unless mode==bootstrap and sentinel absent
  ansible.builtin.stat:
    path: "{{ sovrn_stalwart_sentinel }}"
  register: stalwart_sentinel

- name: Bootstrap block
  when:
    - mode | default('update') == 'bootstrap'
    - not stalwart_sentinel.stat.exists
  block:
    - name: Wait for bootstrap HTTP (missing config => bootstrap mode)
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/healthz/live
        status_code: 200
      register: stalwart_live
      retries: 60
      delay: 1
      until: stalwart_live.status == 200

    - name: Fetch Bootstrap singleton id
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/jmap
        method: POST
        user: "{{ sovrn_stalwart_user }}"
        password: "{{ sovrn_stalwart_recovery_password }}"
        force_basic_auth: true
        body_format: json
        body:
          using: ["urn:stalwart:jmap"]
          methodCalls: [["x:Bootstrap/get", {accountId: a}, "c0"]]
        status_code: 200
      register: bootstrap_get
      no_log: true

    - name: Apply sovrn bootstrap patch (FDB store, mx1.sovrn.at)
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/jmap
        method: POST
        user: "{{ sovrn_stalwart_user }}"
        password: "{{ sovrn_stalwart_recovery_password }}"
        force_basic_auth: true
        body_format: json
        body:
          using: ["urn:stalwart:jmap"]
          methodCalls:
            - - "x:Bootstrap/set"
              - accountId: a
                update:
                  "{{ bootstrap_get.json.methodResponses[0][1].list[0].id }}":
                    serverHostname: "{{ sovrn_mail_hostname }}"
                    defaultDomain: "{{ sovrn_mail_domain }}"
                    requestTlsCertificate: false
                    generateDkimKeys: false
                    dataStore: {"@type": FoundationDb, clusterFile: "{{ sovrn_fdb_cluster_file }}"}
                    tracer: {"@type": Log, path: "{{ sovrn_stalwart_logdir }}/", prefix: stalwart, ansi: false, enable: true}
              - "c1"
        status_code: 200
      register: bootstrap_set
      no_log: true

    - name: Assert bootstrap update accepted
      ansible.builtin.assert:
        that:
          - bootstrap_set.json.notUpdated is not defined or (bootstrap_set.json.notUpdated | length == 0)
        fail_msg: "Bootstrap update rejected: {{ bootstrap_set.json.notUpdated }}"

    - name: Record generated admin credentials (0600, emergency console use)
      ansible.builtin.copy:
        content: "{{ (bootstrap_set.json.updated | dict2items)[0].value.username }}:{{ (bootstrap_set.json.updated | dict2items)[0].value.secret }}\n"
        dest: /etc/sovrn/stalwart-admin-credentials
        owner: root
        group: stalwart
        mode: "0640"
      no_log: true

    - name: Wait for config file to be written
      ansible.builtin.wait_for:
        path: "{{ sovrn_stalwart_config }}"
        timeout: 30

    - name: Create production mail listeners (privileged ports via CAP_NET_BIND_SERVICE)
      ansible.builtin.uri:
        url: http://127.0.0.1:8080/jmap
        method: POST
        user: "{{ sovrn_stalwart_user }}"
        password: "{{ sovrn_stalwart_recovery_password }}"
        force_basic_auth: true
        body_format: json
        body:
          using: ["urn:stalwart:jmap"]
          methodCalls:
            - - "x:NetworkListener/set"
              - accountId: a
                create:
                  n0: {name: "{{ item.name }}", protocol: "{{ item.protocol }}", bind: {"[::]:{{ item.port }}": true}, useTls: "{{ item.usetls }}", tlsImplicit: "{{ item.implicit }}"}
              - "c0"
        status_code: 200
      loop:
        - {name: smtp, protocol: smtp, port: 25, usetls: false, implicit: false}
        - {name: smtps, protocol: smtp, port: 465, usetls: true, implicit: true}
        - {name: submission, protocol: smtp, port: 587, usetls: true, implicit: false}
        - {name: imap, protocol: imap, port: 143, usetls: true, implicit: false}
        - {name: imaps, protocol: imap, port: 993, usetls: true, implicit: true}
      no_log: true

    - name: Restart stalwart into normal mode
      ansible.builtin.systemd:
        name: stalwart
        state: restarted

    - name: Write bootstrap sentinel
      ansible.builtin.file:
        path: "{{ sovrn_stalwart_sentinel }}"
        state: touch
        owner: root
        group: stalwart
        mode: "0640"

- name: Assert config exists outside bootstrap mode
  ansible.builtin.stat:
    path: "{{ sovrn_stalwart_config }}"
  register: stalwart_config
  failed_when: (mode | default('update')) != 'bootstrap' and not stalwart_config.stat.exists

• [ ] Step 2: Include in deployment/playbooks/site.yml after install.yml include:

    - name: Bootstrap Stalwart on first boot
      ansible.builtin.include_role:
        name: stalwart
        tasks_from: bootstrap.yml

• [ ] Step 3: Validate + commit

Run: python3 -c "import yaml; yaml.safe_load(open('deployment/roles/stalwart/tasks/bootstrap.yml')); print('YAML OK')" Expected: YAML OK

jj commit -m "feat(deploy): stalwart bootstrap port (uri tasks, sentinel-gated)"

Task 5: Caddy install (Cloudsmith 2.11.x) + sovrnd build/install + rewire

Files: - Modify: deployment/roles/caddy/tasks/main.yml (prepend install block) - Modify: deployment/roles/sovrnd/tasks/main.yml (prepend build/install block; keep datadir + sovrn.toml) - Create: deployment/roles/sovrnd/templates/sovrnd.service.j2 - Create: deployment/roles/sovrnd/handlers/main.yml (move/keep Restart sovrnd; add daemon reload via listen) - Modify: deployment/playbooks/site.yml (final order)

• [ ] Step 1: Prepend Caddy Cloudsmith install to deployment/roles/caddy/tasks/main.yml

- name: Install Cloudsmith keyring for Caddy stable
  ansible.builtin.get_url:
    url: https://dl.cloudsmith.io/public/caddy/stable/gpg.key
    dest: /usr/share/keyrings/caddy-stable-archive-keyring.gpg
    mode: "0644"

- name: Add Caddy stable apt source
  ansible.builtin.apt_repository:
    repo: "deb [signed-by=/usr/share/keyrings/caddy-stable-archive-keyring.gpg] https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version main"
    filename: caddy-stable
    state: present

- name: Install Caddy 2.11.x (current; trixie ships fixed 2.6.2 with 2026 CVE exposure)
  ansible.builtin.apt:
    name: caddy
    state: present
    update_cache: true

(Existing Caddyfile/enable tasks stay below unchanged.)

• [ ] Step 2: Create deployment/roles/sovrnd/templates/sovrnd.service.j2

[Unit]
Description=sovrn control plane (sovrnd)
After=network.target stalwart.service foundationdb.service
Wants=stalwart.service foundationdb.service

[Service]
Type=simple
User={{ sovrn_user }}
Group={{ sovrn_user }}
WorkingDirectory=/etc/sovrn
ExecStart={{ sovrn_sovrnd_bin }}
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

(WorkingDirectory=/etc/sovrn because LoadConfig reads ./sovrn.toml from cwd; the existing template already renders there.)

• [ ] Step 3: Prepend controller build + install + sqlite3 to deployment/roles/sovrnd/tasks/main.yml

- name: Install sqlite3 CLI (app-DB inspection and debugging)
  ansible.builtin.apt:
    name: sqlite3
    state: present

- name: Build sovrnd on the controller (native amd64 toolchain)
  ansible.builtin.command: go build -trimpath -o /tmp/sovrnd-prod ./cmd/sovrnd
  args:
    chdir: "{{ playbook_dir }}/../.."
  delegate_to: localhost
  become: false
  run_once: true
  changed_when: false

- name: Copy sovrnd binary to the host
  ansible.builtin.copy:
    src: /tmp/sovrnd-prod
    dest: "{{ sovrn_sovrnd_bin }}"
    owner: root
    group: root
    mode: "0755"

- name: Install sovrnd.service
  ansible.builtin.template:
    src: sovrnd.service.j2
    dest: /etc/systemd/system/sovrnd.service
    owner: root
    group: root
    mode: "0644"
  notify: Reload systemd and restart sovrnd

- name: Enable and start sovrnd
  ansible.builtin.systemd:
    name: sovrnd
    enabled: true
    state: started
    daemon_reload: true

(Existing datadir + sovrn.toml render tasks stay below unchanged.)

• [ ] Step 4: Final deployment/playbooks/site.yml order (full file):

# Top-level entry point (docs/deployment.md). Select behavior with
# `-e mode=bootstrap|recover|update` (default: update).
# Order: base -> secrets -> data plane -> control plane -> edge.
- name: Provision sovrn hosts
  hosts: all
  become: true
  tasks:
    - name: Base users, dirs, firewall
      ansible.builtin.include_role:
        name: common

    - name: Install shared sovrn secrets
      ansible.builtin.include_role:
        name: sovrn_secrets

    - name: Install per-host Stalwart credential
      ansible.builtin.include_role:
        name: stalwart
        tasks_from: secret.yml

    - name: Configure single-instance FoundationDB
      ansible.builtin.include_role:
        name: foundationdb

    - name: Install Stalwart binary, unit, env
      ansible.builtin.include_role:
        name: stalwart
        tasks_from: install.yml

    - name: Bootstrap Stalwart on first boot
      ansible.builtin.include_role:
        name: stalwart
        tasks_from: bootstrap.yml

    - name: Render static did.json
      ansible.builtin.include_role:
        name: didwellknown

    - name: Build, install and configure sovrnd
      ansible.builtin.include_role:
        name: sovrnd

    - name: Install Caddy single vhost
      ansible.builtin.include_role:
        name: caddy

• [ ] Step 5: Validate + commit

Run: python3 -c "import yaml; [yaml.safe_load(open(f)) for f in ['deployment/roles/caddy/tasks/main.yml','deployment/roles/sovrnd/tasks/main.yml','deployment/playbooks/site.yml']]; print('YAML OK')" Expected: YAML OK

jj commit -m "feat(deploy): caddy cloudsmith install, sovrnd build+unit, final ordering"

Task 6: Full verification

• [ ] Step 1: YAML sweep + syntax

Run: for f in $(jj file list 'deployment/**/*.yml' | grep -v templates); do python3 -c "import yaml,sys; yaml.safe_load(open('$f'))" || echo "BAD: $f"; done; echo YAML-SWEEP-DONE Expected: YAML-SWEEP-DONE, no BAD lines.

Run: ansible-playbook playbooks/site.yml --syntax-check (from deployment/) Expected: playbook: playbooks/site.yml, no error.

• [ ] Step 2: Controller build gates

Run: go build -trimpath -o /tmp/sovrnd-verify ./cmd/sovrnd && echo SOVRND-BUILD-OK Expected: SOVRND-BUILD-OK

Run: SOVRN_INTEGRATION=0 go test ./... 2>&1 | tail -3 Expected: all packages ok.

• [ ] Step 3: Operator deploy checklist (on sovrn.at, mode=bootstrap)

ansible-playbook playbooks/site.yml -e mode=bootstrap   # from deployment/
# assert: fdb 7.3.x installed; ldd /usr/local/sbin/stalwart resolves libfdb_c;
# sentinel /var/lib/stalwart/.bootstrapped written; admin creds at
# /etc/sovrn/stalwart-admin-credentials (0600); caddy 2.11.x; sovrnd active;
# curl https://sovrn.at/.well-known/did.json -> #svc_mail, bare-host endpoint.

Self-review

• Spec coverage: amd64 + arch assert (T1); FDB pinned .debs + conf + unit via package (T2); Nix Stalwart binary + rpath/ldd gate + env + unit (T3); bootstrap port incl. prod listeners, sentinel, recover/update assert (T4); Cloudsmith Caddy 2.11 (T5); controller sovrnd build + sqlite3 + unit (T5); ordering + verification (T5-T6). Gaps: none for smoke — dns role, keepalived/VIP, and R2 backup remain explicitly deferred.
• Placeholder scan: no TBD/TODO; every step has exact code/commands/expected output. One operator-confirmed value (sovrn_fdb_version vs published assets) is a documented verify step, not a placeholder.
• Type consistency: sovrn_* names identical across vars/tasks/templates/playbook; secret.yml vault var sovrn_stalwart_recovery_password reused in env template + bootstrap auth; sovrn_fdb_cluster_file default reused in bootstrap datastore JSON.

S5 packaging+systemd landed (5 commits): common role (amd64 assert, sovrn user, dirs, ufw mail/web/ssh), FDB pinned-GitHub-.deb install + arch-metadata assert + single-instance conf, Stalwart Nix-binary copy with patchelf rpath + ldd libfdb_c gate + env/unit + uri-ported bootstrap (prod listeners, sentinel, generated-admin 0600 record), Cloudsmith Caddy 2.11.x (trixie 2.6.2 carries 2026 CVE exposure incl. un-backported FastCGI RCE), sovrnd controller build + sqlite3 + unit (WorkingDirectory=/etc/sovrn), final site.yml order base->secrets->dataplane->control->edge. Verified: YAML sweep clean, ansible syntax OK, controller go build OK, full go test suite green. Review fixes during implementation: split dual-module tasks, controller-vs-host copy direction, JSON-string body for real booleans (render-checked). Deploy-time operator confirms: FDB 7.3.68 .deb assets published, then -e mode=bootstrap.

Secrets ceremony scripted: ansible.cfg vault convention (prompt every run, KeePass outside repo), stalwart bootstrap fetch-back to gitignored deployment/.local/, Just recipes provision-shared-secrets (gensecrets->render->encrypt, refuses overwrite) and persist-bootstrap-secrets (validate 32-hex, render host vault, encrypt, shred fetched copy), reminder message points at recipe, runbook in docs/deployment.md. Verified: both –dry-runs green (real keygen), vault encrypt/decrypt round-trip OK, YAML sweep clean, ansible syntax OK, go tests green. Remaining operator steps: run the two recipes, bootstrap, review, commit.

Added just bootstrap/update (default target sovrn.at) with prompt-free prechecks: ansible present, fleet vault encrypted (bootstrap) + host vault encrypted (update), target parsed from inventory/hosts.yml via python (ansible-inventory needs the vault password, so it cannot run before the prompt), non-interactive ssh is warn-only. Verified: bogus host fails closed, sovrn.at shared passes, update correctly demands bootstrap+persist first, syntax-check OK. Also found the shared vault already encrypted in-repo. Runbook steps now reference the recipes.

Fixed bootstrap failure: group_vars/hostvars were split from the inventory file, so Ansible never loaded them and every sovrn* var was undefined. Moved both inside deployment/inventory/ (the only layout Ansible auto-loads, documented in deployment.md). Also fixed ansible_architecture deprecation (ansible_facts[architecture]). Verified: scratch-inventory proof that adjacency loads group+host vars, syntax-check OK, prechecks re-verified against new paths, go tests green. Note: committed vault-shared.yml carries stale in-file path comments (ciphertext, refreshes on next rotation). Ready to re-run just bootstrap sovrn.at.

Fixed both bootstrap issues: (1) fetch+reminder moved from secret.yml to end of the bootstrap block — the old message claimed a bootstrapped credential before Stalwart was even installed; it now fires only after Bootstrap/set + sentinel, and the .local path mismatch (playbooks/.local vs deployment/.local) is fixed. (2) Nix build out of the playbook into just build-stalwart (with the missing –extra-experimental-features flags, out-link GC root at deployment/.local/stalwart-prod); install.yml asserts + copies instead, and the just prechecks fail fast when the binary is missing. go build stays in-Ansible (1 min, not worth the ceremony). Verified: YAML sweep, syntax-check, precheck gate message, nix eval resolves stalwart-0.16.19. Operator: run just build-stalwart now (up to an hour), then just bootstrap.

Ansible speedups: pipelining + ControlPersist multiplex + 24h jsonfile fact cache + deprecation silence in ansible.cfg; per-slice tags on all site.yml includes with ARGS passthrough on just bootstrap/update (e.g. –tags sovrnd,caddy, –start-at-task); apt cache_valid_time hourly; sovrnd controller build skipped when no .go newer than last binary (both branches logic-tested). copy tasks stay checksum-guarded. Documented retry shortcuts + Mitogen as next lever. Verified: –list-tags, syntax, prechecks, tests.

Fixed stalwart 203/EXEC: Nix binary pointed PT_INTERP at /nix/store (absent on Debian); host ldd passed because ldd uses the host loader, masking it. Diagnosed live via ssh (journal: No such file or directory on exec). Fix: patchelf –set-interpreter /lib64/ld-linux-x86-64.so.2 alongside the rpath fix, marker bumped to -v2 so affected hosts re-run it (copy checksum forces a fresh binary first), plus a patchelf –print-interpreter assert. Verified compatible: binary needs max GLIBC_2.39 (host 2.41) and GLIBCXX_3.4.30 (trixie gcc14 provides 3.4.34); just build-stalwart now gates both ceilings locally. Re-run just bootstrap sovrn.at (update mode would trip the config-exists assert since bootstrap never completed).

Fixed bootstrap 401: the recovery password is regenerated every bootstrap run, but the running process kept the previous run’s env — state=started never restarts an active/crash-looping unit and handler restarts flush only at play end, after bootstrap already authenticated. Added an unconditional daemon-reload restart as the first step of the bootstrap block so process and env file always agree. Re-run just bootstrap sovrn.at (clean retry: no config or sentinel was written, the failed run stopped at Bootstrap/get).

Fixed bootstrap undefined-var: sovrn_fdb_cluster_file lived in foundationdb/defaults, which only loads when that role executes — tag-skipped or reordered runs break stalwart bootstrap. Moved all three FDB topology vars to groupvars (always loaded) with a comment explaining why, deleted the role defaults file. Full audit: every sovrn* reference now resolves via group_vars or vault files, no cross-role defaults remain. Verified YAML sweep + syntax-check. Safe to re-run (full bootstrap or –tags stalwart-bootstrap).

Fixed bootstrap assert failure: fail_msg is templated even when the assertion passes, so referencing bootstrap_set.json.notUpdated crashed on success (key absent). Guarded with | default(‘none’). Notably this means the Bootstrap/set itself was ACCEPTED — the patch went through. Re-run just bootstrap sovrn.at to continue past it.

Fixed bootstrap admin-record failure: Bootstrap/set on a re-run returns no updated key (previous run’s set had already applied and created the admin), so the record step now only runs when updated is present and non-empty, with a debug note otherwise. Recovery-admin vault/env path is unaffected. Re-run just bootstrap sovrn.at.

Root-caused the missing config.json live: the accepted-looking Bootstrap/set was a silent no-op (defaults still in place, empty store dir), while an identical manual call applied instantly and wrote a correct FDB-pointer config. Payloads proven byte-equivalent by local render, so the difference was server-side timing/state across the crash-loop-era retries. Reset to a clean slate (deleted config.json, wiped FDB data dir, reconfigured single ssd) and added a read-back task asserting serverHostname/defaultDomain/clusterFile match — future silent no-ops now fail loudly with the actual singleton state. Re-run just bootstrap sovrn.at for a true first-apply (also exercises the record step properly this time).

Replaced the uri-task bootstrap with files/bootstrap-stalwart.sh: static env-driven shell (curl+jq, zero Jinja), Ansible keeps orchestration (restart, copy 0700, run with secret env under no_log, sentinel, fetch). Script covers health-wait, get, set with jq-built payload, admin record, config wait, read-back singleton verification, and the 5 prod listeners. Tested end-to-end against a stub JMAP server: BOOTSTRAP-APPLIED exit 0, real JSON booleans asserted server-side, creds file correct, re-run prints BOOTSTRAP-PRESENT. The stub test caught one real bug pre-deploy (extra brace in listener JSON, transcribed from the deeper set shape). Re-run just bootstrap sovrn.at.

Hardened the credential lifecycle while diagnosing: every bootstrap run regenerated the recovery password, which risks lockout if Stalwart pins the first value DB-side (normal-mode recovery auth verified working with the current value, but rotation was never proven safe). secret.yml now generates only on truly fresh hosts and otherwise reuses the host file via slurp; fetch/reminder gate on needs_persist instead of generated. Next run reuses the known-good credential, creates the 2 missing listeners (25/465/993 already up — partial state from the earlier runs), then sentinel+fetch. Dropped no_log on the script task so stderr is visible on failure.

Implemented wipe-on-bootstrap + listener reconcile + tooling decision. (1) mode=bootstrap now destroys (rm, no move-aside): FDB data+cluster, stalwart datadir/config/sentinel/creds, stale controller fetch; forced reconfigure; update/recover never wipe. Every bootstrap is a true first-apply. (2) Listeners phase reconciles Stalwart’s auto-created defaults: keeps smtp/submissions/imaps/http/sieve, creates submission/587+imap/143, destroys https/443 (Caddy conflict) + pop3s/995, final assert with diff. Sieve stays up but firewalled (4190 absent from ufw allows + code comment). (3) Tooling: Ansible stays thin (files/services only), all logic in stub-tested shell, no pyinfra. Caught pre-deploy via stub: envelope-forgetting jq checks + two extra-brace payloads; shellcheck clean. Re-run just bootstrap sovrn.at.

Fixed cluster-file timeout with live log proof: fdbserver NEVER creates the cluster file — without one it exits 10 and fdbmonitor respawns it every 61s forever (the phantom second fdbmonitor was just a child caught mid-respawn). My dev-container assumption was wrong; there the seed-connection-string did the job. The bootstrap wipe now authors a fresh sovrn:@127.0.0.1:4500 cluster file itself (stable afterwards — only bootstrap writes it), and the stale wait-for-a-file-nobody-creates is gone. Reordered role so install precedes wipe (fresh hosts have no /etc/foundationdb yet). Re-run just bootstrap sovrn.at.

Implemented per plan: (1) central bootstrap stop-all role, dead-first in site.yml (fresh boxes: all four stops are clean no-ops); scattered in-role stops removed. (2) Destroy goes through set+destroy (doc-confirmed RFC 8620, no /destroy method exists). (3) Patchelf creates-marker replaced by copy-changed-conditional + always-on asserts (a Nix rev bump can no longer silently skip the loader fix). (4) Listener reconcile proven against seeded-defaults stub: keep 5, create 587+143, destroy https+pop3s, final assert, converged re-run exit 0; shellcheck + syntax clean. Run just bootstrap sovrn.at for the true first-apply.

Fixed 502: sovrnd binary exists but has Nix PT_INTERP (controller Go is a Nix toolchain + cgo sqlite) — same 203/EXEC class as Stalwart, never caught because the binary was built but never executed. Verified compatible (needs only glibc libs, max GLIBC_2.34 vs host 2.41; sqlite statically linked). Both binaries now share one pattern: fix gated on patchelf –print-interpreter check itself (not copy-changed/markers), so it self-heals already-deployed hosts and survives rev bumps; always-on interpreter + ldd asserts. Just update sovrn.at with no manual steps — the check gate fixes the live binary in place.

Option A implemented: just check-fdb-alignment (nixpkgs client vs pinned .deb, major.minor compare tolerant of nixpkgs suffix noise, fails on skew; wired into build-stalwart), sovrnd libc-ceiling gate in-role on every build (same trixie GLIBC_2.41 pattern), coupling note in deployment.md (also replaced a stale build note). Verified: live gate passes (7.3.⁶⁸⁄₇.3.68), deliberate-skew dry run fails correctly, ceiling logic passes/rejects correctly, syntax + tests green.

Fixed persist rejection: fetched value is valid 32-char hex but mixed-case — Ansible’s password lookup draws from Python hexdigits (0-9a-fA-F), while the persist validator only accepted lowercase. Widened the regex with a comment explaining why; verified via dry-run against the actual fetched file. Deliberately did NOT constrain generation to lowercase instead — that would rotate the live DB-pinned credential. Operator: re-run just persist-bootstrap-secrets sovrn.at (needs your vault password).

Fixed 502: sovrn.toml.j2 nested datadir+env under [server], but the Go struct maps them top-level (viper silently ignores the misplaced keys) — sovrnd ran on default datadir data/dev and died creating it under unwritable /etc/sovrn. This also silently forced env=development (NetProber off). Fixed template + added comment guard; verified by rendering with prod values and asserting top-level placement plus every Validate() requirement via tomllib. Also reordered role so datadir+config land before enable/start (was reversed; restart-loop masked it). Fixed live box with byte-exact template output: sovrnd running, homepage 303 (login redirect), next update is no-change on that task.

Smoke IMAP-SSL root cause + minimal fix (2026-09-11)

Symptom: iPhone to mx1.sovrn.at fails Cannot Connect Using SSL on 993; try without SSL succeeds. Domain test.kilimanjaro.io active, DNS (MX/SPF/DKIM + mx1 A) correct, app-password auth works over plaintext.

Root cause: Stalwart serves IMAP/SMTP TLS itself with useTls=true on all listeners, but bootstrap sets requestTlsCertificate=false (“Caddy terminates”) and no Certificate/AcmeProvider was ever provisioned. Caddy only terminates HTTP for sovrn.at -> sovrnd:8081, so port 993 had no valid cert to present. Plaintext 143 worked, isolating the failure to the TLS layer.

Fix (this change, minimal): Caddyfile.j2 now proxies ONLY /.well-known/acme-challenge/* to Stalwart :8080 (both sovrn.at and {{ sovrn_mail_hostname }} sites; mx1 site 404s everything else), so Stalwart’s own ACME Http01 can issue for mx1.sovrn.at. Management (/jmap, /admin, /healthz) is never proxied — stays localhost-only + firewalled. Next: create staging->prod AcmeProvider + Domain Automatic with explicit SANs=["mx1.sovrn.at"], then re-test iPhone SSL.

Verified: template renders, YAML sweep clean, ansible --syntax-check OK, go test ./... green.

Deferred: single-appview/multi-mail architecture (dedicated imap./smtp. failover names, shared-FDB join vs greenfield bootstrap, sovrnd DB: Litestream vs Postgres vs FDB) tracked in new issue 6ca5959.

ACME in bootstrap (no more one-off) — ready for rebuild test

Stalwart cert provisioning is now first-boot automation, per request.

What changed (uncommitted, 6 files): - roles/stalwart/files/bootstrap-stalwart.sh: new acme phase — ensures AcmeProvider letsencrypt (Http01, LE prod default, contact postmaster@<domain>), sets Domain <mail-domain> to certificateManagement: Automatic with explicit SANs=[mx-host] (update-or-create, re-runs converge), then polls Certificate/get up to 600s for a covering valid cert. Prints CERT-PRESENT / CERT-ISSUED; fails loudly with AcmeRenewal/log pointers on timeout. - roles/stalwart/tasks/bootstrap.yml: runs the acme phase after listeners (staging-directory override documented for test rebuilds). - inventory/group_vars/all/sovrn.yml: sovrn_acme_directory (prod) + sovrn_acme_contact. - playbooks/site.yml: caddy now runs BEFORE stalwart bootstrap — the acme phase validates via the Caddy challenge proxy, so :80 must answer first. sovrnd/didwellknown ordering untouched. - roles/caddy/templates/Caddyfile.j2: mx1 site is now http://-only (no Caddy-managed cert → no challenge-solver conflict); challenge path proxied on both sites, management never proxied. - docs/deployment.md: DNS step names the mx1 A explicitly; bootstrap step documents the ACME phase + staging override.

No clobber risk from sovrnd: internal/domain only writes name/isEnabled/dkimManagement/memberTenantId — never certificateManagement.

Verified: shellcheck clean; stub-JMAP end-to-end (fresh create → CERT-ISSUED, rerun → CERT-PRESENT, pre-existing Manual domain → update → CERT-ISSUED, timeout → exit 1); Jinja renders; YAML sweep clean; ansible --syntax-check OK; go test ./... green.

To test live: blow away smoke host, just build-stalwart if needed, just bootstrap sovrn.at (add -e sovrn_acme_directory=<staging> to dodge LE duplicate-cert limits on repeated rebuilds), then openssl s_client -connect mx1.sovrn.at:993 + iPhone SSL add.

Inbound bounce (Fastmail -> [email protected]) root-caused: Fastmail-side local routing, not our server

Bounce: 550 5.1.1 ... User unknown in virtual mailbox table from mailuser.phl.internal (Fastmail-internal LMTP hop).

Evidence (all checked live): - Public DNS correct: test.kilimanjaro.io MX 10 mx1.sovrn.at, SPF present, mx1.sovrn.at A 2.29.20.216. Parent kilimanjaro.io MX is Fastmail (messagingengine.com). - mx1:25/143/587/993 all bound by stalwart; ufw inactive (no firewall). - Stalwart logs contain ZERO lines for kilimanjaro/messagingengine/fastmail and zero rejects — Fastmail never connected to mx1. - x:Domain/get: test.kilimanjaro.io present and isEnabled: true; account works over IMAP. Our inbound path is provisioned. - Ports 80/443/587/993 reachable remotely; only :25 times out from the dev box (dev-egress filtering, not the server — listener is up).

Conclusion: Fastmail decided the recipient locally (parent domain is Fastmail-hosted, so the subdomain resolves in its own virtual table) and bounced without ever consulting public MX. Nothing to fix on sovrn.at for this incident. Next: re-test from a non-Fastmail sender (Gmail) to prove end-to-end inbound; adjust Fastmail domain/routing settings for the subdomain if Fastmail must be able to send to it.

Unrelated observations: live box still serves no TLS cert (tls.no-certificates-available in today’s log — pending ACME-bootstrap rebuild; harmless for port-25 inbound which falls back to plaintext).

S5 scope amendment (cell architecture, bug 75966cc — locked 2026-09-13)

S1–S4 stand as completed. S5 is rescoped; S6 housekeeping now also covers bugs 6ca5959 + 16df68d (both closed as superseded — see their comments).

S5 old scope (struck): common/stalwart/foundationdb/caddy/goapp/dns roles, Stalwart-FDB packaging, Caddy TLS + MX smoke on shared-FDB active/passive + VIP.

S5 new scope: per-cell roles on SQLite everywhere — common/stalwart(sqlite)/zds/caddy/sovrnd/litestream/rclone/dns (T2/T3/T4), fresh bootstrap + converge green with no FDB artifacts (grep-clean), smoke covering control plane + Stalwart + ZDS behind Caddy with real domain + MX. Cell-model ADR + runbooks land first (T9); floating-IP role follows (T10).

Complete: S1-S5 achieved — real service-auth JWT + authbroker/tokenissuer/appwd wiring, app-view UI skeleton, Ansible roles/playbooks and a live deployment (mx99 cell + infra metrics box), Caddy TLS + MX. S6 (tracker housekeeping) is being executed now via the consolidation into epic 516fcde. Closing.