Stalwart bootstrap as a checked-in JSON apply plan (incl. sovrnd service account) - Issue #b724acb

Replace deployment/roles/stalwart/files/bootstrap-stalwart.sh (482 lines of curl/jq JMAP calls) with a checked-in, auditable JSON plan that stalwart-cli apply consumes. It encodes every step needed to bootstrap a Stalwart cell, including the sovrnd service account.

Source material

Start from scratch, or from stalwart-cli snapshot of mx99.eu.sovrn.at. mx99 is a smoke-test box only, so the snapshot is just a starting point for further edits, not a source of truth. Strip IDs, defaults and anything environment-specific.
Use stalwart-cli describe <Object> against 0.16.23 to confirm property names and shapes.

Format (aim: readable in code review)

The plan source lives in the repo as pretty-printed JSON, e.g. deployment/stalwart/cell.plan.json and relay.plan.json, or split per concern (listeners.json, acme.json, accounts.json, …). One operation per array element, with a "_comment" or adjacent README explaining each block.
A build step compacts it to NDJSON (jq -c '.[]'), because apply reads one operation per line.
Per-host values (hostname, mail domain, ACME directory/contact) are placeholders filled from Nix options. Secrets are never in the file or the Nix store: fill them at runtime in the converge oneshot, e.g. jq --rawfile from the Colmena key file.
Validate with stalwart-cli apply --dry-run as a flake check, so malformed plans fail in CI.

What the plan must encode (current script behavior)

Tracer: Stdout (journal), so Vector ships _SYSTEMD_UNIT=stalwart.service.
Bootstrap/SystemSettings: default domain, hostname, plaintext auth stays off, no DKIM generation (only the relay DKIM-signs), no bootstrap TLS cert request.
Listeners: reconcile with scope.
- Cell: smtp/25, submissions/465, submission/587, imap/143, imaps/993.
- Relay: :25 only.
- Keep sieve/4190 and http/8080; drop https/443 and pop3s/995.
Metrics: Prometheus exporter enabled. Needs ReloadSettings afterwards (verified live: without it the endpoint stays dark), either as a plan op or a follow-up call.
ACME: AcmeProvider (Http01, directory as the natural key; it has no name property), and domain certificateManagement = Automatic with explicit SANs [hostname] (an empty list orders autodiscover/mta-sts names and the order fails).
sovrnd service account (new): a dedicated account with full admin permissions + impersonate, so sovrnd stops authenticating as the recovery admin. Confirm the 0.16 permission/role names (e.g. an admin role + the impersonate permission) via describe. The password comes from a Colmena key at runtime. Update sovrn.toml [stalwart] username/secretfile to match.

Wire-shape gotchas already learned (keep in the README)

Map fields (contact, subjectAlternativeNames) are sets on the wire ({"value": true}), not arrays.
In bootstrap mode, Bootstrap/get serves stale compiled defaults after a successful set.
Prefer upsert/reconcile with explicit matchOn everywhere, so re-applies converge instead of duplicating objects.

Done when applying the plan twice to a fresh Stalwart gives the same state as the old script (listeners, tracer, metrics, cert issued, sovrnd can provision a domain + account via its service account), the second apply reports no changes, and bootstrap-stalwart.sh can be deleted.

2 Comments

agent b376234 Sep 30

From 9f24376: the plan must also pin the admin web UI. Stalwart’s registry Application object defaults to resourceUrl = https://github.com/stalwartlabs/webui/releases/latest/download/webui.zip, fetched on every start: unpinned, and it blocks startup up to 60s offline. Upsert it with resourceUrl = config.sovrn.services.stalwart.webuiUrl (file:// into the nixpkgs-pinned webui bundle) and disable autoUpdateFrequency. Confirm the natural key and field shapes with stalwart-cli describe Application.

agent bf70274 Sep 30

Done (pending review)

Started from stalwart-cli snapshot of mx99 (taken through an SSH tunnel to its loopback :8080; nothing installed on the box). The snapshot was reviewed and cut down to sovrn’s own settings. Stalwart’s built-in defaults (spam data, ~40 settings singletons) and ACME state (the issued Certificate) are left out.

Plans (nix/stalwart/, JSON arrays with a _comment per block, composed in default.nix): - common.plan.json: - Stdout tracer (reconcile) - Prometheus metrics - admin web UI Application pinned to file://<nixpkgs webui-1.0.11>/webui.zip (was GitHub latest, fetched every start) - cell.plan.json: - AcmeProvider (matchOn directory, Http01) - mail domain with Automatic certs and an explicit SAN of the hostname, DKIM/DNS Manual - SystemSettings - listeners (reconcile): smtp/submission/submissions/imap/imaps/sieve, plus http on 127.0.0.1:8080 - dev.plan.json: dev’s domain, identity, unprivileged listeners, plaintext IMAP. - roles.plan.json: Stalwart’s four standard roles (verbatim from mx99, 0.16.23) plus the Authentication default role assignments. - sovrnd.plan.json: the sovrnd service account (sovrnd@<mail domain>, password is $secret sovrnd-stalwart-password) and its role: - extends User + Tenant Administrator - sys* CRUD on Tenant/Domain/Account/AppPassword/DkimSignature/Directory/MtaRoute/MtaDeliverySchedule - Get/Update on Jmap/MtaOutboundStrategy/MtaStageRcpt/SenderAuth/Authentication - Action create/get and actionReloadSettings, Task get/query - impersonate, authenticate - NOT System Administrator

Rendering and applying - nix/scripts/stalwart-render-plan.sh (pkgs.sovrn.stalwart-render-plan): @var@ tokens in keys and values, {"$secret": name} from files at converge time, _comment stripped, a hard error on anything unresolved. - Module (sovrn.services.stalwart.plan.{files,vars,secrets}) plus the read-only plan.converge = stalwart-converge [URL]. - The cell role sets the cell composition and vars (hostname = host.name, mail domain sovrn.at, LE prod, [email protected]). The service is still not enabled (ba3c79f).

Dev: serve-stalwart.sh renders the same dev composition with the same renderer. just -f Justfile.nix dev runs sovrnd as [email protected] (password data/dev/stalwart/sovrnd-password), so missing permissions surface in dev.

Verified - Plan check checks.stalwart-plan (seconds): a real apply ×2 of the cell and dev compositions against the pinned Stalwart in the build sandbox, with a local pebble as the ACME CA. The second apply must create nothing, and a misspelled property fails it. (--dry-run was not enough: it accepts unknown properties.) - VM test checks.stalwart: stalwart-converge in recovery mode twice (idempotent), then normal mode: - exactly the cell listeners (25/465/587/143/993/4190, 8080 on loopback, no ⁴⁴³⁄₉₉₅) - sovrnd reads Domain, is forbidden from Tracer/Certificate - the plan’s ACME provider gets the certificate from pebble - Integration suite as [email protected], against a dev-composition Stalwart: everything passes except two tests that fail identically as the recovery admin, so they’re unrelated to the role: - TestIntegrationAppViewUI (home page title assertion) - TestIntegrationOIDCBearerAuth (ReloadSettings request times out after the Authentication directory switch) - Manually: sovrnd creates tenant/domain/account. A master-user login (alice@dom%[email protected] + sovrnd’s password) yields Alice’s JMAP session with Sieve. Denied for sovrnd: SieveSystemScript, MtaHook, NetworkListener, Certificate, Tracer.

Findings (0.16.23) 1. A declaratively created DB has no roles at all. Default roles come from Bootstrap, which this flow skips; without them users authenticate but get 403. Hence roles.plan.json. 2. Stalwart refuses to grant permissions the granting account lacks. So sovrnd’s role must contain everything the roles it hands out contain (User, Tenant Administrator). 3. Creating an AcmeProvider registers with the CA synchronously during apply, so apply needs outbound access and a contact domain the CA accepts (LE rejects .test). 4. Account has no natural key: matchOn ["name","domainId"] is required, or a re-apply tries to create a duplicate. 5. Stalwart rejects localhost as defaultHostname (needs an FQDN). 6. A fresh DB’s first recovery start takes about 40s: the default Application still points at GitHub until the plan replaces it. After converge, startup is seconds. 7. Stalwart also downloads ASN/Geo data from GitHub (sapics/ip-location-db) at startup, another unpinned runtime fetch (follow-up).

Out of scope, handed on: relay plan to b45a76f; sovrn.toml [stalwart] username/secretfile to 63eb0d9; boot wiring to ba3c79f.