Cell data plane on NixOS: ZDS, sovrnd, Caddy, did.json - Issue #63eb0d9

Port the cell application roles: zds, sovrnd, caddy, didwellknown.

Sketch: - ZDS: - users.users.sovrn.linger = true - systemd.user.services."zds@" template (replaces [email protected]) - base dirs via systemd.tmpfiles - sovrnd keeps managing instances at runtime via systemctl --user; verify this works against NixOS-generated user units - per-cell ZDS secrets as keys - sovrnd: - service from the #3 package - sovrn.toml generated by Nix (pkgs.formats.toml) with secrets by file path only - the Resend key currently travels as a toml value ([pds]): switch it to a file path, or render the toml at runtime - Caddy: - services.caddy with on-demand TLS + ask endpoint (caddy_ask.go) - the Stalwart Http01 challenge proxy - storage in /var/lib/caddy - the Caddyfile must import sovrnd’s runtime-written maps (/etc/caddy/maps.d/*.map), which live outside the store: move them to /var/lib/caddy/maps.d or similar - reload path from sovrnd (admin API on 127.0.0.1:2019) - did.json: static file generated from Nix options.

Done when a scratch cell can onboard a domain end-to-end (domain saga, ZDS instance, Caddy cert, Stalwart account via the sovrnd service account).

3 Comments

agent 643be4b Sep 30

From b724acb: on cells, sovrnd must authenticate as the plan’s service account, not the recovery admin: - [stalwart] username = "[email protected]" (the cell mail domain) - secretfile = config.sovrn.secrets."sovrnd-stalwart-password".path (declared in the cell role, owner sovrn)

The role is in nix/stalwart/sovrnd.plan.json. The integration suite passes with it, and dev already runs this way (just -f Justfile.nix dev).

agent 6033e2b Sep 30

Implemented and VM-tested (pending review; domain onboarding not exercised end to end)

Go (sovrnd): host paths and the Resend key are configurable, so sovrn.toml can live in the Nix store. New [pds] keys: - resend_api_keyfile: read at load into the existing ResendAPIKey; setting both is an error; the error names the file, never the key. - secretsdir: the four zds-* cell secrets. - originsmap, vanitymap: the Caddy map files. - caddy_config: when set, the reload runs caddy reload --config <path> --adapter caddyfile --force.

supervisorConfig(cfg) and caddyReload(cfg) in pds_supervision.go replace three hand-built copies. CaddyReloadTrigger takes extra args. Tests added; go test ./... and gen-check pass.

Nix - nix/fleet.nix: fleet-wide settings (formerly group_vars), passed to modules as fleet. - nix/modules/sovrnd.nix: sovrn.services.sovrnd. settings becomes /etc/sovrn/sovrn.toml. The unit runs as sovrn, after/wants stalwart.service (which is only “started” once converged and ready) and the Colmena key units of every key path in the settings. PATH has systemctl and caddy. - nix/modules/zds.nix: sovrn.services.zds. Linger for sovrn; [email protected] as raw user-unit text with an [Install] section (NixOS emits none, and sovrnd runs systemctl --user enable); /var/lib/zds/{dbs,blobs}. - nix/modules/roles/cell-edge.nix: Caddy. Cell host → sovrnd; the root cell also serves sovrn.at with a static did.json from the store; http://<fqdn> proxies only the ACME challenge path to Stalwart :8080; :443 fallback with on-demand TLS (ask → sovrnd) and the two maps in /var/lib/caddy-maps (sovrn-owned, outside the store and outside sovrn’s 0700 home); /internal/* refused at the edge; firewall ⁸⁰⁄₄₄₃. - nix/modules/roles/cell.nix: enables Stalwart, ZDS and sovrnd with the full sovrn.toml settings. sovrnd authenticates as sovrnd@<mailDomain> with sovrnd-stalwart-password. pds.origin_suffix comes from the optional pdsOriginSuffix in hosts.json. Health backup markers are left unset until 0e32e5a. - new-host now merges into an existing hosts.json entry, so pdsOriginSuffix survives reinstalls.

VM test checks.cell (57s) passes: - stalwart, sovrnd and caddy start in order. - /healthz legs stalwart (“ready ok + jmap query ok”, i.e. as the service account), sqlite, zds and disk are ok. - The Stalwart journal shows [email protected] authenticating; sovrnd logs no forbidden/unauthorized. - HTTPS on the cell host reaches sovrnd; /internal/caddy-ask from outside gets 404. - The landing host serves did.json. - Plain HTTP proxies only the ACME challenge path to Stalwart. - Stalwart’s certificate is issued (pebble). - As the sovrn user, systemctl --user enable/start zds@t1 works with linger (the enable symlink is created) and ZDS answers. - The sovrn user rewrites the origins map and runs the same caddy reload sovrnd uses, and Caddy’s live config picks up the host. - An origin not in the registry is refused by caddy-ask and gets no certificate.

Not tested: the domain saga itself (tenant → domain → PDS instance → Caddy cert → mailbox). It needs an authenticated user session and DNS verification; the VM test covers each mechanism the saga uses, not the saga. That’s the remaining part of this issue’s “done when”, to be done on staging.

Findings - sovrnd creates its relay MtaRoute and issues ReloadSettings at startup. Stalwart validates its whole config on reload, so an unresolvable external host (e.g. the spam filter’s public.pyzor.org) fails the reload and sovrnd exits and restarts. That’s fine with working DNS; the VM tests add /etc/hosts entries. - The 40s Stalwart starts seen in VM tests were DNS timeouts against the base config’s unreachable resolvers. With no resolvers configured in the VM, a full recovery → converge → normal start takes about 21s (see ba3c79f).

agent 6c30efb Oct 8

Closing (user, 2026-10-08): done-condition met on mx99 (staging cell): test.kilimanjaro.io onboarded end to end (domain saga, ZDS instance, Caddy on-demand certificate for the PDS host, Stalwart mailbox via the sovrnd service account). Follow-ups filed separately: a6c4223 (live PDS provisioning for user accounts), 6844d4f (crawl + handle DNS), 9cbbce8 (internal Stalwart domain).