Cells must not claim sovrn.at in Stalwart: use an internal domain for service accounts - Issue #9cbbce8

Problem

Every cell’s Stalwart plan (nix/stalwart/cell.plan.json, the Domain upsert named @mailDomain@ = fleet.mailDomain = “sovrn.at”) creates sovrn.at as a mail domain on that cell. So sovrn.at can’t be hosted like any customer domain: an operator should be able to sign in on a cell and add sovrn.at as a hosted email domain through sovrnd, with sovrnd owning its records, accounts and DNS checks. Today the domain already exists in Stalwart on every cell, outside sovrnd’s database.

Seen on mx99 (2026-10-08): sovrnd’s reconciler logs at ERROR every 5 minutes: verifier: reconcile: stalwart domain has no DB row; NOT deleting (investigate DB corruption/misconfiguration) (orphan.report, name sovrn.at; internal/verifier/orphan.go). It refuses to delete it (correctly), but it’s error-level noise and would count toward CellErrorBurst.

What the domain is used for today

sovrnd’s service account. nix/stalwart/sovrnd.plan.json creates the account sovrnd authenticates as over JMAP (HTTP Basic, username sovrnd@; nix/modules/roles/cell.nix sets stalwart.username = “sovrnd@${fleet.mailDomain}”). The relay’s route-sync account (relay.plan.json) follows the same pattern; check it.
Server default domain. cell.plan.json’s SystemSettings update sets defaultDomainId to it.
The MX hostname’s certificate. The same Domain carries certificateManagement (ACME, SAN = the cell hostname, e.g. mx99.eu.sovrn.at), so the cell’s SMTP/IMAP TLS certificate hangs off it.

Wanted

If Stalwart needs a domain for service accounts (and as the default), make it an internal, non-routable domain that can never collide with a customer or company domain, e.g. a reserved name under .invalid or .internal per cell (sovrn.internal / .internal). Nothing external should ever see it: no MX, no mail for it.
Keep the MX hostname certificate working: either on the internal domain’s certificateManagement (SAN = cell hostname; check Stalwart accepts a SAN outside the domain) or on its own object if 0.16 allows that.
Service account usernames move to the internal domain ([email protected] or similar): plan files, cell.nix’s stalwart.username, relay.plan.json, sovrnd config, tests (checks.stalwart asserts accountName = sovrnd@; checks.cell; stalwart-plan).
The reconciler should know the internal domain is not a tenant (skip it explicitly), so no orphan reports.
Then sovrn.at becomes an ordinary hosted domain: added through sovrnd on whichever cell should host it, like any customer.
Don’t conflate with other uses of “sovrn.at”: fleet.mailDomain is also used for the R2 bucket name derivation (nix/modules/backup.nix: ..sovrn.at -> -), landingHost/rootHost (the website), the service DID did:web:sovrn.at, and notify.sovrn.at for outbound senders. Those are about the company’s web and cell naming, not a hosted mail domain; split the setting so the internal mail domain is its own value.

Migration

Cells are pre-beta (mx99 staging only), so a rebuild or a plan reconcile that deletes the old sovrn.at Domain is fine. Make sure the plan reconciles it away rather than leaving it behind on existing cells.

Found during fleet Phase 9 (mx99 acceptance), ~/projects/servers docs/fleet-migration-plan.md.

3 Comments

agent 9ec1bdb Oct 10

Plan

Findings (Stalwart 0.16.23 / stalwart-cli 1.0.12 source): - An explicit SAN list is ordered verbatim (build_domains), so a SAN outside the domain works: sovrn.internal can carry the MX hostname’s certificate. Certificates are separate objects matched by SAN set (acme_certificate_by_domains), so with reuseKey the renewal under the new domain keeps the existing key: DANE 3 1 1 TLSA stays valid. - apply runs every destroy before any upsert, and Stalwart refuses to destroy a linked object (objectIsLinked). The old Domain is linked from SystemSettings.defaultDomainId and the old sovrnd account. So a single converge can’t remove it, and a permanent destroy Domain name=sovrn.at op would also destroy sovrn.at again on every restart once it is hosted through sovrnd. The migration is therefore a one-off on existing cells (mx99), not a plan op. - domain.Ensure already refuses to adopt a Stalwart domain owned by no tenant (AssignedError), which is also why adding sovrn.at through sovrnd fails today.

Changes: 1. fleet.nix: split mailDomain into serviceDomain = "sovrn.internal" (Stalwart’s own domain: service accounts, default domain, MX certificate) and cellZone = "sovrn.at" (the zone cell hostnames live under; backup.nix bucket naming). 2. cell.plan.json / dev.plan.json: the Domain is @serviceDomain@ / sovrn.internal; [email protected] everywhere (cell.nix, Justfile, serve-stalwart.sh, README, tests). The relay keeps its hostname as its domain: mxb.eu.sovrn.at is never a hosted domain and runs no reconciler. 3. sovrnd: domain.IsInternal(name) (the .internal TLD, ICANN-reserved for private use). The verifier’s reconcile pass skips internal Stalwart domains (debug log, no orphan.report); ProvisionDomain rejects them with ErrInvalidDomain. Unit tests for both. 4. Migration on mx99 after deploy: delete the account [email protected], then the Domain sovrn.at (stalwart-cli as the recovery admin). After that, sovrn.at can be added through sovrnd on its chosen cell.

agent 9dc9bdb Oct 10

Implemented (working copy, not yet committed)

fleet.nix: mailDomain split into zone = "sovrn.at" (website apex + cell hostnames; backup.nix bucket naming) and serviceDomain = "sovrn.internal".
cell.plan.json Domain is @serviceDomain@ (still carries the MX hostname certificate via explicit SAN); dev.plan.json uses sovrn.internal too. sovrnd authenticates as [email protected] (cell.nix, Justfile, serve-stalwart.sh, README). Relay unchanged (its hostname is its domain, never hosted, no reconciler).
domain.IsInternal (.internal TLD). Verifier reconcile skips it (no orphan.report); ProvisionDomain rejects it with ErrInvalidDomain (400). Tests: TestIsInternal, TestSweepOrphanSkipsInternalServiceDomain, TestProvisionDomainRejectsInternal.
Verified: go test ./…, checks.stalwart-plan, checks.stalwart (cert issued for mx1.sovrn.test under sovrn.internal), checks.cell, nix flake check –no-build.

Migration (one-off on mx99, after deploying)

Rehearsed locally on Stalwart 0.16.23: old cell plan, then new plan (adds sovrn.internal + [email protected], moves the default; sovrn.at and [email protected] stay), then:

cli() { STALWART_URL=http://127.0.0.1:8080 STALWART_USER=sovrn-admin STALWART_PASSWORD="$(cat /var/lib/sovrn-keys/stalwart-recovery-password)" stalwart-cli --no-color "$@"; }
dom=$(cli query Domain --where name=sovrn.at --json | jq -r .id)
acct=$(cli query Account --where domainId=$dom --json | jq -r 'select(.emailAddress=="[email protected]") | .id')
cli delete Account --ids "$acct"
cli delete Domain --ids "$dom"

Re-converge afterwards is a no-op. Domain delete leaves the Certificate object (matched by SAN), so the MX key and its TLSA record are unchanged. Then add sovrn.at through sovrnd on its cell.

Dev: the existing dev Stalwart keeps sovrn.test + [email protected] until sovrn-reset (or the same two deletes).

Cross-repo: ~/projects/servers reads fleet.mailDomain (Justfile:296, scripts/update-dns.sh, scripts/check-dns.sh); rename to zone when bumping the sovrn input.

agent 9bc2b4b Oct 10

Deployed to mx99 (2026-10-10)

sovrn d1530622 committed; servers cc97bab0 (fleet mailDomain -> zone in Justfile _dns, check-dns.sh, update-dns.sh; sovrn lock bump). just check-dns passes with the rename.
just deploy mx99.eu.sovrn.at: converge 22 updated, 2 created (0 failed); sovrnd authenticates as [email protected], healthz stalwart ok. The ACME task under sovrn.internal reported the certificate still valid (matched by SAN), so no new order.
Migration run on mx99: deleted Account [email protected] (id b) and Domain sovrn.at (id b). Remaining: sovrn.internal, test.kilimanjaro.io.
Certificate object for mx99.eu.sovrn.at intact; served SPKI unchanged (6d4d9619…bd0a5c6c) and still matches the published TLSA 3 1 1.
sovrnd: no orphan.report or ERROR since the deletes (last report 07:15:40, before them).
infra.mymood.at not deployed: its diff is rebuild-only (render script comment, Go rebuild), so it would just restart the relay’s Stalwart with an identical plan. It picks this up on its next deploy.

Next: add sovrn.at through sovrnd on mx99 as a hosted domain.