sovrn

Experimental AT protocol email hosting

T12: reevaluate amd64+arm64 cell support after FoundationDB removal

closed
#ea055d0 opened by agent Sep 13

Goal

Determine whether a sovrn cell can be deployed on both amd64 and arm64 now that FoundationDB is out of the picture (T3, bug a6181d9).

Context

The only amd64-only constraint in the Ansible stack was FoundationDB: pinned foundationdb-{clients,server}_7.3.68-1_amd64.deb downloads plus the common role assert (Host is ...: FoundationDB publishes amd64-only .debs). T3 deletes that assert entirely, which opens the possibility of running the playbooks on arm64.

Parent: bug 75966cc (cell architecture tracking).

Scope

  • Inventory what (if anything) is still arch-coupled after T3: Nix-built stalwart + sovrnd binaries (glibc ceiling checks in Justfile build-stalwart target trixie x86-64), Caddy/ZDS/Litestream/rclone installs, ufw/firewall assumptions, Hetzner plan availability per region.
  • Test-build the controller binaries for aarch64-linux (or document why cross-build is out of scope) and converge just update on an arm64 Debian host (staging cell).
  • Decide: cells are amd64-only (re-add a scoped arch assert with the new rationale), mixed-arch supported, or arm64-preferred for price/perf.
  • Record the decision in docs/deployment.md + docs/adr/0009-cell-architecture.md (or a new ADR if the decision diverges).

Acceptance

  • Cell boots green on the supported arch(es) via just bootstrap + just update with no changes on second run; arch support matrix documented; any re-added arch assert names the real constraint (not FDB).

2 Comments

agent e8a7045 Sep 13

T12 static evaluation — verdict: FEASIBLE, mixed-arch (EU-arm + US-x86)

Static-only per owner scope (no live host yet). All probes run on x86_64 controller 2026-09-14.

1. Blockers: exactly 3 code items, no FDB residue

  • deployment/inventory/group_vars/all/sovrn.yml:49 — # Packaging: amd64-only. (policy comment only).
  • deployment/roles/sovrnd/tasks/main.yml:19-20 — go build native, no GOARCH; comment says native amd64 toolchain.
  • deployment/roles/sovrnd/tasks/main.yml:63-64,71 + deployment/roles/stalwart/tasks/install.yml:61-62,69 — hardcoded /lib64/ld-linux-x86-64.so.2 in both patchelf --set-interpreter and failed_when asserts. Hard-fails any arm64 host (arm64 Debian loader is /lib/ld-linux-aarch64.so.1).
  • FDB grep: zero hits in deployment/ (only historical docs + vendored lockfile internals). T3 removal holds.
  • No litestream/rclone/zds/unbound/hcloud roles exist yet — nothing to arch-fix, but T2/T4 must write them arch-neutral from birth.

2. Arch-neutral baseline confirmed

  • flake.nix:27-30 is eachDefaultSystem (no arch pin); nix/pkgs/stalwart.nix is sqlite+rocks, no arch conditionals; nix/images/stalwart.nix, devenv.nix, both bootstrap scripts, all templates/handlers: clean.
  • Go source: zero GOARCH/GOOS/amd64/arm64 hits. Only coupling is build-time (cgo), not source.
  • Justfile:56-71 glibc gate + sovrnd/main.yml:32-35 ceiling are version checks, arch-neutral logic (GLIBC_2.41 / GLIBCXX_3.4.34 re-validate on arm64 trixie at live test).
  • sovrnd/main.yml:63 --set-rpath /usr/lib needs verification on arm64 (multiarch libs live under /usr/lib/aarch64-linux-gnu).

3. Probe results (exact outputs)

  • Caddy arm64: PASS. binary-arm64/Packages publishes caddy_2.11.4_linux_arm64.deb (Cloudsmith any-version main, apt resolves arch). Caddy role needs no change.
  • Stalwart cross-build: expected FAIL (out-of-scope, not a code bug). nix build .#stalwart --system aarch64-linux on x86_64 controller → platform mismatch, Required system: aarch64-linux. No remote builder/qemu configured. Expression itself is portable → live test must build on native arm64 controller (or add remote builder).
  • sovrnd cross-build: expected FAIL (missing cross toolchain, not a code bug). CGO_ENABLED=1 GOOS=linux GOARCH=arm64 go build ./cmd/sovrnd → host gcc emits x86 assembler errors (no such instruction: stp x29,x30..., exit 1). Driver is mattn/go-sqlite3 v1.14.28 (cgo, internal/store/sqlite.go:28, internal/authbroker/store.go:30). Note: CGO_ENABLED=0 cross-compiles to a static ARM binary but sqlite via mattn requires cgo at runtime — not production-viable. Live test must compile natively on arm64 (or provision aarch64-linux-gnu-gcc).

4. Hetzner constraint (via hcloud server-type/location list)

  • ARM cax11-41 exists EU-only (fsn1, nbg1, hel1); US locations ash (Ashburn) + hil (Hillsboro) are x86-only (cpx11-51, ccx). There is no arm64 US option.
  • Consequence for EU/US cell model: arm64 adoption forces a mixed-arch fleet (EU-arm + US-x86) or amd64-everywhere. Uniform-arm64 is impossible.
  • Price/perf: cax11 €5.99/mo (2c/4GB/40GB) vs cpx11 €5.49/mo (2c/2GB/40GB) — ~2x RAM for +9%. Live test on cax11 is cheap. (Note: cax11 currently shows Available: no in fsn1/nbg1, Recommended: yes in hel1 — check capacity at provision time.)

5. Recommendation: GO for live test

Feasible with scoped prerequisites, all small: 1. Loader-map fix (both roles): ansible_facts.machine → x86_64: /lib64/ld-linux-x86-64.so.2, aarch64: /lib/ld-linux-aarch64.so.1; verify --set-rpath on arm64 trixie. 2. Build natively on arm64 for the test (Stalwart via just build-stalwart on arm64 controller; sovrnd go build on arm64) — no cross-toolchain work needed for the spike. 3. Live-test target: EU cax11 (e.g. mx99.eu.sovrn.at), just bootstrap + just update + second-run idempotence; re-check glibc ceiling strings on the arm64 binary. 4. Then record matrix in docs/deployment.md + amend docs/adr/0009-cell-architecture.md D24/D25 (arch + server-type + EU/US→DC mapping), expand sovrn.yml:49.

No re-added amd64-only assert recommended — the real constraints (loader path, native builds, US x86-only) are all parameterizable, not architectural.

agent e0aa065 Sep 13

Decision: amd64 everywhere (T12 closed, no live test)

Owner call 2026-09-14: the cost/perf delta (cax11 2c/4GB €5.99 vs cpx11 2c/2GB €5.49) is not worth a mixed-arch install base. Cells stay amd64-only for simplicity; arch optimization deferred to a later issue.

Supporting facts from the static evaluation (comment e8a7045): - Only 3 amd64 couplings exist (2x patchelf loader hardcodes + native-only builds) — they already enforce amd64, so no code change needed to hold the line. - Hetzner ARM is EU-only; US sites are x86-only, so any arm64 adoption would force a mixed fleet. Uniform-arm64 was never on the table. - Caddy arm64 availability and native-build paths are proven viable, so reopening later is low-risk.

Follow-up (when reopened): loader-map fix + native arm64 builds + EU cax11 live converge + docs/deployment.md matrix + ADR-0009 D24/D25 amendment. No docs edits in this close-out; sovrn.yml:49 (Packaging: amd64-only.) remains the standing marker.