Provision NixOS on Hetzner with nixos-anywhere + disko - Issue #bc4b527

First step: Hetzner offers no NixOS image, so we install NixOS over a stock image with nixos-anywhere (kexec) + disko.

Sketch: - Add a Colmena hive to flake.nix (colmenaHive / colmena output) plus nixosConfigurations for nixos-anywhere. - Add a disko layout for Hetzner Cloud disks (single disk, ext4 or btrfs; decide). - Add a base module: - sovrn user/group and admin SSH keys - networking.firewall (replaces ufw: 22/25/80/443/465/587/143/993 per role) - services.resolved with Hetzner primary + Quad9 fallback (replaces roles/common) - timezone, journald limits, auto-GC - Hetzner networking: IPv4 via DHCP, IPv6 declared statically per host (Hetzner Cloud does not hand it out via DHCP). Decide on cloud-init vs hard-coded per-host addresses. - Support x86_64 and aarch64 (CAX) hosts. - Choose the host nixpkgs channel (open decision on the tracking issue).

Gotchas: kexec needs about 1 GB+ RAM; Hetzner primary IPs are recycled across rebuilds (known_hosts rotation, see docs/runbooks/recovery-primary-ip.md).

Done when a scratch Hetzner box goes from nixos-anywhere --flake .#scratch root@<ip> to NixOS, and colmena apply --on scratch is a no-op afterwards.

1 Comment

agent b4c54db Sep 29

Progress: scaffolding in place (not yet run against a real Hetzner box)

Layout (see nix/README.md): - nix/overlay.nix provides pkgs.sovrn.{stalwart,stalwart-cli,zds}, shared by the flake (hosts) and devenv (dev). - nix/hosts.json is the inventory, turned into the Colmena hive by nix/hosts.nix. flake.nix exposes it as colmenaHive and as nixosConfigurations = colmenaHive.nodes, so nixos-anywhere installs exactly what Colmena deploys. - Modules: modules/base.nix (sovrn user, SSH with an ed25519-only host key, firewall, resolved with Hetzner + Quad9), modules/hetzner.nix (disko: BIOS boot + ESP + ext4; GRUB for BIOS and UEFI; DHCPv4 + static IPv6 via fe80::1), and empty modules/roles/{cell,relay,metrics,scratch}.nix. - Justfile.nix recipes: dev, new-host IP HOSTNAME, known-hosts, build, deploy, eval, update-nixpkgs.

Decisions made here - Host nixpkgs is nixos-unstable pinned by flake.lock. nixos-26.05 only has Stalwart 0.15.5; stalwart_0_16 exists only on unstable. The version assert in overlay.nix catches any drift. - devenv.yaml pins the same nixpkgs rev as flake.lock, so dev and hosts share store paths. update-nixpkgs bumps both together. - Colmena comes from nixpkgs 0.5.0 plus the matching flake input github:zhaofengli/colmena/v0.5.0 (0.5 requires the colmenaHive output).

Host-key handling in new-host 1. The host key is pre-generated and stored as secrets → sovrn/hosts/<fqdn>/ssh_host_ed25519_key, or reused if already stored. 2. It is pinned in hosts.json and installed via nixos-anywhere --extra-files. 3. After the reboot, the recipe verifies the box presents exactly that key, then replaces the fqdn/IP known_hosts entries.

nixos-anywhere itself runs with UserKnownHostsFile=/dev/null, so it never trips over key changes. Rebuilds keep their key.

Verified - Both an x86_64 and an aarch64 test host evaluate through nixosConfigurations and colmena. The system build is all cache hits except trivial config drvs. - nixpkgs Stalwart 0.16.23 serves schema hash zUWyYdvO… = internal/stalwart.PinnedSchemaHash. - The dev converge flow works: declared config.json → recovery mode (:18080) → stalwart-cli apply of nix/stalwart/dev.plan.json → normal mode. The second apply creates nothing. It also migrates the old container-bootstrapped dev DB (drops the file tracer and privileged listeners). - go build, go test ./... and just gen-check pass on the new toolchain. nix flake check --no-build --all-systems passes. - new-host’s failure paths (unreachable IP, bad args) leave hosts.json untouched. The secrets generate/reuse step and known_hosts pinning were tested in isolation.

Not yet done - An actual install on a scratch Hetzner box. - Tracer in the plans uses matchOn: ["@type"]; carry that into b724acb.