README.md
servers
The fleet: every NixOS host, deployed with Colmena from this repo. Projects with their own repos (sovrn, moods) export NixOS modules; the fleet pins them as flake inputs and puts their roles on hosts, so one machine can run several projects. The migration that got here, and what’s next, is in docs/fleet-migration-plan.md.
| Host | Provider | Roles | Notes |
|---|---|---|---|
infra.rtw.run |
Hetzner | forge |
Git hosting (soft-serve, kilimanjaro.io). Legacy layout (installed with nixos-infect), nixos-26.05 |
infra.mymood.at |
netcup | moods, sovrn-metrics, sovrn-relay |
The shared box: moods, sovrn’s metrics stack (infra.sovrn.at) and backup MX (mxb.eu.sovrn.at). Inbound ports and UDP replies also need netcup’s firewall policy (provider firewalls) |
mx99.eu.sovrn.at |
Hetzner (CAX11, arm64) | sovrn-cell |
sovrn’s staging cell; also a Nix remote builder for aarch64 |
Layout
| Path | What |
|---|---|
flake.nix |
Inputs: nixpkgs (pinned by rev to the one sovrn and moods lock), nixpkgs-stable (nixos-26.05, by rev), colmena, disko, pgit, sovrn, moods. Outputs colmenaHive and nixosConfigurations (the same systems). |
hosts.json |
Inventory: roles, provider or layout, addresses, pinned SSH host key, stateVersion. See the header of hive.nix. |
hive.nix |
Turns the inventory into the Colmena hive. Evaluation fails if sovrn’s or moods’ nixpkgs differs from the fleet’s. |
modules/ |
Fleet modules for standard hosts: base, vm (disko, GRUB), hetzner, netcup, secrets (servers.secrets), plus fleet (role checks), caddy (ACME email [email protected]) and shell (ghostty terminfo, fish-like bash for interactive sessions) on every host. |
roles/ |
Roles that live here (forge). Project roles come from the project flakes. |
hosts/<fqdn-with-dashes>/ |
Legacy hosts only: hand-written hardware, networking, stateVersion. hosts/common/ is shared by them. |
keys/admins.pub |
Root SSH keys for standard hosts. |
docs/runbooks/provider-firewalls.md |
The rules for provider firewalls (netcup’s policy, Hetzner Cloud Firewalls) in front of the NixOS firewall, per host, with the reason for each. |
Workflow
The recipes run from any shell: the tools only the dev shell provides (colmena, nvd, nixos-anywhere) are called through nix develop --command inside the Justfile. On PATH you need just, nix, jq, ssh/ssh-keygen/ssh-copy-id, and secrets (the age-encrypted store CLI) with SECRETS_DIR (default ~/projects/data). nix develop is only for running those tools by hand.
| Command | What |
|---|---|
just eval |
Evaluate every host (fast; no builds). |
just diff-host <fqdn> |
Build what the fleet would deploy and compare it with what the host runs (nvd diff); prints “identical” for a no-op. |
just deploy <fqdn> [args] |
Check its secrets, then colmena apply. just deploy <fqdn> --reboot makes the new system the boot default and reboots into it. |
just deploy-project <sovrn\|moods> |
Pick up the project’s last commit (nix flake update <p>) and deploy every host running one of its roles. Commit flake.lock afterwards: it is the deploy log. |
just check-secrets [fqdn] |
Verify every secret a host declares decrypts from the store. |
just check-dns |
Compare the sovrn.at zone with the fleet: the Marque record in the PDS, each authoritative nameserver, and DNSSEC validation of the TLSA records. Expected records come from hosts.json and sovrn’s fleet.nix (see scripts/check-dns.sh). Needs dig and openssl. |
just update-dns |
Converge the sovrn.at zone on the fleet: shows check-dns, then the plan (records to remove and add, only in the rrsets the fleet manages), asks, and writes the Marque record with goat (logged in as @rtw.run; swapRecord guards against concurrent edits), then re-checks once the nameservers have it. DRY=1 stops after the plan. |
just known-hosts |
Pin every host’s SSH key (from hosts.json) in ~/.ssh/known_hosts. |
just sync-nixpkgs |
After just update-nixpkgs in sovrn and moods: move the fleet to the revision they lock. |
just new-host <ipv4> <fqdn> <hetzner\|netcup> <roles> |
Install NixOS on a fresh box with nixos-anywhere (erases it): probes the image, records it in hosts.json, generates its secrets and SSH host key, installs, pins the key. |
just gen-host-secrets <fqdn> |
Create the host’s missing secrets: its SSH host key (servers/hosts/<fqdn>/) and the host-scoped secrets its roles declare, from each project’s gen-secret-<project> app. Never rotates; DRY=1 only reports. |
Before a deploy that changes anything risky (networking, boot, a major upgrade), run just diff-host and read the diff, and for networking also the generated network-addresses-* units: 26.05 changed when a default gateway gets installed, which only showed there.
Rollback: the previous generation in GRUB, nixos-rebuild switch --rollback on the host, or just deploy from an earlier commit.
Project inputs
sovrn and moods are git+file:///home/btburke/projects/<p>?ref=HEAD: the last committed change (in jj, @-), never the working copy. Uncommitted work is not deployed.
Secrets
Secrets live in the secrets store, one age file per value, and reach hosts as Colmena keys: Colmena runs secrets decrypt <path> on the deploying machine and uploads the result, so nothing secret enters the Nix store or this repo.
servers.secrets."rclone.conf".scope = "host"; # servers/hosts/<fqdn>/rclone.conf
servers.secrets."<name>".scope = "shared"; # servers/shared/<name>
A service reads config.servers.secrets.<name>.path (under /var/lib/servers-keys, root-owned 0400 by default) and orders itself after config.servers.secrets.<name>.unit. To add one: printf '%s' "$VALUE" | secrets encrypt servers/shared/<name>, declare it, deploy. sovrn and moods have the same scheme under their own prefixes and key directories.
New hosts
Standard hosts are installed with just new-host over the provider’s stock image (nixos-anywhere, disko layout). Hetzner images take FLEET_SSH_KEY (default ~/.ssh/hetzner) for root; netcup images only allow a password, so new-host asks for one of keys/admins.pub and copies it over first. The SSH host key lives at servers/hosts/<fqdn>/ssh_host_ed25519_key and is reused on reinstalls (for infra.mymood.at it is a link to the key moods generated). New hosts get stateVersion 26.05.
The project repos’ host recipes (just deploy, new-host, … in sovrn’s and moods’ Justfile) forward here. Neither project keeps host configuration of its own.