Pilot: metrics box (infra.sovrn.at) on NixOS - Issue #dd3c64a

First real cutover: infra.sovrn.at. It holds the least state and is the lowest risk.

Port roles/metrics_box to NixOS modules: - services.victoriametrics (90d retention) - VictoriaLogs (30d retention, 25GiB max disk) - vmalert with rules-cell.yml and rules-logs.yml (move the Jinja templates to Nix) - Alertmanager (alerts to [email protected] from [email protected]; credentials via keys) - the Caddy infra vhost - metrics-healthz

Check the nixpkgs versions against the current pins (VM 1.152.0, VL 1.52.0). Decide whether to overlay-pin or float.

Decide whether to keep existing TSDB/log data (rsync to the new box) or accept a gap.

Done when cells and the relay push to the NixOS box, alerts fire in a test, and colmena apply --on infra is idempotent.

1 Comment

agent dbd93cc Sep 30

Progress: metrics role implemented and VM-tested; real host install in flight

nix/modules/roles/metrics.nix uses nixpkgs modules throughout, all DynamicUser and loopback-only, with Caddy as the only public listener: - services.victoriametrics: 90d retention, -vmalert.proxyURL. - services.victorialogs: 30d / 25GiB, journald stream fields. - services.vmalert.instances: "" (VM, cell-metrics rules) and logs (VL, vlogs rule; state goes to VM). The rules are now Nix attrsets. - services.prometheus.alertmanager: Resend SMTP, password via LoadCredential (Alertmanager TrimSpace’s the file, so the store’s trailing newline is fine). - metrics-healthz from pkgs.sovrn. Its disk legs point at /var/lib, because the DynamicUser state dirs under /var/lib/private (0700) aren’t traversable; it’s the same filesystem. - Caddy: basic_auth reads the bcrypt hash with {file./run/credentials/caddy.service/metrics-hash}. Verified that Caddy resolves {file.*} at runtime: right password 200, wrong or none 401. There’s no env drop-in any more.

Versions: everything matches the Ansible pins except Alertmanager, which is 0.33.1 in nixpkgs versus the 0.34.0 pin. Accepted.

VM test nix/tests/metrics.nix (nix build .#checks.x86_64-linux.metrics -L) passes, checking: - /healthz is public and ok. - Everything else is 401 without auth or with a wrong password, and 200 with the right one. - Metrics import through the edge lands in VM. - A jsonline log through the edge lands in VL. - Both vmalert instances load their rules. - The Alertmanager credential is present.

Bug found and fixed in the secrets module (4ce0593): the key dir was /var/lib/sovrn/keys, inside the sovrn user’s 0700 home. systemd-tmpfiles refused it as an unsafe path transition: sovrn owned the parent and could swap it, and other service users couldn’t traverse it. Keys now live in /var/lib/sovrn-keys (root, 0711).

Test gotcha: a curl --data-binary without a JSON Content-Type makes VL answer 200 while ingesting 0 rows, because the body is consumed as a form. Vector sends the correct type.

Real host: new-host 95.216.193.29 infra.sovrn.at was started (with SOVRN_YES=1) before explicit user confirmation. Its outcome has not been verified yet. After it finishes, just -f Justfile.nix deploy infra.sovrn.at is still needed to upload the Caddy/Alertmanager keys (new-host doesn’t push Colmena keys); until then those two services fail to start.