servers

NixOS definitions and workflow tools for my servers. You are on one right now. You just got served.

servers / roles / forge
legacy, forge: drop Pushover; the monitor alerts on /healthz (b660c34)
BT  ·  2026-10-11
8a27f49

README.md

roles/forge

Personal git hosting on infra.rtw.run: soft-serve for the repos, and kilimanjaro.io, a static site pgit builds from the public ones.

Name Serves
infra.rtw.run /healthz only: the forge’s health for the monitor. Points at the host.
git.kilimanjaro.io soft-serve’s HTTP (clone, go-get) through Caddy. Points at the host. / is a 404 by design.
kilimanjaro.io /var/www/code through Caddy. Proxied by Cloudflare.
www.kilimanjaro.io Redirect to kilimanjaro.io. Proxied by Cloudflare.
Port What
23231 git over SSH (ssh://git.kilimanjaro.io:23231/<repo>); post-quantum key exchange since soft-serve 0.11
9418 git daemon (git://git.kilimanjaro.io/<repo>)
80, 443 Caddy
23232, 23233 soft-serve HTTP and stats, localhost only

Files

File What
default.nix soft-serve, the hook, the backup and health jobs and their timers, Caddy’s four sites.
soft-serve/config.yaml /etc/soft-serve/config.yaml. soft-serve restarts when it changes.
soft-serve/hooks/post-receive Global hook, linked from /var/soft/data/hooks. Backgrounds forge-rebuild-site and returns at once.
rebuild-site.sh forge-rebuild-site: rebuilds every public, non-hidden repo’s pages and the index.
backup.sh Daily (20:30 UTC) rclone sync of /var/soft/data to r2:soft-serve; stamps last-success for /healthz.
healthz.sh forge-healthz, every 5 minutes: disk, load, backup age, soft-serve. Writes the /healthz response (below).
recover.sh Restores /var/soft/data from R2 and rebuilds the site.

On the host

Path What
/var/soft/data soft-serve: repos, soft-serve.db, its SSH host keys (ssh/). Backed up.
/run/forge-healthz The /healthz response: ok.json (Caddy answers 200) or degraded.json (503). Neither until the first run after boot (404).
/var/lib/forge-healthz/load 24 hours of load samples, for max_24h.
/var/lib/forge-backup/last-success Touched by each successful backup; /healthz fails when it’s over 26 hours old.
/var/www/code The generated site. Not backed up: forge-rebuild-site recreates it.
/var/log/pgit.log Output of every site build.
/var/lib/caddy Caddy’s certificates and ACME account (Caddy runs as caddy).
/var/lib/servers-keys rclone.conf (a Colmena key).

/healthz

The monitor (~/projects/monitor) polls https://infra.rtw.run/healthz and alerts by email and Pushover. It answers 200 when every check passes, 503 otherwise, with the body sovrn’s metrics-healthz uses:

{
  "status": "ok",
  "checked_at": "2026-10-11T05:15:18Z",
  "checks": {
    "disk": { "status": "ok", "detail": "used=18% threshold=80%" },
    "load": { "status": "ok", "detail": "load=0.00 0.00 0.00 max_24h=0.14 threshold=1.9 (15m)" },
    "backup": { "status": "ok", "detail": "last_success=2026-10-10T20:31:00Z age=8h max_age=26h" },
    "soft-serve": { "status": "ok", "detail": "active" }
  }
}

A failing check also has error. Caddy serves the file forge-healthz last wrote, so the answer can be up to 5 minutes old (checked_at). If the script itself fails, it replaces ok.json with a degraded.json naming the journal.

Things that bit before