1 files changed,
+37,
-8
+37,
-8
1@@ -508,20 +508,49 @@ localhost. `nix/module.nix` packages all of this as
2 VM.
3
4 Multi-arch: jobs use the host nix-daemon, which routes derivations by
5-`system`: `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]` (as the fleet
6-already does with `buildOnTarget = false`), and optionally `nix.buildMachines`
7-for a dedicated ARM builder if emulation is too slow (not a production host).
8-Steps run natively on x86_64; aarch64 work goes through `nix build` /
9-`colmena build` / flake `checks`.
10+`system`. x86_64 work runs natively. aarch64 work (`nix build` /
11+`colmena build` / flake `checks`) goes to **`mx99.eu.sovrn.at`** as a remote
12+builder (`nix.buildMachines`, native aarch64, staging host so it may be
13+loaded). `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]` stays as a slow
14+fallback only. The builder login is a restricted `nix-ssh` user, not root, and
15+mx99 limits builds (`maxJobs = 1`, a build slice with `MemoryMax`, low
16+`CPUWeight`/`IOWeight`, `Nice`, zram) so a build cannot starve the staging
17+app.
18+
19+Binary cache: the CI host's own `/nix/store` is the cache, served read-only by
20+`harmonia` behind Caddy (authenticated: store paths reveal host layout, even
21+though they hold no secrets). Producers push signed paths with
22+`nix copy --to ssh-ng://<ci host>`, from a `post-build-hook` where useful:
23+
24+- the dev machine (64 GB RAM) builds x86_64 closures, which `ci run` there and
25+ the runner evaluate to identical store paths, so the runner substitutes
26+ instead of building;
27+- mx99 pushes what it builds for aarch64;
28+- every fleet host and the runner list the cache in `substituters` and its
29+ public key in `trusted-public-keys`; the `ci` user needs no extra trust.
30+
31+The cache is not a repository of project state (the runner pushes nothing to
32+repos): losing it only costs rebuild time. GC on the CI host (`min-free` /
33+`max-free`) must not eat it silently: pin GC roots for the current closure of
34+each host (last N deployed generations), and watch disk (30 GB on
35+`infra.rtw.run`). Only identical derivations hit; a dirty tree or different
36+input pin builds fresh, which is correct.
37
38 Workspaces: per-run worktrees (a gate can hold a run open for hours); `HOME`
39 and `.cache` shared per repo (accepted shared state). `.devenv/` is per run;
40 activation still hits a warm store. Host substituters include
41 `devenv.cachix.org`, since the `ci` user can't add substituters itself.
42
43-Host sizing: `infra.rtw.run` is 2 vCPU x86_64. Resize to 4–8 vCPU / 8–16 GB
44-and stay x86_64 (matches the dev machine and most of the fleet). Do it with
45-fleet plan step 2.8 (rebuild on the standard layout).
46+Host sizing: `infra.rtw.run` stays **unresized** (2 vCPU / 3.8 GB, x86_64,
47+no swap). That is enough because builds happen elsewhere: x86_64 on the dev
48+machine, aarch64 on mx99, both pushed to the cache, so the runner mostly
49+evaluates and substitutes (fleet eval ~1-2 GB). Requirements: zram on the
50+host, `ci.slice` with `MemoryMax` about 2.5 GB, one concurrent run, and
51+`nix.settings.max-jobs` of 1. Revisit (resize, or a dedicated ARM builder) if
52+runs start building locally or two overlapping runs become necessary.
53+`infra.mymood.at` (4 vCPU / 8 GB) was considered as the runner host and
54+rejected: it carries the live moods app and backup MX, and the runner holds
55+root on the fleet.
56
57 ## 11. Appview
58