6 files changed,
+817,
-0
A
.envrc
+2,
-0
1@@ -0,0 +1,2 @@
2+eval "$(devenv direnvrc)"
3+use devenv
+11,
-0
1@@ -0,0 +1,11 @@
2+# Devenv
3+.devenv*
4+devenv.local.nix
5+devenv.local.yaml
6+result*
7+
8+# direnv
9+.direnv/
10+
11+# Node
12+node_modules/
+516,
-0
1@@ -0,0 +1,516 @@
2+# Nix-native CI/CD for Tangled — Design
3+
4+Status: design agreed, not yet implemented (2026-10-16).
5+
6+A Tangled.org-compatible CI/CD system for a self-hosted soft-serve forge, built
7+on Nix, devenv, NixOS and Colmena. It speaks Tangled's lexicons and ATProto
8+semantics (DIDs, AT-URIs, lexicon-typed records, cursor-resumable event
9+streams), runs workflows in each repo's devenv shell, and gates deploys behind
10+a human decision that can be made from a phone.
11+
12+## 1. Goals and non-goals
13+
14+Goals
15+
16+- Push to soft-serve → workflow runs → status and live logs visible in a
17+ minimal web UI, usable from a phone.
18+- Gated CD: a person approves deploys; health is reported, rollback is one
19+ button. Deploys can be triggered from a remote Claude session (`/rc`)
20+ without the dev machine.
21+- **Local parity**: the exact same flow runs from the command line on the dev
22+ machine. No push-and-pray loop.
23+- **Generic**: nothing in the system knows about sovrn, moods, Colmena or
24+ NixOS deploys. Any project can reuse it.
25+- Multi-arch (x86_64 + aarch64) from one runner, through Nix.
26+- Interoperate with Tangled's protocol: pipeline and status records on
27+ Tangled-shaped event streams.
28+
29+Non-goals (for now)
30+
31+- Running untrusted code. Only the owner's code runs; isolation exists to
32+ keep CI from changing the host's config or competing with its services, not
33+ to contain an attacker.
34+- Multiple runners, autoscaling, job claiming.
35+- A PDS, relay or Jetstream. Records are served by the services that own them.
36+- A polished UI.
37+
38+## 2. Principles (the decisions behind this design)
39+
40+1. **No programming in YAML.** The workflow file is an inventory of actions
41+ plus the default order. All logic (what to show, what happens next, when
42+ to ask a human) lives in code — Nix, shell, Justfiles — and reaches the
43+ engine through a typed JSON result contract.
44+2. **The engine is the product; the runner is a host for it.** The same `ci`
45+ binary runs locally and on the runner, pinned per repo through devenv.
46+3. **Steps are the happy path; actions are everything the workflow can do.**
47+ A step's result can route to any declared action, including happy-path
48+ steps.
49+4. **Gates are monotonic.** Code can add a gate, never remove one.
50+5. **Health is notify-only.** Transient errors are common after a deploy;
51+ the operator decides whether to roll back.
52+6. **Secrets are a repo.** An encrypted vault cloned at a known path; the
53+ `secrets` tool behaves identically everywhere. The engine has no secrets
54+ concept.
55+7. **Simplicity over isolation strength.** Sandboxed systemd units, not VMs,
56+ until something needs more.
57+
58+## 3. System overview
59+
60+```
61+ forge host (infra.rtw.run, initially everything lives here)
62+ ┌──────────────────────────────────────────────────────────────────────────┐
63+ │ soft-serve ──post-receive──► bridge ── /events (sh.tangled.git.refUpdate)┐│
64+ │ ▼│
65+ │ runner daemon ── reads .tangled/workflows @ newSha, matches `when:` │
66+ │ │ per run: worktrees, sandboxed unit ─► ci (engine) │
67+ │ │ (ci.slice, user ci, pasta netns) in devenv shell │
68+ │ │ tails <run>/events.ndjson │
69+ │ └─► /events (sh.tangled.pipeline[.status], run.rtw.ci.*) , /logs/... │
70+ │ │
71+ │ appview ◄── bridge /events + runner /events,/logs ── SQLite index │
72+ │ └─► browser / phone (Tailscale or authed public), ntfy, JSON+text API │
73+ │ │
74+ │ host nix-daemon ─► binfmt aarch64-linux (─► optional remote builder) │
75+ └──────────────────────────────────────────────────────────────────────────┘
76+ dev machine: `ci run <workflow>` — same engine, same manifest, same recipes
77+```
78+
79+The bridge, runner daemon and appview are three roles of one Go binary. They
80+can run as one service but keep real websocket boundaries, so they can be
81+split across hosts later.
82+
83+| Tangled role | Here | Responsibility |
84+|---|---|---|
85+| Knot | **bridge** next to soft-serve | Turn pushes into `sh.tangled.git.refUpdate` events |
86+| Spindle | **runner daemon** + **engine** | Discover and match workflows, run them, emit pipelines, status and logs |
87+| Appview | **appview** | Index streams, show runs, gates, actions |
88+
89+## 4. Protocol and identity
90+
91+- Each service has a `did:web` (e.g. `did:web:git.kilimanjaro.io` for the
92+ bridge, `did:web:ci.rtw.run` for the runner) serving a DID document.
93+- Repos have DIDs, as upstream Tangled repos now do. A soft-serve repo is
94+ `did:web:<forge-host>:<repo>` (e.g. `did:web:git.kilimanjaro.io:sovrn`).
95+ It is an identifier first; serving its DID document is optional.
96+- Records use TIDs as keys and reference each other by AT-URI. Following the
97+ spindle's convention, a pipeline is
98+ `at://did:web:<knot-host>/sh.tangled.pipeline/<tid>`.
99+- Records are **not** written to a PDS. Each service serves its own records
100+ on a cursor-resumable websocket, as Tangled's knot and spindle do:
101+ - `GET /events?cursor=N` — replay events with `created > N`, then tail. The
102+ envelope is upstream's `eventstream.Event`: `{rkey, nsid, event, created}`,
103+ where `created` is a monotonic local clock value and doubles as the cursor.
104+ - `GET /logs/{knot}/{rkey}/{workflow}` — replay the log, then tail (runner).
105+ Lines are upstream's `sh.tangled.ci.subscribePipelineLogs` `#control` /
106+ `#data` objects.
107+- Tangled lexicons used as-is (checked against tangled core `e0ef7981d`,
108+ 2026-10-11):
109+ - `sh.tangled.git.refUpdate` — emitted by the bridge, as a knot does
110+ (`ref, committerDid, ownerDid, repo, oldSha, newSha, changedFiles,
111+ pushOptions, meta{isDefaultRef, commitCount}`).
112+ - `sh.tangled.pipeline` and `sh.tangled.pipeline.status` — emitted by the
113+ runner on `/events`. Upstream marks both records deprecated in favour of
114+ `sh.tangled.ci.pipeline`, but the spindle still emits them on `/events` and
115+ the appview still consumes them, so they remain the compatible choice.
116+ - `sh.tangled.ci.*` XRPC (`queryPipelines`, `getPipeline`,
117+ `triggerPipeline`, `cancelPipeline`, `subscribePipelineLogs`) is the
118+ newer spindle API. The runner implements `triggerPipeline` (manual runs)
119+ first; the queries are a later addition.
120+- Custom lexicons under `run.rtw.ci.*` (namespace is a placeholder; pick a
121+ domain you control):
122+ - `run.rtw.ci.workflow` — the manifest schema (§6)
123+ - `run.rtw.ci.step.result` — the step result contract (§7)
124+ - `run.rtw.ci.gate` — `{pipeline, workflow, action, options[], state:
125+ waiting|approved|rejected|expired|superseded, actor, prompt}`
126+ - `run.rtw.ci.transition` — engine routing decisions, for the timeline
127+ Tangled clients ignore unknown NSIDs, so nothing breaks.
128+- Lexicons are the single schema source. Go types are hand-written for the
129+ handful of records used (Tangled's `api/tangled` package pulls in the whole
130+ monorepo), and a test checks every Go struct against its lexicon's
131+ properties so they cannot drift.
132+- Auth between services: bearer token over TLS (or Tailscale) to start.
133+ ATProto inter-service JWTs signed by the services' DID keys later.
134+
135+## 5. Bridge (forge side)
136+
137+- A line appended to the existing soft-serve `post-receive` hook (which
138+ already rebuilds the pgit site) calls `ci bridge notify <repo> <old> <new>
139+ <ref>`. It must never block or fail a push.
140+- For each pushed ref the bridge emits one `sh.tangled.git.refUpdate` event:
141+ repo DID, ref, old/new sha, changed files, push options (`-o skip-ci` is
142+ honoured by the runner, as upstream). The bridge does not read workflows;
143+ that moved to the spindle upstream, and the runner does the same (§10).
144+- Manual triggers (appview "Run", re-run, `ci trigger`) go to the runner's
145+ `sh.tangled.ci.triggerPipeline`, as upstream.
146+- Event log: SQLite `(seq INTEGER PRIMARY KEY, nsid, rkey, json,
147+ created_at)`. Reconnect and crash recovery are "resume from cursor".
148+- Routing: one runner, configured in the bridge. (Tangled's model — the repo
149+ names its spindle — is the extension point if a second runner ever exists.)
150+
151+## 6. Workflow manifest
152+
153+`.tangled/workflows/<name>.yml`. Pure data, validated against
154+`run.rtw.ci.workflow`. No conditionals, expressions or loops.
155+
156+```yaml
157+when: # matched by the runner when a ref update arrives
158+ - event: [push]
159+ branch: [main]
160+engine: devenv # Tangled's spindle ignores unknown engines
161+repos: # extra checkouts, pinned per run
162+ - { repo: servers, path: ../servers }
163+ - { repo: data, path: ../data, env: SECRETS_DIR } # exports the path
164+
165+steps: [test, plan, deploy, health-check, record, notify] # the happy path
166+
167+actions:
168+ test: { command: just test }
169+ plan: { dir: ../servers, command: just plan-project sovrn }
170+ deploy: { dir: ../servers, command: just deploy-project sovrn,
171+ gate: { prompt: "Deploy sovrn?" }, concurrency: fleet,
172+ audited: true }
173+ health-check: { dir: ../servers, command: just prod-health-check sovrn }
174+ record: { dir: ../servers, command: just record-deploy sovrn }
175+ notify: { command: just notify-deploy }
176+ rollback: { dir: ../servers, command: just rollback, style: danger,
177+ manual: true, audited: true,
178+ gate: { prompt: "Roll back sovrn?" } }
179+```
180+
181+A smaller project needs only what it uses:
182+
183+```yaml
184+when: [{ event: [push], branch: [main] }]
185+engine: devenv
186+steps: [test, deploy]
187+actions:
188+ test: { command: devenv test }
189+ deploy: { command: just deploy infra.mymood.at, gate: { prompt: "Deploy moods?" } }
190+```
191+
192+Action fields
193+
194+| Field | Meaning |
195+|---|---|
196+| `command` | Shell command, run as `devenv shell -- bash -euo pipefail -c '<command>'` |
197+| `dir` | Working directory relative to the repo checkout (default `.`) |
198+| `gate` | `{ prompt, timeout }` — always ask a human before running (default timeout 24h) |
199+| `concurrency` | Named lock; `{ group, supersede }` — supersede cancels older runs waiting at a gate for the same group |
200+| `manual` | Show as a button on the run page at any time |
201+| `style` | `normal` or `danger` (rendering only) |
202+| `audited` | Local runs that execute this action report to the appview (§10) |
203+
204+The set of actions is closed: results may only route to declared actions.
205+That makes the manifest an honest list of what the workflow can do, and it is
206+what the UI renders as buttons.
207+
208+## 7. Step result contract
209+
210+stdout/stderr are logs. The result is written to the file at `$CI_RESULT`.
211+
212+```json
213+{
214+ "version": 1,
215+ "status": "failure",
216+ "summary": "mx99.eu.sovrn.at /healthz 503 for 4m (stalwart: connection refused)",
217+ "next": [
218+ { "action": "rollback", "prompt": "mx99 degraded after deploy. Roll back sovrn?", "then": "stop" },
219+ { "action": "health-check", "label": "Check again" }
220+ ],
221+ "gated": true,
222+ "attachments": [ { "label": "health probes", "path": "health.ndjson", "type": "ndjson" } ],
223+ "outputs": { "hosts": ["mx99.eu.sovrn.at"] }
224+}
225+```
226+
227+| Field | Meaning |
228+|---|---|
229+| `status` | `success` · `warning` · `failure` |
230+| `summary` | Markdown shown on the run page, at gates and in notifications |
231+| `next` | Omitted → default routing. `"stop"` → end the run. Otherwise one option or a list |
232+| `next[].action` | A declared action (happy-path step or not) |
233+| `next[].then` | After that action succeeds: `continue` (resume the happy path) or `stop`. Default: `continue` for happy-path steps, `stop` otherwise |
234+| `next[].prompt`, `label`, `inputs` | Text for the person; data passed to the action as `$CI_INPUTS` |
235+| `gated` | Ask a human before taking `next` |
236+| `attachments` | Files under `$CI_ARTIFACTS`, rendered by type (`ndjson`, `text`, `markdown`) |
237+| `outputs` | Merged into `$CI_STATE/outputs.json` |
238+
239+Fallbacks and errors
240+
241+- No result file: exit 0 → `{status: success}`; non-zero → `{status: failure,
242+ next: "stop"}`. Plain recipes work unchanged.
243+- Invalid JSON, schema mismatch, an undeclared action, a path outside
244+ `$CI_ARTIFACTS`, or non-zero exit claiming success → treated as failure with
245+ the contract error shown.
246+- Each step runs with a private `/tmp`; anything worth keeping goes in
247+ `$CI_ARTIFACTS` or `$CI_STATE`.
248+
249+A `ci-result` helper on `PATH` writes valid results from shell:
250+
251+```sh
252+ci-result fail --summary "mx99 /healthz 503 for 4m" \
253+ --next rollback --prompt "Roll back sovrn?" --then stop \
254+ --next health-check --label "Check again" \
255+ --attach "health probes=health.ndjson"
256+```
257+
258+## 8. Routing semantics
259+
260+The run keeps a **cursor** into `steps`.
261+
262+```
263+run action → read result
264+ ├─ next omitted → success: happy-path action → next step; other action → stop
265+ │ failure: stop
266+ ├─ next = "stop" → end run
267+ └─ next = [A, …] → gate if (any option's action declares gate) OR result.gated OR >1 option
268+ gate offers each option + Dismiss (end run with this step's status)
269+ no gate → run A
270+ naming a happy-path step moves the cursor there (forward skips, backward repeats)
271+ after A succeeds: then=continue → next step after cursor; then=stop → end run
272+```
273+
274+Rules
275+
276+1. **Gates are monotonic**: a declared `gate` can't be bypassed by a result.
277+ A choice between several options always needs a person.
278+2. Off-path actions end the run unless their `then` or their own result says
279+ otherwise. Nothing slides back onto the happy path by accident.
280+3. The chosen action receives `$CI_TRIGGER_RESULT` (the result that routed to
281+ it), `$CI_INPUTS`, and the shared `$CI_STATE`.
282+4. Loop guard: at most 50 transitions per run (configurable).
283+5. Final status: `success` only if every happy-path step completed; otherwise
284+ `failure`, annotated with the off-path actions that ran (e.g. "rolled back
285+ via rollback"). This maps onto `sh.tangled.pipeline.status`.
286+6. Gates always notify, and runs that end unsuccessfully always notify
287+ (runner config, not code).
288+7. Approving a gate or running a `danger` action requires the owner's logged-in
289+ session. The agent API token (used from `/rc`) can read, trigger and re-run,
290+ but not approve.
291+
292+## 9. Engine (`ci`)
293+
294+One Go binary. Repos pin it as a devenv input (`packages = [ inputs.ci… ]`), so
295+the dev machine and the runner use the identical engine for a given commit.
296+
297+Commands
298+
299+```sh
300+ci validate # manifest + actions resolvable; schemas
301+ci run <wf> # from the working copy; gates prompt in the terminal
302+ci run <wf> --clean [--rev @-] # fresh worktree of a commit, as the runner does
303+ci run <wf> --until plan # stop before an action: safe rehearsal
304+ci run <wf> --stub health-check=fixtures/degraded.json # fake a result, test routing
305+ci run <wf> --detach # exit at gates, as on the runner
306+ci action <name> --no-follow # run one action, print result + what routing would do
307+ci resume <run> [--choose rollback] # answer a gate / continue a paused run
308+ci action rollback --run <run> # manual action against an earlier run's state
309+ci view <run> # render events.ndjson
310+ci trigger <repo> <wf> [--rev] # ask the bridge for a manual pipeline
311+ci wait <repo> <sha> # block until done; print failing step's logs
312+```
313+
314+A local `ci run deploy` deploys production for real, behind the same gates.
315+`--until` and `--stub` are the rehearsal tools.
316+
317+Environment contract (versioned with the result schema)
318+
319+| Variable | Contents |
320+|---|---|
321+| `CI_RUN_DIR` | The run's directory |
322+| `CI_RESULT` | Where this action writes its result |
323+| `CI_STATE` | Shared state for the run's actions |
324+| `CI_ARTIFACTS` | Attachment directory |
325+| `CI_INPUTS` | Data from the `next` entry that routed here |
326+| `CI_TRIGGER_RESULT` | The result that routed here |
327+| `CI_SHA`, `CI_REF`, `CI_REPO` | What's being run |
328+| `CI_RUN_ID`, `CI_ACTION` | Identity of the run and action |
329+| `CI_HOST` | `local` or `runner`. Informational; recipes must not branch on it |
330+| `CI`, `TANGLED_*` | `CI=true` and upstream's `TANGLED_SHA`, `TANGLED_REF`, `TANGLED_REF_NAME`, `TANGLED_REF_TYPE`, `TANGLED_PIPELINE_ID`, `TANGLED_PIPELINE_KIND`, `TANGLED_REPO_*`, set in both modes for compatibility with upstream workflows |
331+| per-repo `env:` | Paths exported by `repos:` entries (e.g. `SECRETS_DIR`) |
332+
333+Run directory
334+
335+```
336+<run>/manifest.json state.json (cursor, history, pending gate)
337+ events.ndjson (log lines, results, gates, transitions — one format everywhere)
338+ state/ artifacts/ results/<n>-<action>.json
339+ <repo>/ <extra repos>/ (worktrees; layout mirrors ~/projects)
340+```
341+
342+Repos
343+
344+- `repos:` entries are cloned at the pushed revision of their default branch
345+ (or a pinned rev) in both modes; the shas are recorded in `events.ndjson`.
346+- Paths are relative to the main checkout. Keep the runner's layout identical
347+ to `~/projects` so `../servers` resolves the same in both places.
348+- `--repo name=/path` overrides a checkout locally (explicit; off by default).
349+
350+Segments: a gate ends the process. The engine runs actions until it reaches a
351+gate, writes a `gate` event, saves `state.json` and exits with a "paused"
352+status. Locally it prompts inline instead (unless `--detach`). On the runner
353+the daemon publishes the gate and later starts `ci resume <run> --choose X` in
354+a new unit. Nothing runs while a gate waits, and runner restarts never lose a
355+run.
356+
357+Concurrency: `flock` on a lock directory (local state dir, or
358+`/var/lib/ci/locks` on the runner). Optional later: ask the runner for the lock
359+when reachable, so a laptop deploy and a CI deploy can't overlap.
360+
361+## 10. Runner daemon (CI host)
362+
363+- Subscribes to the bridge `/events` and keeps its cursor in SQLite.
364+- On `sh.tangled.git.refUpdate`: honour `skip-ci`/`ci-skip` push options,
365+ fetch the repo into its bare mirror, read `.tangled/workflows/*.yml` at
366+ `newSha`, keep manifests with `engine: devenv` whose `when:` matches
367+ (events `push` and `manual`; `branch`/`tag` globs with `*` and `**`, as
368+ upstream), and emit one `sh.tangled.pipeline` record with the raw YAML of
369+ each matching workflow. `pull_request` and `schedule` are not supported yet.
370+- Per run: creates the run dir and worktrees (from a persistent bare mirror
371+ per repo), then starts a segment as a transient unit:
372+
373+```
374+systemd-run --unit=ci-<run>-<seg> --slice=ci.slice -p User=ci \
375+ -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes \
376+ -p ReadWritePaths=<run> <cache> -p LoadCredential=age-identity:<path> \
377+ pasta --config-net -- devenv shell -- ci run|resume …
378+```
379+
380+- Tails `<run>/events.ndjson` and serves it as `/logs`, plus status, gate and
381+ transition records on `/events`. Translates run state into
382+ `sh.tangled.pipeline.status`.
383+- On restart, re-attaches to running units by name and resumes tailing.
384+ Transient units survive daemon restarts, including `switch-to-configuration`
385+ during a self-deploy.
386+
387+Isolation (protects the host's config and services, not against attack)
388+
389+| Concern | Control |
390+|---|---|
391+| Writes to `/etc`, units, users, system profile | `ProtectSystem=strict`, non-root `ci` user, not in `trusted-users`. Only an explicit deploy action can change host config |
392+| Port clashes (devenv postgres vs. production postgres) | Network namespace per segment via `pasta`, with egress everywhere |
393+| CPU / memory / IO starvation | `ci.slice` with low `CPUWeight`/`IOWeight`, `MemoryMax`; swap or zram so builds can't OOM soft-serve; 1–2 concurrent runs |
394+| Disk | Run dirs expire after N days; host `nix.gc` and `min-free`/`max-free` |
395+| Nix store | Shared through the host nix-daemon socket; append-only, so growth is disk usage, not pollution |
396+
397+Multi-arch: jobs use the host nix-daemon, which routes derivations by
398+`system`: `boot.binfmt.emulatedSystems = [ "aarch64-linux" ]` (as the fleet
399+already does with `buildOnTarget = false`), and optionally `nix.buildMachines`
400+for a dedicated ARM builder if emulation is too slow (not a production host).
401+Steps run natively on x86_64; aarch64 work goes through `nix build` /
402+`colmena build` / flake `checks`.
403+
404+Workspaces: per-run worktrees (a gate can hold a run open for hours); `HOME`
405+and `.cache` shared per repo (accepted shared state). `.devenv/` is per run;
406+activation still hits a warm store. Host substituters include
407+`devenv.cachix.org`, since the `ci` user can't add substituters itself.
408+
409+Host sizing: `infra.rtw.run` is 2 vCPU x86_64. Resize to 4–8 vCPU / 8–16 GB
410+and stay x86_64 (matches the dev machine and most of the fleet). Do it with
411+fleet plan step 2.8 (rebuild on the standard layout).
412+
413+## 11. Appview
414+
415+Minimal server-rendered HTML (Go templates + htmx), reachable from a phone.
416+
417+- **Pipelines**: repo, sha, per-workflow status; live logs over SSE (proxied
418+ from the runner's `/logs`).
419+- **Run page**: timeline of actions and transitions, summaries, attachments,
420+ manual action buttons (`danger` rendered big and red).
421+- **Gates**: prompt, the summaries of earlier actions (e.g. an `nvd` diff), one
422+ button per option, Dismiss. Approval requires the owner's session.
423+- **History**: past runs, including reported local runs.
424+- **API**: JSON status and plain-text logs for agents; `ci wait` uses it.
425+ Agent tokens can read, trigger and re-run, not approve gates or run `danger`
426+ actions.
427+- **ntfy**: gate waiting, run failed, run succeeded; *view* actions link to the
428+ page (no auth tokens inside notifications).
429+- **Local run reporting**: `ci run --report` (automatic when an `audited`
430+ action runs) publishes a local run's events, so laptop deploys appear in the
431+ history and their rollback buttons work.
432+- Later: ATProto OAuth login, so approvals carry the owner's DID.
433+
434+## 12. Secrets
435+
436+- The encrypted vault (`data` repo) is a `repos:` entry exported as
437+ `SECRETS_DIR`. It is cloned at its pushed revision in both modes, so only
438+ committed, pushed secrets run anywhere, and the vault sha is recorded per
439+ run.
440+- Colmena and Justfiles call `secrets decrypt <selector>` exactly as today.
441+ The engine has no secrets concept and does no masking; recipes pipe secrets
442+ into tools and never print them.
443+- The `secrets` tool gains (first implementation step):
444+ - `$SECRETS_DIR/.recipients`: `encrypt` targets every listed recipient.
445+ - `secrets rekey`: re-encrypt every secret to the current recipients.
446+ - `SECRETS_IDENTITY`: identity file override (default
447+ `~/.config/age/identity.txt`).
448+- The dev machine and the runner use **different** age identities; secrets
449+ are encrypted to both and rekeyed on demand. Revoking the runner = remove
450+ its recipient, rekey, rotate what it could read.
451+- The runner's identity is itself a vault secret (`ci/age-identity`),
452+ delivered by Colmena `deployment.keys` (`keyCommand = [ "secrets"
453+ "decrypt" "ci/age-identity" ]`) to a root-only path, and handed to segments
454+ via `LoadCredential=`; the engine sets `SECRETS_IDENTITY` to
455+ `$CREDENTIALS_DIRECTORY/age-identity`.
456+- Repo access: the runner's forge deploy key (it lives on the forge host, so a
457+ local path also works).
458+
459+## 13. CD pattern for Colmena projects (lives in project repos, not the engine)
460+
461+`servers/Justfile` recipes that any Colmena fleet can copy:
462+
463+| Recipe | Does | Result |
464+|---|---|---|
465+| `plan-project P` | `nix flake update P`; `colmena build` affected hosts; `nvd diff` against `readlink /run/current-system` | summary = diffs |
466+| `deploy-project P` | Record each host's current system to `$CI_STATE/prev-systems.json`; `colmena apply --on <hosts>`; CI host last | — |
467+| `prod-health-check P` | Poll the project's health URLs (status code only, as sovrn's `/healthz` contract defines); N consecutive passes within a window | success, or failure offering `rollback` / "Check again", gated, with probes attached |
468+| `rollback` | Per host from `prev-systems.json`: `nix-env -p /nix/var/nix/profiles/system --set <prev> && <prev>/bin/switch-to-configuration switch` | — |
469+| `record-deploy P` | Commit and push `flake.lock` (the lock bump stays the deploy log) | — |
470+
471+Notes
472+
473+- Health URLs come from code (a flake output such as
474+ `ci.hosts.<fqdn>.healthChecks`, or `hosts.json`). The existing monitor can
475+ later read the same list.
476+- Rollback needs no evaluation or build, so it works with a broken flake.
477+ It does not undo state (database migrations) or uploaded keys; migrations
478+ must be backward-compatible, or recovery is roll-forward.
479+- The CI host itself deploys last, ideally in its own gated step. If a
480+ self-deploy breaks the appview, recovery is SSH.
481+- `servers` flake inputs move from `git+file:///home/…` to forge URLs; use
482+ `--override-input` locally for unpushed work.
483+
484+## 14. Local vs CI parity
485+
486+| | Local default | Runner | Narrowing the gap |
487+|---|---|---|---|
488+| Engine, manifest, recipes, devenv | identical (pinned by commit) | identical | — |
489+| Source | working copy | clean worktree at sha | `--clean --rev` |
490+| Extra repos, vault | clean clones at pushed revs | same | `--repo` override |
491+| Sandbox | none, your user | `ci` user, ProtectSystem, pasta, slice | `--isolated` (`systemd-run --user` + pasta) |
492+| Nix config | your substituters, binfmt | runner's | keep both in a shared config fragment |
493+| Secrets identity | yours | runner's | both are vault recipients |
494+| Locks | local flock | runner flock | optional remote lock |
495+| History | local run dir | appview | `--report` / `audited` |
496+
497+## 15. Implementation order
498+
499+Tracked as issues in this repo under `eae4f87` (`bug ls`), one at a time:
500+
501+1. `f25b56a` `secrets`: multiple recipients, `rekey`, `SECRETS_IDENTITY`.
502+2. `5434405` Contracts and project skeleton: lexicons, generated types, Go module.
503+3. `de85a99` Engine core: manifest, action execution, result contract, routing, events, local CLI.
504+4. `b9fb99a` Engine: repo checkouts at pinned revisions, `env:` exports, `--clean`.
505+5. `dea75b4` Engine: gates, pause/resume, manual actions, concurrency, loop guard.
506+6. `65e8495` Adopt locally: sovrn, servers, moods workflows and recipes; local gated CD works end to end.
507+7. `de35782` Bridge.
508+8. `7a3d5cf` Runner daemon.
509+9. `7f8343d` Appview.
510+10. `db62e70` Deploy onto the forge host: NixOS modules, host resize/rebuild, CI identity via Colmena.
511+11. `adca87e` Cutover and end-to-end validation (phone + `/rc` flow, self-deploy).
512+12. `78c84e0` Local run reporting and cross-host locks.
513+
514+Later: remote aarch64 builder; contributing a `devenv` engine to upstream's
515+spindle (it now has a microVM engine and pluggable engines); ATProto OAuth login; service-auth JWTs; monitor
516+reading health checks; nspawn/microvm engines for untrusted code; publishing
517+records to a PDS.
+255,
-0
1@@ -0,0 +1,255 @@
2+{
3+ "nodes": {
4+ "bug": {
5+ "inputs": {
6+ "flake-utils": "flake-utils",
7+ "nixpkgs": "nixpkgs"
8+ },
9+ "locked": {
10+ "lastModified": 1792141974,
11+ "narHash": "sha256-lXRuKVHgvawLKlRnVtxYfo80BUrDr4Y0+zp5OF+xaik=",
12+ "ref": "refs/heads/main",
13+ "rev": "f07be9c96e9b13fda86600af842e8ed6c0747ef1",
14+ "revCount": 16,
15+ "type": "git",
16+ "url": "https://git.kilimanjaro.io/bug"
17+ },
18+ "original": {
19+ "type": "git",
20+ "url": "https://git.kilimanjaro.io/bug"
21+ }
22+ },
23+ "devenv": {
24+ "locked": {
25+ "dir": "src/modules",
26+ "lastModified": 1792071910,
27+ "narHash": "sha256-DUCfod12b6e/r9FCjE3+x7mJlVEmIpC10BZKGz+Nrsw=",
28+ "owner": "cachix",
29+ "repo": "devenv",
30+ "rev": "1566050ec14ce238983376e118af9fb220454d39",
31+ "type": "github"
32+ },
33+ "original": {
34+ "dir": "src/modules",
35+ "owner": "cachix",
36+ "repo": "devenv",
37+ "type": "github"
38+ }
39+ },
40+ "devkit": {
41+ "flake": false,
42+ "locked": {
43+ "lastModified": 1792142785,
44+ "narHash": "sha256-wXEnZQY5ocyuFQ82KQ7faLJNjEso6rFNw4MDYKh1BBg=",
45+ "ref": "refs/heads/main",
46+ "rev": "68bd7f79bfe95a8b08f22b26ebcdc1dcdff4204f",
47+ "revCount": 5,
48+ "type": "git",
49+ "url": "https://git.kilimanjaro.io/devkit"
50+ },
51+ "original": {
52+ "type": "git",
53+ "url": "https://git.kilimanjaro.io/devkit"
54+ }
55+ },
56+ "flake-utils": {
57+ "inputs": {
58+ "systems": "systems"
59+ },
60+ "locked": {
61+ "lastModified": 1731533236,
62+ "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
63+ "owner": "numtide",
64+ "repo": "flake-utils",
65+ "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
66+ "type": "github"
67+ },
68+ "original": {
69+ "owner": "numtide",
70+ "repo": "flake-utils",
71+ "type": "github"
72+ }
73+ },
74+ "flake-utils_2": {
75+ "inputs": {
76+ "systems": "systems_2"
77+ },
78+ "locked": {
79+ "lastModified": 1731533236,
80+ "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=",
81+ "owner": "numtide",
82+ "repo": "flake-utils",
83+ "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b",
84+ "type": "github"
85+ },
86+ "original": {
87+ "owner": "numtide",
88+ "repo": "flake-utils",
89+ "type": "github"
90+ }
91+ },
92+ "local-code-review": {
93+ "inputs": {
94+ "nixpkgs": "nixpkgs_2"
95+ },
96+ "locked": {
97+ "lastModified": 1790664642,
98+ "narHash": "sha256-Pl7zVsnoxxrbrg/up/5Ue2a9ajtROQptM4AOuo7AYts=",
99+ "ref": "refs/heads/main",
100+ "rev": "95ae244a839007c6aa89fa5230be193d878ee56e",
101+ "revCount": 17,
102+ "type": "git",
103+ "url": "https://git.kilimanjaro.io/local-code-review"
104+ },
105+ "original": {
106+ "type": "git",
107+ "url": "https://git.kilimanjaro.io/local-code-review"
108+ }
109+ },
110+ "nixpkgs": {
111+ "locked": {
112+ "lastModified": 1789149629,
113+ "narHash": "sha256-H6GwaZzZf+4npqv0tph94w9tZddSjFjmQrVsW0z78uk=",
114+ "owner": "NixOS",
115+ "repo": "nixpkgs",
116+ "rev": "eaad089433ca2bb662274377d33df3d0e51ef28b",
117+ "type": "github"
118+ },
119+ "original": {
120+ "owner": "NixOS",
121+ "ref": "nixos-unstable",
122+ "repo": "nixpkgs",
123+ "type": "github"
124+ }
125+ },
126+ "nixpkgs-src": {
127+ "flake": false,
128+ "locked": {
129+ "lastModified": 1791487292,
130+ "narHash": "sha256-tqM7xDEsaeK1p6VvRygeZ9LkQRpUUkbRT02BHv6hnTM=",
131+ "owner": "NixOS",
132+ "repo": "nixpkgs",
133+ "rev": "8edc0c72e3a38faf5434e40d1f19431125fe4b30",
134+ "type": "github"
135+ },
136+ "original": {
137+ "owner": "NixOS",
138+ "ref": "nixpkgs-unstable",
139+ "repo": "nixpkgs",
140+ "type": "github"
141+ }
142+ },
143+ "nixpkgs_2": {
144+ "locked": {
145+ "lastModified": 1790323409,
146+ "narHash": "sha256-VVTPf+Hyd5ebpjBMHmrLMSBIeW6ls48Bqtosj7CNKLA=",
147+ "owner": "NixOS",
148+ "repo": "nixpkgs",
149+ "rev": "e94cb152ed51bd6e24eb4a41f1460252beb52cd2",
150+ "type": "github"
151+ },
152+ "original": {
153+ "owner": "NixOS",
154+ "ref": "nixos-unstable",
155+ "repo": "nixpkgs",
156+ "type": "github"
157+ }
158+ },
159+ "nixpkgs_3": {
160+ "inputs": {
161+ "nixpkgs-src": "nixpkgs-src"
162+ },
163+ "locked": {
164+ "lastModified": 1791812882,
165+ "narHash": "sha256-4AdYH2HoD21mys5lk9NoDcM2Z7iPeEUjxJeGE3Pw278=",
166+ "owner": "cachix",
167+ "repo": "devenv-nixpkgs",
168+ "rev": "760b85220489f69855e6df78a6d1dad5873f8bf6",
169+ "type": "github"
170+ },
171+ "original": {
172+ "owner": "cachix",
173+ "ref": "rolling",
174+ "repo": "devenv-nixpkgs",
175+ "type": "github"
176+ }
177+ },
178+ "nixpkgs_4": {
179+ "locked": {
180+ "lastModified": 1790578696,
181+ "narHash": "sha256-ZoxIApko70jCdbH3l20HWXOBaT2HZd87orzd2yJ9dVE=",
182+ "owner": "NixOS",
183+ "repo": "nixpkgs",
184+ "rev": "7a0f122f5090cf4c2ade2a13a0e229d4e19ba71f",
185+ "type": "github"
186+ },
187+ "original": {
188+ "owner": "NixOS",
189+ "ref": "nixos-unstable",
190+ "repo": "nixpkgs",
191+ "type": "github"
192+ }
193+ },
194+ "root": {
195+ "inputs": {
196+ "bug": "bug",
197+ "devenv": "devenv",
198+ "devkit": "devkit",
199+ "local-code-review": "local-code-review",
200+ "nixpkgs": "nixpkgs_3",
201+ "secrets": "secrets"
202+ }
203+ },
204+ "secrets": {
205+ "inputs": {
206+ "flake-utils": "flake-utils_2",
207+ "nixpkgs": "nixpkgs_4"
208+ },
209+ "locked": {
210+ "lastModified": 1790680131,
211+ "narHash": "sha256-p3xPff0d3O4T8AW0Zbv7sZDZcJk61f5LEHRKcSTRF58=",
212+ "ref": "refs/heads/main",
213+ "rev": "96a2b2da934d769c76c19073e89e19e231964868",
214+ "revCount": 1,
215+ "type": "git",
216+ "url": "https://git.kilimanjaro.io/secrets"
217+ },
218+ "original": {
219+ "type": "git",
220+ "url": "https://git.kilimanjaro.io/secrets"
221+ }
222+ },
223+ "systems": {
224+ "locked": {
225+ "lastModified": 1681028828,
226+ "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
227+ "owner": "nix-systems",
228+ "repo": "default",
229+ "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
230+ "type": "github"
231+ },
232+ "original": {
233+ "owner": "nix-systems",
234+ "repo": "default",
235+ "type": "github"
236+ }
237+ },
238+ "systems_2": {
239+ "locked": {
240+ "lastModified": 1681028828,
241+ "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
242+ "owner": "nix-systems",
243+ "repo": "default",
244+ "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e",
245+ "type": "github"
246+ },
247+ "original": {
248+ "owner": "nix-systems",
249+ "repo": "default",
250+ "type": "github"
251+ }
252+ }
253+ },
254+ "root": "root",
255+ "version": 7
256+}
+14,
-0
1@@ -0,0 +1,14 @@
2+{ ... }:
3+
4+# Scaffold for a new project. devkit (imported in devenv.yaml) already provides
5+# bug, secrets, local-code-review, jujutsu, and the customized interactive bash
6+# environment, so don't repeat those here.
7+{
8+ packages = [
9+ # project-specific packages
10+ ];
11+
12+ # enterShell = ''
13+ # project-specific setup; concatenated onto devkit's
14+ # '';
15+}
+19,
-0
1@@ -0,0 +1,19 @@
2+# New-project scaffold. Copied by `nix flake init -t git+https://git.kilimanjaro.io/devkit`.
3+#
4+# devenv#2205: the three tool inputs must be declared here (devkit's own
5+# devenv.yaml is not evaluated when imported).
6+inputs:
7+ nixpkgs:
8+ url: github:cachix/devenv-nixpkgs/rolling
9+ bug:
10+ url: git+https://git.kilimanjaro.io/bug
11+ local-code-review:
12+ url: git+https://git.kilimanjaro.io/local-code-review
13+ secrets:
14+ url: git+https://git.kilimanjaro.io/secrets
15+ devkit:
16+ url: git+https://git.kilimanjaro.io/devkit
17+ flake: false
18+imports:
19+ - devkit
20+reload: false