DESIGN.md

Nix-native CI/CD for Tangled — Design

Status: design agreed, not yet implemented (2026-10-16).

A Tangled.org-compatible CI/CD system for a self-hosted soft-serve forge, built on Nix, devenv, NixOS and Colmena. It speaks Tangled’s lexicons and ATProto semantics (DIDs, AT-URIs, lexicon-typed records, cursor-resumable event streams), runs workflows in each repo’s devenv shell, and gates deploys behind a human decision that can be made from a phone.

1. Goals and non-goals

Goals

• Push to soft-serve → workflow runs → status and live logs visible in a minimal web UI, usable from a phone.
• Gated CD: a person approves deploys; health is reported, rollback is one button. Deploys can be triggered from a remote Claude session (/rc) without the dev machine.
• Local parity: the exact same flow runs from the command line on the dev machine. No push-and-pray loop.
• Generic: nothing in the system knows about sovrn, moods, Colmena or NixOS deploys. Any project can reuse it.
• Multi-arch (x86_64 + aarch64) from one runner, through Nix.
• Interoperate with Tangled’s protocol: pipeline and status records on Tangled-shaped event streams.

Non-goals (for now)

• Running untrusted code. Only the owner’s code runs; isolation exists to keep CI from changing the host’s config or competing with its services, not to contain an attacker.
• Multiple runners, autoscaling, job claiming.
• A PDS, relay or Jetstream. Records are served by the services that own them.
• A polished UI.

2. Principles (the decisions behind this design)

1. No programming in YAML. The workflow file is an inventory of actions plus the default order. All logic (what to show, what happens next, when to ask a human) lives in code — Nix, shell, Justfiles — and reaches the engine through a typed JSON result contract.
2. The engine is the product; the runner is a host for it. The same ci binary runs locally and on the runner, pinned per repo through devenv.
3. Steps are the happy path; actions are everything the workflow can do. A step’s result can route to any declared action, including happy-path steps.
4. Gates are monotonic. Code can add a gate, never remove one.
5. Health is notify-only. Transient errors are common after a deploy; the operator decides whether to roll back.
6. Secrets are a repo. An encrypted vault cloned at a known path; the secrets tool behaves identically everywhere. The engine has no secrets concept.
7. Simplicity over isolation strength. Sandboxed systemd units, not VMs, until something needs more.

3. System overview

 forge host (infra.rtw.run, initially everything lives here)
 ┌──────────────────────────────────────────────────────────────────────────┐
 │ soft-serve ──post-receive──► bridge ── /events (sh.tangled.git.refUpdate)┐│
 │                                                                         ▼│
 │ runner daemon ── reads .tangled/workflows @ newSha, matches `when:`      │
 │    │             per run: worktrees, sandboxed unit ─► ci (engine)        │
 │    │              (ci.slice, user ci, pasta netns)       in devenv shell │
 │    │  tails <run>/events.ndjson                                          │
 │    └─► /events (sh.tangled.pipeline[.status], run.rtw.ci.*) , /logs/...  │
 │                                                                          │
 │ appview ◄── bridge /events + runner /events,/logs ── SQLite index        │
 │    └─► browser / phone (Tailscale or authed public), ntfy, JSON+text API │
 │                                                                          │
 │ host nix-daemon ─► binfmt aarch64-linux (─► optional remote builder)     │
 └──────────────────────────────────────────────────────────────────────────┘
 dev machine:  `ci run <workflow>` — same engine, same manifest, same recipes

The bridge, runner daemon and appview are three roles of one Go binary. They can run as one service but keep real websocket boundaries, so they can be split across hosts later.

4. Protocol and identity

• Each service has a did:web (e.g. did:web:git.kilimanjaro.io for the bridge, did:web:ci.rtw.run for the runner) serving a DID document.
• Repos have DIDs, as upstream Tangled repos now do. A soft-serve repo is did:web:<forge-host>:<repo> (e.g. did:web:git.kilimanjaro.io:sovrn). It is an identifier first; serving its DID document is optional.
• Records use TIDs as keys and reference each other by AT-URI. Following the spindle’s convention, a pipeline is at://did:web:<knot-host>/sh.tangled.pipeline/<tid>.
• Records are not written to a PDS. Each service serves its own records on a cursor-resumable websocket, as Tangled’s knot and spindle do:
  • GET /events?cursor=N — replay events with created > N, then tail. The envelope is upstream’s eventstream.Event: {rkey, nsid, event, created}, where created is a monotonic local clock value and doubles as the cursor.
  • GET /logs/{knot}/{rkey}/{workflow} — replay the log, then tail (runner). Messages are upstream’s spindle/models.LogLine JSON (kind: data | control, content, time, step_id, stream, step_status, step_kind, step_command), which the upstream appview reads. (The newer sh.tangled.ci.subscribePipelineLogs XRPC sends the same content as CBOR #control/#data frames; not implemented.)
• Tangled lexicons used as-is (checked against tangled core e0ef7981d, 2026-10-11):
  • sh.tangled.git.refUpdate — emitted by the bridge, as a knot does (ref, committerDid, ownerDid, repo, oldSha, newSha, changedFiles, pushOptions, meta{isDefaultRef, commitCount}).
  • sh.tangled.pipeline and sh.tangled.pipeline.status — emitted by the runner on /events. Upstream marks both records deprecated in favour of sh.tangled.ci.pipeline, but the spindle still emits them on /events and the appview still consumes them, so they remain the compatible choice.
  • sh.tangled.ci.* XRPC (queryPipelines, getPipeline, triggerPipeline, cancelPipeline, subscribePipelineLogs) is the newer spindle API. The runner implements triggerPipeline (manual runs) first; the queries are a later addition.
• Custom lexicons under run.rtw.ci.* (namespace is a placeholder; pick a domain you control):
  • run.rtw.ci.workflow — the manifest schema (§6)
  • run.rtw.ci.step.result — the step result contract (§7)
  • run.rtw.ci.gate — {pipeline, workflow, action, options[], state: waiting|approved|rejected|expired|superseded, actor, prompt}
  • run.rtw.ci.transition — engine routing decisions, for the timeline
  • run.rtw.ci.event — one line of a run’s events.ndjson Tangled clients ignore unknown NSIDs, so nothing breaks.
• Lexicons are the single schema source. Go types are hand-written for the handful of records used (Tangled’s api/tangled package pulls in the whole monorepo), and a test checks every Go struct against its lexicon’s properties so they cannot drift.
• Auth between services: bearer token over TLS (or Tailscale) to start. ATProto inter-service JWTs signed by the services’ DID keys later.

5. Bridge (forge side)

• A line appended to the existing soft-serve post-receive hook (which already rebuilds the pgit site) runs ci bridge notify --spool <dir> --server <bridge>. It reads git’s <old> <new> <ref> lines on stdin and soft-serve’s SOFT_SERVE_REPO_NAME/SOFT_SERVE_REPO_PATH, computes each update from the bare repo (changed files, commits per author, default ref, push options), writes one JSON file per push into the spool, pokes the bridge, and always exits 0: it never blocks or fails a push.
• The bridge server ingests the spool on start, on every poke and every 10 s, and deletes each file once its events are in the log; unparsable files are moved aside (.bad). A bridge that is down only delays events. Repos are did:web:<forge-host>:<repo>; ownerDid and committerDid are the configured owner DID (soft-serve users have no DIDs).
• For each pushed ref the bridge emits one sh.tangled.git.refUpdate event: repo DID, ref, old/new sha, changed files, push options (-o skip-ci is honoured by the runner, as upstream). The bridge does not read workflows; that moved to the spindle upstream, and the runner does the same (§10).
• Manual triggers (appview “Run”, re-run, ci trigger) go to the runner’s sh.tangled.ci.triggerPipeline, as upstream.
• Event log (internal/eventlog, shared with the runner): SQLite (created INTEGER PRIMARY KEY, rkey, nsid, event), where created is a monotonic UnixNano and the cursor, as upstream’s eventstream. Reconnect and crash recovery are “resume from cursor”. /events needs the bearer token when one is configured.
• Routing: one runner, configured in the bridge. (Tangled’s model — the repo names its spindle — is the extension point if a second runner ever exists.)

6. Workflow manifest

.tangled/workflows/<name>.yml. Pure data, validated against run.rtw.ci.workflow. No conditionals, expressions or loops.

when:                                   # matched by the runner when a ref update arrives
  - event: [push]
    branch: [main]
engine: devenv                          # Tangled's spindle ignores unknown engines
repos:                                  # extra checkouts, pinned per run
  - { repo: servers, path: ../servers }
  - { repo: data,    path: ../data, env: SECRETS_DIR }   # exports the path

steps: [test, plan, deploy, health-check, record, notify]   # the happy path

actions:
  test:         { command: just test }
  plan:         { dir: ../servers, command: just plan-project sovrn }
  deploy:       { dir: ../servers, command: just deploy-project sovrn,
                  gate: { prompt: "Deploy sovrn?" }, concurrency: fleet,
                  audited: true }
  health-check: { dir: ../servers, command: just prod-health-check sovrn }
  record:       { dir: ../servers, command: just record-deploy sovrn }
  notify:       { command: just notify-deploy }
  rollback:     { dir: ../servers, command: just rollback, style: danger,
                  manual: true, audited: true,
                  gate: { prompt: "Roll back sovrn?" } }

A smaller project needs only what it uses:

when: [{ event: [push], branch: [main] }]
engine: devenv
steps: [test, deploy]
actions:
  test:   { command: devenv test }
  deploy: { command: just deploy infra.mymood.at, gate: { prompt: "Deploy moods?" } }

Action fields

The set of actions is closed: results may only route to declared actions. That makes the manifest an honest list of what the workflow can do, and it is what the UI renders as buttons.

7. Step result contract

stdout/stderr are logs. The result is written to the file at $CI_RESULT.

{
  "version": 1,
  "status": "failure",
  "summary": "mx99.eu.sovrn.at /healthz 503 for 4m (stalwart: connection refused)",
  "next": [
    { "action": "rollback", "prompt": "mx99 degraded after deploy. Roll back sovrn?", "then": "stop" },
    { "action": "health-check", "label": "Check again" }
  ],
  "gated": true,
  "attachments": [ { "label": "health probes", "path": "health.ndjson", "type": "ndjson" } ],
  "outputs": { "hosts": ["mx99.eu.sovrn.at"] }
}

Fallbacks and errors

• No result file: exit 0 → {status: success}; non-zero → {status: failure, next: "stop"}. Plain recipes work unchanged.
• Invalid JSON, schema mismatch, an undeclared action, a path outside $CI_ARTIFACTS, or non-zero exit claiming success → treated as failure with the contract error shown.
• Each step runs with a private /tmp; anything worth keeping goes in $CI_ARTIFACTS or $CI_STATE.

A ci-result helper on PATH writes valid results from shell:

ci-result fail --summary "mx99 /healthz 503 for 4m" \
  --next rollback --prompt "Roll back sovrn?" --then stop \
  --next health-check --label "Check again" \
  --attach "health probes=health.ndjson"

8. Routing semantics

The run keeps a cursor into steps.

run action → read result
  ├─ next omitted  → success: happy-path action → next step; other action → stop
  │                   failure: stop
  ├─ next = "stop" → end run
  └─ next = [A, …] → gate if (any option's action declares gate) OR result.gated OR >1 option
                      gate offers each option + Dismiss (end run with this step's status)
                      no gate → run A
  naming a happy-path step moves the cursor there (forward skips, backward repeats)
  after A succeeds: then=continue → next step after cursor; then=stop → end run

Rules

1. Gates are monotonic: a declared gate can’t be bypassed by a result. A choice between several options always needs a person.
2. Off-path actions end the run unless their then or their own result says otherwise. Nothing slides back onto the happy path by accident.
3. The chosen action receives $CI_TRIGGER_RESULT (the result that routed to it), $CI_INPUTS, and the shared $CI_STATE.
4. Loop guard: at most 50 transitions per run (configurable).
5. Final status, by how the run ends:
   • it runs off the end of the happy path → success;
   • a result says next: "stop" → success, or failure if that result is a failure (the way for an action to end a run early on purpose);
   • an action’s then: stop → failure if that action is off the happy path (“diverted: stopped after rollback”), even when the last step had already started; for a happy-path step, success only if it was the last one;
   • a failure with no next → failure; a contract or engine error → failure; the loop guard → failure;
   • dismissing a declared gate reached by default routing → cancelled; dismissing a gate raised by a result → that result’s status (failure stays failure, success/warning → success); an expired gate, --until or Ctrl-C → cancelled. These map onto sh.tangled.pipeline.status (cancelled and failed).
6. Gates always notify, and runs that end unsuccessfully always notify (runner config, not code).
7. Approving a gate or running a danger action requires the owner’s logged-in session. The agent API token (used from /rc) can read, trigger and re-run, but not approve.

9. Engine (ci)

One Go binary. Repos pin it as a devenv input (packages = [ inputs.ci… ]), so the dev machine and the runner use the identical engine for a given commit.

Commands

ci validate                         # manifest + actions resolvable; schemas
ci run <wf>                         # from the working copy; gates prompt in the terminal
ci run <wf> --clean [--rev @-]      # fresh worktree of a commit, as the runner does
ci run <wf> --until plan            # stop before an action: safe rehearsal
ci run <wf> --stub health-check=fixtures/degraded.json   # fake a result, test routing
ci run <wf> --detach                # exit at gates, as on the runner
ci action <name> --no-follow        # run one action, print result + what routing would do
ci resume <run> [--choose rollback|N] [--dismiss] [--expire]   # answer a paused run's gate
ci action rollback --run <run>      # manual action against an earlier run's state
ci view <run>                       # render events.ndjson
ci runs                             # list local runs ($CI_RUNS_DIR, default ~/.local/state/ci/runs)
ci trigger <repo> <wf> [--rev]      # ask the runner for a manual pipeline
ci wait <repo> <sha>                # block until done; print failing step's logs

Actions run as devenv shell bash -euo pipefail -c 'cd "$CI_ACTION_DIR"; <command>' started in the main checkout, so the workflow’s repo pins the toolchain and the engine even when an action’s dir is another repo (../servers). --shell bash skips devenv, for debugging only. Exit codes: 0 success, 1 failure, 2 usage, 3 cancelled, 75 paused at a gate.

A local ci run deploy deploys production for real, behind the same gates. --until and --stub are the rehearsal tools.

Environment contract (versioned with the result schema)

Run directory

<run>/manifest.json        state.json   (cursor, history, pending gate)
      events.ndjson        (log lines, results, gates, transitions — one format everywhere)
      state/  artifacts/  results/<n>-<action>.json
      <repo>/  <extra repos>/   (worktrees; layout mirrors ~/projects)

Repos

• Layout: the main checkout is <run>/<repo> and each repos: entry is <run>/<repo>/<path>, which must be a sibling (../servers → <run>/servers). Locally, without --clean, <run>/<repo> is a symlink to the working copy, so uncommitted changes are visible; with --clean it is a fresh checkout of --rev (jj revset, default @; git rev, default HEAD), i.e. exactly what the VCS tracks.
• Extra repos are fetched into a bare mirror per repo ($CI_CACHE_DIR/mirrors/<repo>.git, default ~/.cache/ci) on every run, then checked out with git clone --shared at a detached commit: the forge’s default branch, or the entry’s rev. Checkouts borrow the mirror’s objects but are independent, so a run is deleted with rm -rf and nothing is written into any repository’s metadata. Each checkout’s origin is the forge URL, so recipes can push (git push origin HEAD:main from the detached checkout).
• Clone URLs are siblings of the main repo’s origin ([email protected]:sovrn → [email protected]:servers), or $CI_CLONE_URL with {repo} (e.g. git@host:{repo}).
• env: on an entry exports its checkout path (e.g. SECRETS_DIR).
• Every checkout’s commit is recorded as a checkout event in events.ndjson; linked directories are recorded as linked.
• --repo name=/path links an extra repo to a local directory instead (explicit; off by default).
• ci gc [--keep 168h] deletes finished runs older than --keep (never running or paused ones; symlinked working copies are only unlinked) and compacts the mirrors. Mirrors have gc.auto=0, so objects a live checkout borrows aren’t pruned under it.

Manual actions against a run: ci action --run <run> <action> (only manual: true actions, as in the appview) starts a child run that shares the parent’s checkouts and $CI_STATE (and gets $CI_PARENT_RUN). It runs the action, follows only what its result explicitly routes to, and ends with the action’s own status, so a rollback is recorded as its own run (“rollback done”) instead of rewriting the deploy run’s history. Declared gates apply.

Segments: a gate ends the process. The engine runs actions until it reaches a gate, writes a gate event, saves state.json and exits with a “paused” status. Locally it prompts inline instead (unless --detach). On the runner the daemon publishes the gate and later starts ci resume <run> --choose X in a new unit. Nothing runs while a gate waits, and runner restarts never lose a run.

Concurrency: flock on <runs>/.locks/<group>.lock (the runner passes /var/lib/ci/locks). A run takes a group’s lock when it first runs an action in the group and holds it until it ends or pauses at a gate, so a run waiting for a person never blocks others. supersede: true cancels older runs of the same repo and workflow waiting at a gate in that group when a newer run reaches one. Gates past expiresAt expire on ci resume (any answer) or ci resume --expire (the runner calls it on a timer). Local runs with the appview configured (CI_APPVIEW_URL, CI_TOKEN) take their groups from the runner instead: a lease on the same file lock the runner’s segments use (POST /api/locks/<group>, renewed every 30 s, released after 2 minutes without renewal), so a laptop deploy and a CI deploy can’t overlap. If the runner can’t be reached they warn once and lock locally; --local-locks forces that. Optional later: ask the runner for the lock when reachable, so a laptop deploy and a CI deploy can’t overlap.

10. Runner daemon (CI host)

• Subscribes to the bridge /events and keeps its cursor in SQLite.
• On sh.tangled.git.refUpdate: honour skip-ci/ci-skip push options, fetch the repo into its bare mirror, read .tangled/workflows/*.yml at newSha, keep manifests with engine: devenv whose when: matches (events push and manual; branch/tag globs with * and **, as upstream), and emit one sh.tangled.pipeline record with the raw YAML of each matching workflow. pull_request and schedule are not supported yet.
• Per run: creates the run dir and worktrees (from a persistent bare mirror per repo), then starts a segment as a transient unit:

systemd-run --unit=ci-<run>-<seg> --slice=ci.slice --uid=ci --gid=ci \
  -p ProtectSystem=strict -p ProtectHome=yes -p PrivateTmp=yes -p NoNewPrivileges=yes \
  -p "ReadWritePaths=<run> <cache>/home/<repo> <locks> /nix/var/nix/daemon-socket" \
  -p LoadCredential=age-identity:<path> -p StandardOutput=append:<run>/segment-<n>.log \
  --setenv=HOME=<cache>/home/<repo> \
  -- pasta --config-net -t none -u none -T none -U none -- ci runner-exec <run>

• Run ids are <pipeline-rkey>-<workflow>; manual action runs are <run>-<action>-<tid> (child runs, DESIGN §9). The daemon prepares the run directory and state.json (engine.Create, Host runner, Pipeline AT-URI), then each segment runs ci runner-exec <run-dir> (fresh, --from <action>, or --choose N | --dismiss | --expire), always pausing at gates. A runs table (in runner.db, next to the event log) tracks status, segment, unit and the events offset already published.
• Tails <run>/events.ndjson: gates and transitions become run.rtw.ci.gate / run.rtw.ci.transition records on /events; status becomes sh.tangled.pipeline.status (pending when prepared, running when the first segment starts, then success / failed / cancelled, or timeout for an expired gate). Waiting at a gate stays running. Manual action runs publish no Tangled status for the parent’s workflow.
• /logs/{knot}/{rkey}/{workflow} streams upstream LogLines from the events (control start/end per action execution, step_id = execution number, data per line) and closes when the run ends. /api/... serves runs, log.txt (plain text for agents), attachments, gate answers, manual actions and cancel, all behind one bearer token held by the appview; /xrpc/sh.tangled.ci.triggerPipeline starts manual pipelines.
• Segments are queued behind --max-concurrent slots; a segment that can’t start ends its run as failed. Gates past expiresAt are expired on a 30 s timer. Notifications (ntfy) go out for gates and failed runs (optionally successes), linking to the appview’s run page.
• On restart, Reconcile follows runs whose unit is still active, settles finished ones, starts never-started ones, and marks runs whose process is gone without a final state as failed (“ended without finishing”). Transient units survive daemon restarts, including switch-to-configuration during a self-deploy.
• Invalid workflow files are listed in the pipeline with a failed status and the validation error, so their author sees why.

Isolation (protects the host’s config and services, not against attack)

The runner daemon runs as root (it creates transient units for ci; granting ci polkit’s manage-units would let a segment start root units), hands each run directory and its repo’s HOME to ci before a segment starts, and is reachable on localhost only. Segments get a fixed PATH (engine, bash, coreutils, git, openssh, nix, jq, curl, just, devenv), NIX_REMOTE=daemon, CI_SELF_HOST (deploy recipes put that host last) and credentials: the age identity (SECRETS_IDENTITY) and an SSH key, found by the ci user’s static ~/.ssh/config (IdentityFile ${CREDENTIALS_DIRECTORY}/ssh-key; OpenSSH reads the passwd home, and ${VAR} expands per unit), with strict host keys except the forge on localhost. nix/module.nix packages all of this as services.ci.{bridge,runner,appview}; nix/tests/e2e.nix checks it in a VM.

Multi-arch: jobs use the host nix-daemon, which routes derivations by system. x86_64 work runs natively. aarch64 work (nix build / colmena build / flake checks) goes to a builder host as a remote builder: any host with the builder role in servers’ hosts.json (planned: mx99.eu.sovrn.at, the aarch64 staging host), listed in the CI host’s nix.buildMachines from that data (servers/roles/ci/builders.nix). boot.binfmt.emulatedSystems = [ "aarch64-linux" ] stays as a slow fallback. The builder role (servers/roles/builder) is separate from the sovrn roles: a key-only nix-ssh login whose forced command is nix-daemon --stdio, with its own key pair (keys/nix-builder.pub, secret nix-builder-ssh-key). It is a trusted user, since the CI host’s own build outputs carry no signature the builder knows; that grants nothing new, because the CI host already has root on the fleet. The builder fetches inputs from the caches itself (builders-use-substitutes), and limits builds: max-jobs = 1, cores = 1, idle CPU/IO scheduling, and a memory ceiling plus low weights on nix-daemon.service (the daemon runs every build, including those that arrive over SSH; a VM test checks the cgroup), and zram. servers’ checks.x86_64-linux.builder exercises both halves. Measured on mx99 (2 vCPU / 3.8 GB): small builds are unnoticeable, but a from-source Rust compile the size of Stalwart’s needs well over 4 GB. With the limits above it was throttled (MemoryHigh hit millions of times, zram full, about a third of a core of CPU) and never killed, and the host’s services stayed up. Builders of this size are not what mx99 is for: such builds go to the dev machine (qemu, slow, 64 GB) or, later, a dedicated ARM builder.

Binary cache: the CI host’s own /nix/store is the cache, served read-only by harmonia behind Caddy (basic auth, one shared netrc: store paths reveal host layout, even though they hold no secrets). Harmonia signs what it serves, so producers need no key: they push with nix copy --to ssh-ng://root@<ci host> (root over SSH, as deploys do); outputs a remote builder returns to the CI host’s store arrive without any push:

• the dev machine (64 GB RAM) builds x86_64 closures, which ci run there and the runner evaluate to identical store paths, so the runner substitutes instead of building;
• aarch64 builds the runner sends to mx99 come back into the CI host’s store by themselves; builds mx99 does on its own (staging previews) can be pushed too;
• every other fleet host lists the cache in substituters (after cache.nixos.org), its public key in trusted-public-keys and the netrc in netrc-file, with fallback and a short connect-timeout so an outage costs time, not deploys; the ci user needs no extra trust. Implemented in servers: modules/nix-cache.nix, roles/ci/cache.nix, just cache-push, ci-cache-pin.

The cache is not a repository of project state (the runner pushes nothing to repos): losing it only costs rebuild time. GC on the CI host (min-free / max-free) must not eat it silently: pin GC roots for the current closure of each host (last N deployed generations), and watch disk (30 GB on infra.rtw.run). Only identical derivations hit; a dirty tree or different input pin builds fresh, which is correct.

Workspaces: per-run worktrees (a gate can hold a run open for hours); HOME and .cache shared per repo (accepted shared state). .devenv/ is per run; activation still hits a warm store. Host substituters include devenv.cachix.org, since the ci user can’t add substituters itself.

Host sizing: infra.rtw.run stays unresized (2 vCPU / 3.8 GB, x86_64, no swap). That is enough because builds happen elsewhere: x86_64 on the dev machine, aarch64 on mx99, both pushed to the cache, so the runner mostly evaluates and substitutes (fleet eval ~1-2 GB). Requirements: zram on the host, ci.slice with MemoryMax about 2.5 GB, one concurrent run, and nix.settings.max-jobs of 1. Revisit (resize, or a dedicated ARM builder) if runs start building locally or two overlapping runs become necessary. infra.mymood.at (4 vCPU / 8 GB) was considered as the runner host and rejected: it carries the live moods app and backup MX, and the runner holds root on the fleet.

11. Appview

Minimal server-rendered HTML (Go templates, plain CSS, ~40 lines of vanilla JS for live logs and confirmations; every page works without JS), reachable from a phone. It is a stateless view over the runner’s API (one runner, so no index of its own); a second runner would bring back an index fed by the runners’ /events.

Auth: the owner signs in with a password (bcrypt hash file, e.g. mkpasswd -m bcrypt -R 12); the session is an HMAC-signed, SameSite=Strict cookie, and every form carries a session-bound CSRF token. Agents use bearer tokens from a <name> <token> file: they can read runs and plain-text logs, trigger workflows and run manual actions that aren’t style: danger, never approve gates or cancel. Approvals are recorded with the owner’s DID as actor. ci trigger <repo> [workflow] and ci wait <repo> [sha] use the agent API (CI_APPVIEW_URL, CI_TOKEN); ci wait exits like the run (75 when it is waiting for you).

• Pipelines: repo, sha, per-workflow status; live logs over SSE (proxied from the runner’s /logs).
• Run page: timeline of actions and transitions, summaries, attachments, manual action buttons (danger rendered big and red).
• Gates: prompt, the summaries of earlier actions (e.g. an nvd diff), one button per option, Dismiss. Approval requires the owner’s session.
• History: past runs, including reported local runs.
• API: JSON status and plain-text logs for agents; ci wait uses it. Agent tokens can read, trigger and re-run, not approve gates or run danger actions.
• ntfy (sent by the runner, which sees every transition): gate waiting, failure, optionally success; view actions link to the appview’s run page (no auth tokens inside notifications).
• Local run reporting: ci run|action|resume --report (automatic when the run executed an audited action and the appview is configured) uploads a snapshot of the run (state, events, results, artifacts; no checkouts) after each segment, so laptop deploys appear in the history. Their manual actions (Roll back) run on the runner, which checks the recorded commits out again; a run of a working copy with changes, or with --repo links, has to be acted on locally (ci action --run).
• Later: ATProto OAuth login, so approvals carry the owner’s DID.

12. Secrets

• The encrypted vault (data repo) is a repos: entry exported as SECRETS_DIR. It is cloned at its pushed revision in both modes, so only committed, pushed secrets run anywhere, and the vault sha is recorded per run.
• Colmena and Justfiles call secrets decrypt <selector> exactly as today. The engine has no secrets concept and does no masking; recipes pipe secrets into tools and never print them.
• The secrets tool gains (first implementation step):
  • $SECRETS_DIR/.recipients: encrypt targets every listed recipient.
  • secrets rekey: re-encrypt every secret to the current recipients.
  • SECRETS_IDENTITY: identity file override (default ~/.config/age/identity.txt).
• The dev machine and the runner use different age identities; secrets are encrypted to both and rekeyed on demand. Revoking the runner = remove its recipient, rekey, rotate what it could read.
• The runner’s identity is itself a vault secret (ci/age-identity), delivered by Colmena deployment.keys (keyCommand = [ "secrets" "decrypt" "ci/age-identity" ]) to a root-only path, and handed to segments via LoadCredential=; the engine sets SECRETS_IDENTITY to $CREDENTIALS_DIRECTORY/age-identity.
• Repo access: the runner’s forge deploy key (it lives on the forge host, so a local path also works).

13. CD pattern for Colmena projects (lives in project repos, not the engine)

servers/Justfile recipes that any Colmena fleet can copy:

Notes

• Health URLs come from code (a flake output such as ci.hosts.<fqdn>.healthChecks, or hosts.json). The existing monitor can later read the same list.
• Rollback needs no evaluation or build, so it works with a broken flake. It does not undo state (database migrations) or uploaded keys; migrations must be backward-compatible, or recovery is roll-forward.
• The CI host itself deploys last, ideally in its own gated step. If a self-deploy breaks the appview, recovery is SSH.
• servers flake inputs move from git+file:///home/… to forge URLs; use --override-input locally for unpushed work.

14. Local vs CI parity

15. Implementation order

Tracked as issues in this repo under eae4f87 (bug ls), one at a time:

1. f25b56a secrets: multiple recipients, rekey, SECRETS_IDENTITY.
2. 5434405 Contracts and project skeleton: lexicons, generated types, Go module.
3. de85a99 Engine core: manifest, action execution, result contract, routing, events, local CLI.
4. b9fb99a Engine: repo checkouts at pinned revisions, env: exports, --clean.
5. dea75b4 Engine: gates, pause/resume, manual actions, concurrency, loop guard.
6. 65e8495 Adopt locally: sovrn, servers, moods workflows and recipes; local gated CD works end to end.
7. de35782 Bridge.
8. 7a3d5cf Runner daemon.
9. 7f8343d Appview.
10. db62e70 Deploy onto the forge host: NixOS modules, host resize/rebuild, CI identity via Colmena.
11. adca87e Cutover and end-to-end validation (phone + /rc flow, self-deploy).
12. 78c84e0 Local run reporting and cross-host locks.

Later: remote aarch64 builder; contributing a devenv engine to upstream’s spindle (it now has a microVM engine and pluggable engines); ATProto OAuth login; service-auth JWTs; monitor reading health checks; nspawn/microvm engines for untrusted code; publishing records to a PDS.