[C1] Deployment harness: Nix-built Stalwart OSS + spaces-alpha PDS containers via devenv up
closedGoal
One command brings up a local stack able to host integration tests for every later phase: Stalwart OSS 0.16.x @ rev 2add611e (built from source, true OSS feature set), spaces-alpha reference PDS, both as Nix-built OCI containers following the ../rtw/services pattern, orchestrated by devenv up (no Justfile for lifecycle).
Revisions 2026-08-24: SQLite replaces Postgres everywhere (ADR-0002); service containers are built with Nix (rtw pattern) instead of upstream images where possible.
Layout
sovrn/
├── flake.nix # packages: stalwart, pds-spaces; images: image-stalwart, image-pds
├── nix/
│ ├── lib.nix # mkImage adapted from rtw (nonroot uid1000, /data,/var/log)
│ ├── pkgs/
│ │ ├── stalwart.nix # buildRustPackage @ 2add611e; --no-default-features -enterprise
│ │ ├── pds-spaces.nix # buildNpmPackage @atproto/pds@alpha
│ │ └── pds-spaces-service/{package.json,index.mjs} (+ package-lock.json, one-time)
│ ├── vendor/stalwart-Cargo.lock # pinned from rev (git-dep outputHashes inline)
│ ├── images/{stalwart,pds}.nix
│ └── scripts/{serve-stalwart.sh,serve-pds-spaces.sh}
├── devenv.nix # processes: stalwart, pds, health; sovrn-reset helper
└── data/dev/{stalwart,pds}/ # gitignored state
Implementation notes
Stalwart packaging (nix/pkgs/stalwart.nix)
- fetchFromGitHub @
2add611ee42303c5cef8870b261339a7979e688e(0.16.19), src hash verified. buildNoDefaultFeatures = true; buildFeatures = ["sqlite" "rocks"];— the crate’s DEFAULT features includeenterprise; this is how we get a true OSS build.- nixpkgs’ stalwart-mail is 0.15.5 → disqualified (pre-Registry architecture).
- Cargo.lock vendored at
nix/pkgs/stalwart-Cargo.lock; git deps pinned via cargoLock.outputHashes:- hickory-dns @ e645086f… = sha256-kF/AyYZH7To15a5dmzGOcTwBIm7rThDRH02C3h81dxQ=
- opentelemetry-rust @ 274b4d32… = sha256-6qbfRpD3Q0Q942V/MuxFb8hyseIgdXjEMAwyqtIxlRI=
Bootstrap automation (nix/scripts/serve-stalwart.sh)
Reverse-engineered from crates/common/src/manager/boot.rs + crates/jmap/src/registry/mapping/bootstrap.rs:
1. CONFIG_PATH=/data/config.json missing ⇒ server enters bootstrap mode on :8080.
2. Auth via STALWART_RECOVERY_ADMIN=user:pass env (Basic ⇒ AccessToken::new_admin()); dev secret generated once and persisted to /data/service-secret. This same credential model feeds C3.
3. x:Bootstrap/get (capability urn:stalwart:jmap, accountId “a”) returns singleton id.
4. x:Bootstrap/set update patches: sqlite dataStore (/data/stalwart.db), serverHostname=mail.sovrn.test, defaultDomain=sovrn.test, requestTlsCertificate=false, tracer→/data/logs/.
5. Response updated.<id> carries auto-created admin credentials ([email protected]) → written to /data/admin-credentials.txt (chmod 600).
6. Wait for CONFIG_PATH write, restart binary into normal mode.
PDS packaging (nix/pkgs/pds-spaces.nix) — BLOCKED on lockfile
buildNpmPackage over vendored package.json pinning @atproto/[email protected]. npm exits(1) silently during resolution under node 26/npm12 AND node24/npm11 in this environment, and direct npm execution is outside agent permissions. Manual step required:
cd nix/pkgs/pds-spaces-service && npm install --package-lock-only --ignore-scripts
then set npmDepsHash via first-build error. Until then devenv automatically falls back to pulling ghcr.io/bluesky-social/atproto:pds-spaces-alpha.
devenv bring-up (devenv.nix)
processes.stalwart/processes.pds: ensure-image (podman load, idempotent by ref existence) →podman run --rmforeground, volumes into ./data/dev/*, ENV=development.processes.health: polls /healthz/ready (:8080) and describeServer (:3000).sovrn-resethelper wipes state + containers.- Upstream-image fallback wired automatically when the Nix PDS build isn’t available.
Tasks
- [x] flake.nix + nix/lib.nix + image derivations
- [x] stalwart.nix w/ verified hashes; OSS feature set
- [x] serve-stalwart.sh bootstrap flow (reverse-engineered wire format)
- [x] serve-pds-spaces.sh (adapted rtw dev-secret pattern)
- [x] devenv processes + reset helper + .gitignore
- [ ] Stalwart image builds green (in progress: full rust compile running)
- [ ] PDS lockfile generated (manual) → pds-spaces build green
- [ ]
devenv upfresh-stack acceptance pass (< 5 min, idempotent second boot) - [ ] CI headless job
Acceptance criteria
- Fresh clone →
devenv up→ both services healthy < 5 min, zero manual UI steps; re-run performs no re-bootstrap - Stalwart answers authenticated Registry requests using persisted service credential; SQLite file present in mount
- PDS describeServer + jetstream WS reachable; test account creatable (invite requirement off in dev)
- No enterprise license key anywhere (grep gate)
Risks this surfaces
- Bootstrap API shape vs source reading (serve-stalwart.sh fails loud if divergent)
- @atproto/pds@alpha npm resolution failure (already observed — documented manual path)
- Rootless podman quirks inside devenv shell
2 Comments
Implementation progress — 2026-08-24
All code implemented and Stalwart harness verified live end-to-end. PDS verified live via upstream-image fallback.
Verified against real containers
Stalwart OSS image (
sovrn.local/stalwart:dev, binary 0.16.19 built from rev 2add611e,--no-default-features --features "sqlite,rocks"): - [x] Image builds; loads into rootless podman - [x] First boot: bootstrap mode detected → recovery-admin auth →x:Bootstrap/get→x:Bootstrap/setwith SQLite patch → admin account auto-created ([email protected]) and credentials captured from response → config.json written → automatic restart into normal mode - [x] Normal mode:/healthz/ready=200, authenticatedx:Domain/queryreturns bootstrap-created domain (idb) - [x] Idempotent second boot: skips straight to normal mode, no re-bootstrapPDS (upstream spaces-alpha image fallback path): - [x]
describeServer= 200 (did:web:localhost, user domains.test, invites off)Discoveries that would have bitten later (the point of Phase 0)
methodCalls, notcalls(crates/jmap-proto/src/request/parser.rs). Registry methods wire asx:<ObjectType>/<fn>under capabilityurn:stalwart:jmap;accountId:"a"accepted for registry objects.x:Bootstrap/set— registry objects other than Bootstrap return “server is in bootstrap mode” until restart. serve-stalwart.sh restarts automatically.enterprise(crates/main/Cargo.toml) — OSS build requires explicit--no-default-features. Nix derivation pins this.--userns=keep-id[:uid=1000,gid=1000]for bind-mounted /data writes.runUpstreamPdssecret-generation script.nix/vendor/subdir — Cargo.lock vendored atnix/pkgs/stalwart-Cargo.lockinstead.Remaining for C1 acceptance
nix/pkgs/pds-spaces-service/package-lock.json:cd nix/pkgs/pds-spaces-service && npm install --package-lock-only --ignore-scriptsthen setnpmDepsHashfrom first build error →nix build .#image-pds. Until then devenv uses the upstream image automatically.devenv upinteractive pass (both halves validated independently with identical commands; health-probe logic unchanged)Files: flake.nix, devenv.nix, .gitignore, nix/{lib.nix,pkgs/{stalwart.nix,pds-spaces.nix,pds-spaces-service/*},pkgs/stalwart-Cargo.lock,images/{stalwart,pds}.nix,scripts/{serve-stalwart.sh,serve-pds-spaces.sh}}, docs/adr/0002-sqlite-for-v1.md (+ docs 01/07/09 updates).
Decision: upstream PDS image for dev — npm packaging abandoned — 2026-08-24
Root cause of the npm failure (diagnosed via bisection)
Reproducing with dependency subsets isolated the crash to the alpha @atproto packages’ published manifests: at least one carries a
workspace:*dependency range leaked from the Bluesky monorepo into the npm tarball. npm fails withUnsupported URL Type "workspace:": workspace:*; during full-tree resolution the error surfaces as a silent exit(1) (npm sets exitCode without printing — misleading debug log). Confirmed reproducible on node 26/npm 12 and node 24/npm 11.@atproto/[email protected]*alone reproduces; exact leaking edge not pinned down further because we abandoned this path.Decision
Dev containers do not need bit-for-bit reproducibility, especially for a Thursday-churn alpha. We run Bluesky’s published
ghcr.io/bluesky-social/atproto:pds-spaces-alphaimage directly:nix/scripts/serve-pds-upstream.sh: pull-latest-with-offline-fallback → fresh ephemeral secrets per boot (jwt/admin/plc k256) →podman run --userns=keep-idwith required env (blobstore, PDS_DEV_MODE=true etc.)devenv up; offline starts use cached imagenix/pkgs/pds-spaces.nix,pds-spaces-service/,nix/images/pds.nix, old serve script; flake/devenv conditionals deletedVerification
nix flake check: cleanbash nix/scripts/serve-pds-upstream.shwith SOVRN_DATA_DIR set): container up,describeServer=200 (did:web:localhost,.testdomains, invites off)