sovrn

Experimental AT protocol email hosting

T1: dev PDS moves to ZDS (devenv runner + smoke)

closed
#f1be097 opened by agent Sep 13

Parent: bug 75966cc (cell architecture tracking).

Goal

Replace the Bluesky reference-image dev PDS with ZDS in devenv up, so dev matches the locked production pick (ZDS) from day one.

Scope

  • New runner (replace/retire nix/scripts/serve-pds-upstream.sh): build ZDS (Zig via devenv or pinned container), env ZDS_DB=$SOVRN_DATA_DIR/zds/zds.sqlite3, ZDS_BLOBSTORE_PATH=$SOVRN_DATA_DIR/zds/blobs, fixed dev secrets, :3000.
  • devenv.nix: processes.pds points at the ZDS runner; health gate on describeServer; extend sovrn-reset to wipe ZDS state.
  • PDSProvisioner stub docs note ZDS as the live target (no prod wiring here).

Acceptance

  • devenv up brings ZDS healthy; smoke passes: create account, resolve handle, OAuth login via authbroker, upload blob, export CAR.
  • CI matrix runs the same smoke; reference-image leftovers removed.
  • Split follow-up if big: 1a runner, 1b smoke/contract suite.

2 Comments

agent f010b6e Sep 14

ZDS Dev PDS Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build ZDS from source with Nix, run it natively via devenv processes up on :3000, and verify with a 5-step smoke (account, handle, OAuth, blob, CAR); the same Nix binary ships to the production cell (T2).

Architecture: New nix/pkgs/zds.nix Zig package (main-branch rev pin, fetchZigDeps FOD for simplest robust offline deps) exposed as flake.packages.zds; native runner nix/scripts/serve-zds.sh execs ${zds}/bin/zds with fixed dev env (ZDS_DB, ZDS_BLOBSTORE_PATH, secrets); devenv.nix:processes.pds points at the runner; reference-image leftovers removed; bash smoke scripts/smoke-zds.sh + just smoke-zds covers acceptance.

Tech Stack: Zig 0.16.0 (pkgs.zig_0_16, zig.hook), ZDS tangled.org/zat.dev/zds main-rev, SQLite + disk blobs, bash/curl/jq smoke (curl+jq already in devenv shell), podman removed from PDS path, Go authbroker for OAuth leg.

User choices locked: main-branch rev pin · native binary (not OCI) · runner+smoke in one plan · simplest build over hermeticity (fetchZigDeps FOD). No CI yet (hosting/CI still being evaluated) — smoke is just smoke-zds, CI wiring is a later follow-up.

Task 1: Nix ZDS package from source

Files: - Create: nix/pkgs/zds.nix - Create: nix/pkgs/zds-deps.nix (FOD wrapper) - Reference: nix/pkgs/stalwart.nix:1-73, nix/lib.nix:1-84, flake.nix:1-43

  • [ ] Step 1: Verify Zig toolchain available

Run: nix eval --raw nixpkgs#zig_0_16.version 2>&1 || devenv shell nixpkgs#zig -- zig version Expected: 0.16.0 (matches ZDS minimum_zig_version = "0.16.0").

  • [ ] Step 2: Resolve main-branch rev to pin

Run: git ls-remote https://tangled.org/zat.dev/zds HEAD Expected: one SHA. Record full 40-char SHA + date for rev/version (e.g. version = "0.3.1+unstable-2026-09-14").

  • [ ] Step 3: Create nix/pkgs/zds.nix
# ZDS PDS, built from source. Main-branch rev pin (spaces-alpha moves fast).
# Same binary ships to prod cells (T2): keep glibc within trixie ceilings.
{ pkgs, src-rev ? "MAIN_SHA_HERE", src-hash ? "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
  depsHash ? "sha256-BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB=",
}:
let
  version = "0.3.1+unstable-2026-09-14";
  src = pkgs.fetchgit {
    url = "https://tangled.org/zat.dev/zds";
    rev = src-rev;
    hash = src-hash;
  };
  zigDeps = pkgs.callPackage ./zds-deps.nix { inherit src version; depsHash = depsHash; };
in
pkgs.stdenv.mkDerivation {
  pname = "zds";
  inherit version src;
  nativeBuildInputs = [ pkgs.zig_0_16.hook pkgs.cacert ];
  buildInputs = [ pkgs.sqlite ];
  dontUseZigCheck = false;
  zigBuildFlags = [ "-Doptimize=ReleaseSafe" "-Dversion=${version}" ];
  postPatch = ''
    ln -sfn ${zigDeps} "$ZIG_GLOBAL_CACHE_DIR/p"
  '';
  meta = {
    description = "ZDS ATProto PDS (Zig)";
    homepage = "https://tangled.org/zat.dev/zds";
    mainProgram = "zds";
  };
}
  • [ ] Step 4: First build, fill hashes

Run: nix --extra-experimental-features 'nix-command flakes' build .#zds --print-out-paths -L 2>&1 | tail -20 Expected: FAIL first time with expected sha256:... for src-hash/depsHash → paste back, re-run until PASS and result/bin/zds --help lists --port --db --host.

  • [ ] Step 5: Commit
jj commit -m "feat(pds): build ZDS from source with nix" nix/pkgs/zds.nix nix/pkgs/zds-deps.nix

Task 2: Flake + Justfile build entrypoints (prod-reusable binary)

Files: - Modify: flake.nix:30-42 - Modify: Justfile:50-71 (mirror build-stalwart)

  • [ ] Step 1: Expose packages.zds
# in flake.nix outputs let-block, add:
zds = pkgs.callPackage ./nix/pkgs/zds.nix { };
# and in packages = { ... } add:
inherit zds;

Update the NOTE: comment at flake.nix:10-16 → ZDS is now Nix-built; keep one line noting the old @atproto/pds workspace:* breakage is historical.

  • [ ] Step 2: Add just build-zds (prod reuse + libc gate)
# Build the ZDS PDS binary on the controller via Nix. Result is an out-link at
# deployment/.local/zds-prod (GC root, gitignored); T2 copies bin/zds to the cell.
build-zds:
    #!/usr/bin/env bash
    set -euo pipefail
    mkdir -p deployment/.local
    nix --extra-experimental-features 'nix-command flakes' build .#zds \
      --out-link deployment/.local/zds-prod --print-out-paths
    test -x deployment/.local/zds-prod/bin/zds
    BIN="deployment/.local/zds-prod/bin/zds"
    MAX_GLIBC=$(strings "$BIN" | grep -o 'GLIBC_[0-9.]*' | sort -Vu | tail -1)
    [ "$(printf '%s\n%s\n' "$MAX_GLIBC" GLIBC_2.41 | sort -V | tail -1)" = "GLIBC_2.41" ] || { echo "zds needs $MAX_GLIBC, trixie ships GLIBC_2.41" >&2; exit 1; }
    echo "zds binary ready: deployment/.local/zds-prod/bin/zds ($MAX_GLIBC OK)"
  • [ ] Step 3: Verify

Run: nix --extra-experimental-features 'nix-command flakes' build .#zds --print-out-paths && ls -la result/bin/zds && just build-zds Expected: PASS, deployment/.local/zds-prod/bin/zds executable, glibc check OK.

  • [ ] Step 4: Commit
jj commit -m "feat(pds): expose zds flake package and build-zds" flake.nix Justfile

Task 3: Dev runner script (native binary)

Files: - Create: nix/scripts/serve-zds.sh - Reference: nix/scripts/serve-pds-upstream.sh:1-58, nix/scripts/serve-stalwart.sh:40-51

  • [ ] Step 1: Write nix/scripts/serve-zds.sh
#!/usr/bin/env bash
# ZDS PDS — NIX-BUILT native runner (dev).
# Replaces serve-pds-upstream.sh (Bluesky spaces-alpha image retired, T1).
# Binary comes from nix/pkgs/zds.nix (same derivation T2 ships to cells).
# ZDS defaults to :2583; dev listens on :3000 to keep sovrn wiring stable.
set -euo pipefail

ZDS_BIN="${ZDS_BIN:?ZDS_BIN not set (devenv.nix passes \${zds}/bin/zds)}"
DATA_DIR="${SOVRN_DATA_DIR:?SOVRN_DATA_DIR not set}/zds"
mkdir -p "$DATA_DIR/blobs"

log() { echo "[serve-zds] $*"; }

# Fixed dev secrets (stable across restarts so smoke/OAuth is reproducible; never ship to prod).
export ZDS_HOST="${ZDS_HOST:-127.0.0.1}"
export ZDS_PORT="${ZDS_PORT:-3000}"
export ZDS_DB="$DATA_DIR/zds.sqlite3"
export ZDS_BLOBSTORE_PATH="$DATA_DIR/blobs"
export ZDS_PUBLIC_URL="${ZDS_PUBLIC_URL:-http://localhost:3000}"
export ZDS_SERVER_DID="${ZDS_SERVER_DID:-did:web:localhost}"
export ZDS_JWT_SECRET="${ZDS_JWT_SECRET:-zds-dev-jwt-secret-fixed-0123456789abcdef}"
export ZDS_DPOP_SECRET="${ZDS_DPOP_SECRET:-zds-dev-dpop-secret-fixed-0123456789abcdef}"
export ZDS_ADMIN_TOKEN="${ZDS_ADMIN_TOKEN:-zds-dev-admin-token-fixed}"
export ZDS_PLC_ROTATION_KEY="${ZDS_PLC_ROTATION_KEY:-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa}"
export ZDS_INVITE_REQUIRED="${ZDS_INVITE_REQUIRED:-false}"
export ZDS_HANDLE_DOMAINS="${ZDS_HANDLE_DOMAINS:-.test}"
export ZDS_PERMISSIONED_DATA="${ZDS_PERMISSIONED_DATA:-true}"

log "starting nix-built ZDS on $ZDS_HOST:$ZDS_PORT (db=$ZDS_DB blobs=$ZDS_BLOBSTORE_PATH)"
exec "$ZDS_BIN"

Make executable: chmod +x nix/scripts/serve-zds.sh.

  • [ ] Step 2: Syntax check

Run: bash -n nix/scripts/serve-zds.sh && shellcheck nix/scripts/serve-zds.sh Expected: PASS.

  • [ ] Step 3: Commit
jj commit -m "feat(pds): add nix-built ZDS dev runner" nix/scripts/serve-zds.sh

Open question: confirm ZDS_PERMISSIONED_DATA=true in dev (matches prod; gates com.atproto.space.*). If smoke hits spaces lexicons, keep true.

Task 4: devenv.nix wiring (processes, reset, health, toolchain)

Files: - Modify: devenv.nix:6-11,42-83,94-117,119-143,177-187

  • [ ] Step 1: Add zds derivation + Zig toolchain
# in let-block alongside stalwart (devenv.nix:6-11):
zds = pkgs.callPackage ./nix/pkgs/zds.nix { };
# in packages list add:
pkgs.zig_0_16

(curl+jq already added to devenv shell per user — verify present, do not duplicate.)

  • [ ] Step 2: Replace processes.pds with native runner
# Replace devenv.nix:113-117 block:
# Decision 2026-09-13 (bug 75966cc/9303770): dev runs nix-built ZDS natively.
# Same derivation ships to cells (just build-zds / T2). Listens on :3000.
processes.pds.exec =
  let
    run = pkgs.writeShellScript "sovrn-run-zds" ''
      set -euo pipefail
      export ZDS_BIN="${zds}/bin/zds"
      export SOVRN_DATA_DIR="$SOVRN_DATA_DIR"
      exec ${./nix/scripts/serve-zds.sh}
    '';
  in
  "${run}";

Delete the Decision 2026-08-24 comment about the spaces-alpha image.

  • [ ] Step 3: Extend sovrn-reset to wipe ZDS state
# devenv.nix:77-82, replace:
(pks.writeShellScriptBin "sovrn-reset" ''
  set -euo pipefail
  podman rm -f sovrn-stalwart 2>/dev/null || true
  rm -rf "$SOVRN_DATA_DIR/stalwart" "$SOVRN_DATA_DIR/zds"
  echo "sovrn dev state wiped ($SOVRN_DATA_DIR)"
'')

sovrn-pds container name removed (native process); old pds/ dir intentionally orphaned (one-time stale reference state).

  • [ ] Step 4: Keep health gate on describeServer

Health at devenv.nix:119-143 already polls 3000/xrpc/com.atproto.server.describeServer — keep. Same for enterTest.

  • [ ] Step 5: Verify bring-up

Run: devenv processes up -d && sleep 5 && curl -fsS http://127.0.0.1:3000/xrpc/com.atproto.server.describeServer | jq . && curl -fsS http://127.0.0.1:3000/xrpc/_health; devenv processes down Expected: both 200; describeServer shows inviteCodeRequired:false.

  • [ ] Step 6: Commit
jj commit -m "feat(pds): run nix-built ZDS in devenv processes" devenv.nix

Task 5: Retire reference-image leftovers

Files: - Delete: nix/scripts/serve-pds-upstream.sh - Modify: flake.nix:10-16, docs/06-pds-selection.md:8-16

  • [ ] Step 1: Delete upstream runner + update docs

Run: rm nix/scripts/serve-pds-upstream.sh Edit docs/06-pds-selection.md:8-16 dev-runtime note → Development runs nix-built ZDS natively on :3000 (nix/pkgs/zds.nix + nix/scripts/serve-zds.sh); the Bluesky spaces-alpha image was retired in T1 (bug f1be097).

  • [ ] Step 2: Verify no stragglers

Run: rg -n "serve-pds-upstream|pds-spaces-alpha|ghcr.io/bluesky-social/atproto|SOVRN_PDS_IMAGE" --glob '!devenv.lock' --glob '!.devenv/**' Expected: zero hits except historical ADR mentions.

  • [ ] Step 3: Commit
jj commit -m "chore(pds): retire reference-image runner" nix/scripts/serve-pds-upstream.sh flake.nix docs/06-pds-selection.md

Task 6: Smoke suite (5-step acceptance, no CI yet)

Files: - Create: scripts/smoke-zds.sh - Modify: Justfile:185-189 (add smoke-zds target), internal/pdsprovisioner/provisioner.go:39-40 (doc note) - Reference: upstream tools/smoke.sh, internal/authbroker/authbroker.go, internal/integration/helpers_test.go:27-38

  • [ ] Step 1: Write scripts/smoke-zds.sh (bash, set -euo pipefail, curl+jq from devenv shell; PDS at http://127.0.0.1:3000)
#!/usr/bin/env bash
# ZDS dev smoke: create account, resolve handle, OAuth metadata gate,
# upload blob, export CAR. Mirrors upstream tools/smoke.sh against :3000.
PDS="${ZDS_URL:-http://127.0.0.1:3000}"
HANDLE="smoke-$(date +%s).test"
EMAIL="[email protected]"
PASS="smoke-pass-0123456789"
ACC=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.server.createAccount" \
  -H 'Content-Type: application/json' \
  -d "{\"handle\":\"$HANDLE\",\"email\":\"$EMAIL\",\"password\":\"$PASS\"}")
DID=$(echo "$ACC" | jq -r .did); ACCESS=$(echo "$ACC" | jq -r .accessJwt)
[ -n "$DID" ] && [ "$DID" != null ] || { echo "createAccount failed: $ACC"; exit 1; }
echo "account: $DID $HANDLE"
curl -fsS "$PDS/xrpc/com.atproto.identity.resolveHandle?handle=$HANDLE" | jq -e --arg d "$DID" '.did == $d' >/dev/null
echo "handle resolves"
curl -fsS "$PDS/.well-known/oauth-authorization-server" >/dev/null \
  && echo "oauth metadata OK" || echo "WARN: oauth metadata check skipped (authbroker leg runs in Go)"
# blob: 1x1 png fixture
FIXTURE="${SMOKE_FIXTURE:-scripts/testdata/pixel.png}"
CID=$(curl -fsS -X POST "$PDS/xrpc/com.atproto.repo.uploadBlob" \
  -H "Authorization: Bearer $ACCESS" -H 'Content-Type: image/png' \
  --data-binary @"$FIXTURE" | jq -r .blob.cid)
curl -fsS "$PDS/xrpc/com.atproto.sync.getBlob?did=$DID&cid=$CID" -o /tmp/smoke-blob.out
echo "blob: $CID"
curl -fsS "$PDS/xrpc/com.atproto.sync.getRepo?did=$DID" -o /tmp/smoke-repo.car
test -s /tmp/smoke-repo.car && echo "CAR: $(wc -c < /tmp/smoke-repo.car) bytes"
echo "SMOKE OK: $DID"

Implementer: add scripts/testdata/pixel.png 1×1 fixture, strict OAuth leg via authbroker.StartLogin($HANDLE) helper if cheap, else metadata-gate + WARN as above.

  • [ ] Step 2: Add just smoke-zds
# Run ZDS dev smoke against devenv PDS (:3000). Requires `devenv processes up`.
smoke-zds:
    #!/usr/bin/env bash
    set -euo pipefail
    scripts/smoke-zds.sh
  • [ ] Step 3: Run smoke green

Run: devenv processes up -d && just smoke-zds; ST=$?; devenv processes down; exit $ST Expected: SMOKE OK: did:plc:..., exit 0.

  • [ ] Step 4: PDSProvisioner stub doc note
// in internal/pdsprovisioner/provisioner.go, above Provisioner interface:
// Production PDS wiring targets nix-built ZDS (T1, bug f1be097); Fake/Manual
// cover UX until Issue 4 lands.
  • [ ] Step 5: Commit
jj commit -m "feat(pds): add ZDS dev smoke suite" scripts/smoke-zds.sh Justfile internal/pdsprovisioner/provisioner.go

Task 7: Final verification + handoff

  • [ ] Step 1: Full clean-room pass

Run: sovrn-reset; devenv processes up -d; sleep 5; curl -fsS http://127.0.0.1:3000/xrpc/com.atproto.server.describeServer; just smoke-zds; just test Expected: describeServer 200, SMOKE OK, just test PASS.

  • [ ] Step 2: Record rev + binary for T2

Run: nix --extra-experimental-features 'nix-command flakes' build .#zds --print-out-paths; ./result/bin/zds --help | head -20 Expected: out-path recorded; --help matches serve-zds.sh env. Paste out-path + rev into bug f1be097 comment for T2.

agent f31cbbe Sep 14

Done (commit zwrxyorl). T1 dev-ZDS complete, all acceptance green.

  • Native nix-built ZDS 0.3.1 (tangled.org main rev e17872d3, 2026-09-09) live on :3000 via devenv processes up. Store path for T2: /nix/store/qmacxlflh1kdilq4phzjhlpdwrj221va-zds-0.3.1 (bin/zds, GLIBC_2.36 — within trixie 2.41 ceiling). just build-zds refreshes deployment/.local/zds-prod.
  • Smoke just smoke-zds GREEN: createAccount, resolveHandle, OAuth discovery, blob round-trip (70B), CAR export (714B), latestCommit rev.
  • just test (SOVRN_INTEGRATION=0) all PASS. Reference-image runner deleted; sovrn-reset wipes zds state.
  • Notes for T2: same derivation ships to cells; full down/up cycle needed once to flush old daemon task config (restart alone re-attaches stale config); ZDS getBlob requires a referencing record (orphan blobs 404) — smoke links via app.bsky.feed.post.