T2: Per-domain ZDS behind Caddy — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Each hosted domain gets its own unpatched ZDS process on the cell, reachable at <slug>.pdsN.<region>.sovrn.at + vanity *.at.<domain>, behind Caddy, with sovrnd owning the full unit lifecycle. No ZDS patches, no restarts for domain onboarding, no customer state in Ansible.

Architecture: Ansible ships the binary, a user-level [email protected] template, Caddy with a sovrnd-rendered upstream map, and the cell’s single ZDS secret set. sovrnd reconciles all units on startup (daemon-reload, start units in the hosted-domain inventory, stop any not in inventory) and owns lifecycle thereafter — sovrnd + ZDS units are a package. One sqlite + blob dir per domain; migration unit == data directory (ADR-0009 D32).

Tech Stack: ZDS (nix-pinned in nix/pkgs/zds.nix, unpatched), Caddy 2.11.x (on-demand TLS + map + loopback admin API), systemd user units (--user, lingering), Go (sovrnd lifecycle module), ansible-vault, wildcard DNS. Logs forward to VictoriaLogs (T7); journald is extremis-only fallback.

Locked decisions (from design discussion)

1. Topology: one ZDS per hosted domain (<20/cell). No ZDS source patches — upstream would not accept our gating changes, and carrying a forever-patch against ZDS’s most-churned files (server.zig, identity.zig) costs more over time than the supervisor code, which is ours and doesn’t rot on upstream refactors.
2. Origins: cell-qualified <slug>.pdsN.<region>.sovrn.at. One wildcard record per cell (*.pdsN.<region>.sovrn.at → cell) covers all future slugs — no per-domain DNS provisioning or propagation wait. Accepted cost: cross-cell moves rotate the PDS server DID via the standard PLC/serviceEndpoint flow. (Cell-independent origins rejected: per-domain DNS provisioning breaks self-serve onboarding.)
3. Vanity handles: <user>.at.<domain> via customer wildcard *.at.<domain> CNAME <slug>.pdsN.<region>.sovrn.at. One record per hosted domain, zero touch per user. ZDS resolves via /.well-known/atproto-did (lookup is authoritative; unknown names 404).
4. Secrets: one set per cell, shared by all its ZDS instances (ZDS_JWT_SECRET, ZDS_DPOP_SECRET, ZDS_ADMIN_TOKEN, ZDS_PLC_ROTATION_KEY), vault-first at bootstrap, living in host_vars/<cell>/vault.yml. Blast radius is already the cell. Email via shared comail creds (ZDS_COMAIL_API_KEY/ZDS_COMAIL_DID, [email protected]).
5. Creation is sovrnd-exclusive. No public signup: Caddy L7-blocks createAccount/reserveSigningKey/updateHandle//signup from public ingress; ZDS binds loopback-only; sovrnd calls over loopback. INVITE_REQUIRED=true + sovrnd-minted single-use invite codes (admin token) as defense in depth with audit lineage. Operator account @sovrn.at (did:plc:54ba6vxrdbzi6nexczpk5xzb) is externally hosted (selfhosted.social) — ZDS_OPERATOR_HANDLE=sovrn.at is display-only.
6. Routing: Caddy map-file + graceful admin reload (not sovrnd-proxy — keeps PDS bytes on the direct Caddy→ZDS path). sovrnd renders the map (origin + vanity hosts → 127.0.0.1:<port>) and triggers reload synchronously in the activation flow.
7. Units: systemd user units, zero privilege. No sudoers. sovrn user gets a real home (/var/lib/sovrn), lingering enabled and asserted; units in ~/.config/systemd/user/. Accepted: no UID isolation between domains (cell = blast radius), soft boot ordering vs system Caddy (health gates cover), lingering is load-bearing (asserted in prechecks + monitored in T7).
8. Logging: forward to VictoriaLogs (T7). journald is transport + extremis fallback only. Required: N cells × M instances cannot be operated via journalctl.
9. DB files: reverse domain notation in a single directory. Each domain’s sqlite lives at /var/lib/zds/dbs/<reversed>.db (hosted domain mydomain.com → com.mydomain.db), all in one directory so Litestream watches a single path and each file is self-identifying per domain. Mapping is deterministic from the domain (lowercase ASCII/punycode, strip trailing dot, reverse labels, + ".db"); the exact filename is also stored in the sovrn.db registry next to slug+port. Blob dirs stay per-slug (/var/lib/zds/blobs/<slug>/).

Secrets classification

• Fleet-shared (group_vars/all/vault-shared.yml + existing OIDC/OAuth keys): ZDS_COMAIL_API_KEY, ZDS_COMAIL_DID; non-secret ZDS_EMAIL_FROM, crawlers, proxy defaults.
• Per-cell (host_vars/<cell>/vault.yml, provisioned by recipe before bootstrap, committed ciphertext): the four ZDS secrets above. Never generated at runtime, never fetched back.
• Per-domain: nothing secret beyond the cell set. Slug+port+origin+db-filename registry lives in sovrn.db (Litestream-replicated — restores with the cell, no fetch-back anywhere).

Task 0: Spike — measure and verify assumptions

• [ ] Step 1: Measure ZDS baseline RSS (idle + under just smoke). 20 instances must fit a cell with headroom for Stalwart+sovrnd.
• [ ] Step 2: Verify Caddy map + admin /load graceful reload — rewrite map, reload, confirm in-flight drain + immediate new-host routing.
• [ ] Step 3: Verify per-instance behavior — two instances, distinct ports/suffixes/DIDs; vanity atprotoDid per instance; describeServer advertises only its suffix; confirm loopback-only + Caddy blocks make creation sovrnd-exclusive with zero ZDS changes.
• [ ] Step 4: Record results on this bug. Go/no-go for Tasks 1–5.

Run: ps -o rss / dev Caddy + curl -fsS probes. Expected: all green; RSS budget documented.

Task 1: Cell-level ZDS secrets, vault-first

Files: - Create: deployment/scripts/provision-zds-secrets - Modify: Justfile (provision-zds-secrets CELL, _precheck requires host vault for cells) - Modify: deployment/inventory/host_vars/<cell>/vault.yml (ciphertext only) - Create: deployment/roles/zds/tasks/secret.yml (assert + materialize 0600 files, no_log)

• [ ] Step 1: Write failing-shape test — recipe --dry-run renders REDACTED, asserts structural validity, changes nothing.
• [ ] Step 2: Implement recipe — fill-missing-only generation; --force / --rotate=<key> for explicit rotation.
• [ ] Step 3: Role secret tasks — assert defined/non-placeholder, write files. No generation/slurp/fetch.
• [ ] Step 4: Document rotation — JWT/DPOP/ADMIN: restart instances, sessions re-login. PLC key: new accounts only; cross-cell moves update rotation keys via standard PLC flow.
• [ ] Step 5: Commit — jj commit recipe + role + docs.

Task 2: [email protected] user template + cell provisioning

Files: - Create: deployment/roles/zds/templates/[email protected] (EnvironmentFile=%h/.config/zds/%i.env, Restart=always, hardening) - Create: deployment/roles/zds/tasks/main.yml (binary from deployment/.local/zds-prod with existence assert + trixie glibc gate, template unit, dirs, lingering, Caddy skeleton) - Modify: deployment/roles/common/tasks/main.yml (sovrn home → /var/lib/sovrn) - Modify: deployment/playbooks/site.yml (zds tags), docs/deployment.md (tag list)

• [ ] Step 1: Template + dirs — %h/.config/systemd/user/, %h/.config/zds/, /var/lib/zds/dbs/ (one directory holding every <reversed>.db, sovrn-owned), /var/lib/zds/blobs/<slug>/ per domain.
• [ ] Step 2: Lingering — loginctl enable-linger sovrn; assert Linger=yes every converge (fence: fail loudly, never silently run without it).
• [ ] Step 3: Converge — just update <cell> --tags zds green; rerun no-change; missing just build-zds artifact fails fast.
• [ ] Step 4: Runbook snippet — sudo -u sovrn XDG_RUNTIME_DIR=/run/user/$(id -u sovrn) systemctl --user status zds@<slug> + just pds-status wrapper (journald = extremis only).
• [ ] Step 5: Commit.

Task 3: sovrnd lifecycle module (reconcile-on-startup supervisor)

Files (sovrnd side, exact paths at implementation): - Create: lifecycle module — sovrn.db registry (slug, port, origin, db filename, status), env renderer (ZDS_DB=/var/lib/zds/dbs/<reversed>.db derived deterministically from the domain and stored in the registry), systemctl --user driver (daemon-reload, start/stop/enable/disable), rolling-restart orchestration, per-instance health for T6 - Create: Caddy map renderer + admin-API reload trigger (synchronous in activation) - Tests: reverse-notation mapping (case, trailing dot, punycode/IDNA, multi-label e.g. example.co.uk → uk.co.example.db, Città di… — sample table in test), allocator (uniqueness, DNS-safe charset, never-reuse-after-retire, port range), renderer golden files, reload-failure rollback

• [ ] Step 1: Failing tests — domain→filename mapping table, allocator, renderer goldens.
• [ ] Step 2: Startup reconciliation — on entry into main: daemon-reload, start every in-inventory unit, stop any unit not in inventory. This coupling is intentional: sovrnd + units are a package.
• [ ] Step 3: Activation flow — allocate → render env → enable+start → render map → Caddy reload → health-gate describeServer before marking active.
• [ ] Step 4: Retire flow — stop+disable, tombstone slug (never reuse), map update + reload (data retention per deferred D32 decision).
• [ ] Step 5: Upgrade flow — staggered restart with per-instance health gate.
• [ ] Step 6: Handle-derivation rules — single label, 3–18 chars, reserved list (mail, admin, …; postmaster OK); non-conforming email names get documented alias rule. Reconciler pre-validates everything (ZDS keeps only syntax backstops).
• [ ] Step 7: Commit per step.

Task 4: Caddy — catch-all, on-demand TLS, map routing, creation blocks

Files: - Modify: deployment/roles/caddy/templates/Caddyfile.j2 - Modify: deployment/roles/caddy/tasks/main.yml (admin API loopback-only, map path)

• [ ] Step 1: Catch-all with on_demand_tls { ask … } — ask allow-listed by active .at.<domain> suffixes (per-suffix, not per-user); upstream from map file.
• [ ] Step 2: L7 blocks (public) — ⁴⁰³⁄₄₀₄ for POST /xrpc/com.atproto.server.createAccount, POST /xrpc/com.atproto.server.reserveSigningKey, POST /xrpc/com.atproto.server.updateHandle, GET /signup. All other authed user ops (sessions, OAuth, records, blobs, sync) stay public.
• [ ] Step 3: Verify — new domain active → origin + random vanity handle serve valid certs with no Ansible run and no existing-instance restart (only the new unit starts).
• [ ] Step 4: Commit.

Task 5: Smoke, migration, recovery drills

• [ ] Step 1: Per-domain smoke — via sovrnd path: createAccount → vanity /.well-known/atproto-did → OAuth discovery → blob round-trip → CAR (ZDS_URL=https://<slug>.pdsN… bash scripts/smoke-zds.sh shape).
• [ ] Step 2: Migration drill — copy domain DB (<reversed>.db, self-identifying) + blobs to second host, start unit, flip vanity CNAME, PLC serviceEndpoint/rotation update; assert continuity.
• [ ] Step 3: Recovery drill — rebuild from R2 + repo: sovrn.db restores registry (incl. db filenames), vault supplies cell secrets, reconciler regenerates env+map (incl. lingering + unit state checks), per-domain DBs restore into the single watched directory (Litestream single-path watch — T4 detail). No fetch-back at any step.
• [ ] Step 4: Log-forwarding check — per-instance logs arriving in VictoriaLogs with domain/instance labels (with T7).

Self-review

• Spec coverage: role + Caddy/TLS + site.yml + smoke + version pin (T2 scope intact); lifecycle, routing, secrets, migration, recovery, logging-handoff all tasked.
• No placeholders: exact files/commands/expected outputs per step; TDD + frequent commits.
• Constraint traceability: no ZDS patches (T0) · no per-domain deploys (map+reload, T4) · no customer state in repo (DB registry + vault cell secrets, T1/T3) · static wildcard origins (T4) · shared cell secrets (T1) · unprivileged supervision (T2) · centralized logging noted for T7 · reverse-notation DB files in one Litestream-watched dir (T2/T3/T5).

Task 0 spike results (2026-09-14): GO across the board

All read-only; repo untouched, scratch in /tmp, no processes left behind. Binary reused from deployment/.local/zds-prod (no rebuild).

• RSS (GO): ~20 MB idle, ~22-23 MB after full smoke-zds.sh run; second instance consistent. 20 x 23 MB ~= 460 MB — fits a cell with large margin. Caveat: single-account measurement; per-account growth unmeasured (follow-up spike recommended before freezing cell capacity).
• Caddy map + reload (GO, live-verified with Caddy 2.11.4): map {host} {upstream} + caddy reload routes new hosts immediately, in-flight slow request drained across reload (PID unchanged), unmapped hosts 502 fail-closed. Gotcha recorded: caddy reload --address takes host:port, not a URL. Note: Caddyfile.j2 has no map/ZDS vhost yet — that lands in Task 4.
• Per-instance ZDS (GO, live-verified): two instances, distinct ports/suffixes/DIDs; each describeServer advertises only its suffix; vanity /.well-known/atproto-did resolves per-instance, 404 cross-instance; wrong-suffix createAccount -> 400 UnsupportedDomain; loopback-only bind confirmed (ss -ltn). Source cross-check: env-at-startup, zero reload hooks — one process per domain set is required, matching the plan.
• Go/no-go: Tasks 1 (units/env), 2 (map routing) GO. Task 3 (supervision) conditional GO (unit behavior not exercised). Task 4 (on-demand TLS) conditional (needs public DNS/ACME). Task 5 (backup/restore) conditional GO (layout verified, restore untested).

Proceeding to Task 1.

Task 1 complete (spec ✅ file-level, quality ✅ on re-review)

• deployment/scripts/provision-zds-secrets (vault-first per-cell recipe: 4 secrets, secp256k1 range-checked rotation key, fill-missing-only, REDACTED dry-run), Justfile recipe, deployment/roles/zds/tasks/secret.yml (asserts incl. hex/dpop/range guards mirror the generator, 0600 materialization).
• Review loop: spec reviewer found role asserts weaker than generator + wiring ownership question; asserts hardened (real-Ansible Jinja probed); quality reviewer found 5 should-fix (unknown-key abort, safe_load leak wrap, encrypt-failure shred, wording, nargs comment) — all fixed and re-review APPROVED.
• Ruling recorded: secret.yml gets wired into site.yml in Task 2 (not Task 1, which was forbidden from changing bootstrap semantics).
• Two accepted residual nits in the recipe (pathological-input only, no leak): non-string YAML keys in unknown-key join; OSError traceback on unreadable plaintext vault.
• Changes uncommitted in working copy (alongside pre-existing unrelated mods); commit placement deferred to end of T2.

Task 2 complete (spec ✅, quality ✅ on re-review)

• [email protected] user template (default deps, %i slug, %h paths, Restart=always, moderate hardening with rationale), roles/zds/tasks/main.yml (binary assert → glibc gate → copy → dirs incl. single /var/lib/zds/dbs/ → template → linger+assert → user-manager start+reload), site.yml wiring (zds-secret after stalwart-secret, zds after stalwart-install — also resolves the Task-1 wiring ruling), sovrn home → /var/lib/sovrn + sovrn_home var, docs tag hunk.
• Review loop: quality found reload bus-env race, hardcoded binary path, changed_when contract — all fixed and re-review APPROVED. Notable catch: systemd WorkingDirectory does not strip quotes (quoting attempt correctly reverted); sovrn_zds_bin var added and consumed in copy dest + ExecStart.
• Residual risk recorded: live convergence untested (no cell exists) — first real just update <cell> --tags zds must confirm linger + manager start on a fresh box.
• Changes uncommitted (commit placement deferred to end of T2).

Task 3 complete (spec ✅, quality ✅ on re-review after fixes)

• New internal/pdslifecycle (registry with tombstones, reverse-notation mapping with IDNA, env renderer with goldens, Runner interface + systemd/fake, supervisor with reclaim semantics, Caddy map renderer + contract doc, handle rules mirroring ZDS) + opt-in wiring (pds.supervision, default off, never hard-fails startup).
• Review loop: quality found a real blocker (orphaned retiring rows bricked retries) — fixed via reclaim-same-slug+port + fail→retry tests; plus Supervisor/FakeRunner mutexes (-race clean), failure-path test coverage, SetStatus-by-id, errors.Is. Re-review APPROVED (checked reclaim/port-uniqueness walk, lock hierarchy deadlock-free).
• Carried to Task 4: Caddy snippet in testdata is an incomplete stanza (no final upstream selection) and the regex-extraction chain was never live-probed — Task 4 must complete + verify against real Caddy.
• Changes uncommitted (commit placement deferred to end of T2).

Task 4 live-probe transcript (G2 evidence, real Caddy 2.11.4 via nix + real sovrnd + stub backends)

Rendered production-form Caddyfile from the new Caddyfile.j2 with sample vars → caddy validate: Valid configuration. Test variant differed only in high ports + cert_issuer internal + auto_https disable_redirects (routing/L7/map/ask logic identical).

Ask matrix, real sovrnd (GET /internal/caddy-ask?domain=):

mydomain-com.pds1.eu.sovrn.at            -> 200
mydomain-com.pds1.eu.sovrn.at.           -> 403   # trailing dot denies (G4)
random123.at.mydomain.com                -> 200
alice.at.mydomain.com.                   -> 403   # trailing dot denies (G4)
alice.at.evil.com                        -> 403
at.mydomain.com                          -> 403
mydomain.com                             -> 403
gone-example.pds1.eu.sovrn.at            -> 403
alice.at.gone.example                    -> 403
stale-example.pds1.eu.sovrn.at           -> 200   # ask-ok but unmapped -> 502 at map layer
(missing param)                          -> 400

Routing over https://…:18443 (internal CA, curl -k --resolve):

R1 exact origin A:      pds-A:/xrpc/com.atproto.server.describeServer
R1b exact origin B:     pds-B:/xrpc/test
R2 unknown-user vanity: pds-A:/xrpc/com.atproto.server.describeServer   # zero per-user entries
R3 vanity B:            pds-B:/xrpc/test
R4 stale (ask-ok, unmapped): body "unknown PDS host", HTTP 502
R5 unknown suffix:      HTTP 000 (TLS handshake refused, ask denied issuance)

L7 via PDS host (all 403): POST createAccount, POST reserveSigningKey, POST updateHandle, GET /signup, GET /signup/ (G5). Method scoping intact: POST /signup and GET on createAccount-path proxy through (not blocked).

App/mail preserved (real sovrnd behind app vhost — ⁴⁰¹⁄₃₀₃ = proxied, never 403): three POST creation paths → 401; GET /signup + GET /signup/ → 303; app /internal/caddy-ask → 404; did.json OK; mail / 404 + challenge path reaches proxy.

Reload: map append + backend C + 4s in-flight /slow → caddy reload --address 127.0.0.1:2019 clean; in-flight drained (pds-A:/slow); new origin + new unknown-user vanity route immediately. Adapted servers ['srv0','srv1','srv2']; SNI sovrn.at on PDS port → handshake fail (never reaches PDS).

SOVRN_INTEGRATION=0 go test ./... green; snippet/goldens consistent. All probe processes killed, ports clear, scratch deleted.

Still real-cell-only: LE ACME issuance, Http01/Stalwart coexistence, wildcard DNS, first just update <cell> --tags caddy converge.

Task 4 complete (spec + quality ✅ after fix round)

• Caddyfile.j2: global on-demand TLS (ask from sovrn vars), preserved app/mail blocks + deliberate /internal/* 404 guard (verified: no legitimate app route under /internal/), least-specific :443 PDS fallback (most-specific-wins proven), origins map + regex vanity collapse (no per-user entries), fail-closed 502, 4 L7 blocks PDS-scoped, staging caddy_acme_ca var, LE-storage backup note.
• Privilege separation fixed in review: maps in /etc/caddy/maps.d (root:sovrn 0775), Caddyfile dir root:root 0755 — sovrn moves map data + loopback reload only. Ask trailing-dot deny, /signup* coverage, perm consistency all fixed.
• New: GET /internal/caddy-ask?domain= (decision fn + table tests in pdslifecycle, thin router wiring, deny-by-default incl. supervision-off).
• Evidence: full live-probe transcript posted (2ef08fa) — exact origins, unknown-user vanity, unknown-suffix handshake refusal, reload immediacy + drain, ask matrix, L7 ×4 on PDS vs app-unaffected, SNI isolation.
• Still real-cell-only: LE issuance, Http01/Stalwart coexistence, wildcard DNS, first just update <cell> --tags caddy converge.
• Changes uncommitted (commit placement deferred to end of T2).

Task 5 complete (spec + quality ✅ after fix round)

• Smoke: scripts/smoke-zds.sh gains ZDS_HANDLE_SUFFIX + opt-in ZDS_CHECK_VANITY (default path byte-identical, proven live); vanity documented in both modes (loopback Host-spoof + Caddy-path https).
• Drills EXECUTED fresh with transcripts: per-domain smoke on two live instances (reverse-notation DBs in one dir, cross-isolation 404s, wrong-suffix 400s) + Caddy-path L7 (4×403 via origin, 200 via loopback, unknown-user vanity 200 with zero per-user entries, unknown host 502); migration continuity (DB/ blob/CAR sha256 identical pre/post, same DID); recovery regen through real lifecycle paths (registry→env→map, perms 0600/0644).
• Runbooks: docs/runbooks/pds-domain-migration.md (steps + verification + rollback + TLD onboarding constraint) and docs/runbooks/pds-cell-recovery.md (regen procedure, verified-vs-cell-only boundary, §6 T7 label contract). Drill findings written in-repo (TLD note, on_demand ask-form comment).
• Hygiene: /tmp/t2-spike removed, __pycache__ ignored + untracked, drill ports clear, go test ./... green.
• Still real-cell-only: LE/DNS/fresh-box converge/R2 round-trip/vault materialization (all recorded as explicit residuals across Tasks 0–5).
• Changes uncommitted — commit placement is the remaining decision (working copy mixes T2 work with pre-existing unrelated mods).

T2 implementation complete (all Tasks 0–5 spec+quality approved)

Final cross-task review found one blocker and it is fixed + verified:

• D1 (was blocker): Reconcile now regenerates env. LoadCellSecrets (4 host files, trailing-newline trim, skip-with-log vs hard errors, values never in errors/logs) + deterministic derivation (origin→host→did:web, handles .<host>,.at.<domain>) + Reconcile order env→units→maps→reload with per-unit skip. Recovery invariant documented: registry + cell secrets + binary ⇒ all runtime state recomputed. Verified incl. -race, redaction tests, golden byte-identity.
• D2/D3: just pds-status recipe added (3 runbook refs resolve); provision-cell.md TODO/order fixed to match site.yml.
• Two-suffix handle model assessed safe (disjoint host sets; ask + regex + map layers agree; origin self-DID is standard).
• Full suite SOVRN_INTEGRATION=0 go test ./... green (22 pkgs), vet/gofmt clean, reproducibly.

Tracked follow-up (real, out of T2 scope): comail/crawler provisioning. Sovrnd reads SOVRN_PDS_COMAIL_API_KEY/_DID/_CRAWLERS via new bindings, but nothing sets them — sovrn.toml.j2 has no [pds] section and no sovrn_comail_* vars exist anywhere. Until provisioned, empty values fail per-record env loudly by design (unit skipped, tested). Needs: storage decision (fleet vs per-cell vault), recipe support, group-vars + [pds] template block, first-cell converge proof. Suggest T2-followup bug.

Residuals (all previously recorded, unchanged): live LE/DNS/fresh-box converge/R2 round-trip/vault materialization; RSS-under-load spike before freezing cell capacity.

Working copy: T2 work complete but UNCOMMITTED, interleaved with pre-existing unrelated mods — commit topology left to owner (see chat).