Summary

Replace the per-domain ZDS processes on each cell with one vlpds (https://github.com/jazware/vlpds) serving every hosted domain’s handles. vlpds keeps all state in object storage (the cell’s R2 bucket), and upstream now supports multiple handle domains added at runtime (jazware/vlpds#1, merge f247960) and Spaces phase 1 behind --spaces (repo host for its accounts’ space repos + simplespace host).

Today every active domain costs a zds@<slug> systemd unit, a port, a SQLite DB + Litestream replica, a blob dir + rclone sync, an env file and two Caddy map entries, all supervised by internal/pdslifecycle. With vlpds a new domain becomes one admin call (vlpds.admin.addHandleDomain at.<domain>) and the per-domain process machinery goes away.

Scope: a total replacement, not a migration. sovrn is pre-beta with no production accounts, so ZDS and everything around it is removed outright: no dual-backend period, no cutover, no account migration. Test domains and accounts are simply recreated on vlpds.

Why it fits

• Handles: <user>.at.<domain> works as-is. Add at.<domain> as a handle domain. vlpds matches the longest served suffix on a label boundary, and /.well-known/atproto-did and /tls-check answer for every served domain. Label rules (one label, 3-18 chars, reserved list) are the same as ZDS’s (pdslifecycle/handles.go mirrors them).
• API surface sovrnd uses is all standard XRPC that vlpds serves: server.createAccount, server.createInviteCode, server.createSession, server.getSession, server.getServiceAuth, server.describeServer, repo.getRecord / putRecord / deleteRecord, admin.updateSubjectStatus.
• Identity: did:plc on plc.directory (--plc-url, --plc-rotation-key-file, optional --plc-recovery-did-key).
• Abuse gates: --invite-required. Invite codes can be scoped to one domain (createInviteCode {handleDomain}), so a code minted for tenant A can’t create an account under tenant B.
• Ops model (ADR-0009): no coordinator, no Raft. A single node on one bucket is a supported deployment, so one vlpds per cell keeps the cell blast-radius model. Durability = the bucket, so there’s no Litestream/rclone for the PDS.

Known gaps and workarounds

1. Cross-tenant updateHandle. vlpds lets an account move its own handle to any served domain (e.g. from alice.at.a.com to alice.at.b.com). Domain-scoped invites only constrain creation. Workaround: keep the existing Caddy rule that returns 403 for POST /xrpc/com.atproto.identity.updateHandle on the PDS host (handle changes stay sovrnd-exclusive via com.atproto.admin.updateAccountHandle). vlpds’s own account page uses the same XRPC, so its handle-change panel will just show the refusal. (A “home domain” rule was prototyped in a vlpds fork, branch handle-domains on BTBurke/vlpds; could be proposed upstream later.)
2. postmaster is a reserved handle label in vlpds (the reference PDS’s list), so createAccount refuses postmaster.at.<domain>. ZDS allowed it. Decision: rename the per-domain authority account’s handle to a label neither vlpds nor ZDS reserves. Suggested: mailadmin.at.<domain>; domain-admin is also free. (Also reserved, so ruled out: mail, hostmaster, operator, admin.) This changes only the ATProto handle. The postmaster@<domain> mailbox alias (ADR-0009 D32c, RFC 5321) is unaffected. Update ADR-0009 D32 and every place that builds postmaster.at.<domain> (postmaster package, saga, verifier, docs/tests).
3. Duplicate-account error shape differs. postmaster.ErrExists matches ZDS’s 400 InvalidRequest "account already exists". vlpds answers 400 HandleNotAvailable "Handle already taken: <handle>" (and InvalidRequest "Account already exists" only for an existing DID). Update the matcher and its tests.
4. No per-domain origin. All accounts get the one vlpds public URL as #atproto_pds (e.g. https://pds.mxN.<region>.sovrn.at). This amends ADR-0009 D32’s “migration unit = domain + its own ZDS”; moving a domain to another cell becomes per-account atproto migration (D29 says no tenant migration in v1 anyway).
5. Spaces parity is unverified. vlpds Spaces is phase 1 and tracks the reference alpha (PR #5187 @ 5b95b2f2). It must pass sovrn’s spaces contract checks (postmaster as space authority, space credentials, space: scopes, simplespace management, whatever 12ce549 / 6a79018 rely on) before ZDS is removed. Check whether DID docs need #atproto_space_host and whether vlpds emits it.
6. Removing a domain (removeHandleDomain) is refused while active accounts are under it (counted across all shards) unless force. A forced removal leaves those accounts’ handles unverifiable. The reaper should rename or delete accounts first, then remove.
7. Startup window. After a restart vlpds loads the added domains in the background, so for a moment /tls-check and well-known may 404 for added domains. Harmless with Caddy’s cached certs; note it for health gates.

Target architecture (per cell)

• vlpds, single node, --prefix vlpds inside the cell bucket. Per ADR-0009 D26 the prefix is owned by vlpds’s own GC only and is never shared with litestream/ or stalwart-blobs/.
• Flags:
  • --public-url https://pds.<cell> (also the passkey RP, so never change it);
  • --handle-domain = a cell-owned primary (e.g. pds.<cell>; it can’t be removed);
  • --service-did did:web:pds.<cell>, --invite-required;
  • --spaces;
  • SMTP via --email-smtp-url (SMTP2GO, see 2b097db) instead of ZDS’s Resend;
  • secrets as *_FILEs: JWT, admin, internal, KEK, PLC rotation key.
• Caddy: one on-demand-TLS catch-all site, with ask to vlpds /tls-check (or keep sovrnd’s caddy-ask and have it delegate). It reverse-proxies to vlpds with Host preserved. It replaces the pds-origins.map / pds-vanity.map + regex stanza. It keeps: 403 on updateHandle, blocks on /admin, vlpds.admin.* and /internal/* from the public listener, and (optionally) the public sign-up pages.
• DNS: unchanged per domain (*.at.<domain> → cell primary IPs), plus pds.<cell>.
• sovrnd talks to vlpds over loopback with the admin token. Same calls as today, one base URL.

Changes in sovrn

• internal/pdslifecycle: replace the registry/ports/env/systemd/Caddy-map machinery with a thin “handle domain” step:
  • on verified → active: vlpds.admin.addHandleDomain at.<domain> (idempotent: treat DomainExists as success);
  • on retire: removeHandleDomain after accounts are handled.
  • Drop pds_supervision.go / SOVRN_PDS_SUPERVISION, RenderEnv, SystemdRunner, RenderOriginsMap / RenderVanityMap.
• internal/appview/provision.go: PDSProvisioner.EnsureProvisioned(domain) returns the single vlpds base URL. The saga’s ZDS compensation becomes “remove the domain if we added it”. Mint invite codes with handleDomain: "at.<domain>".
• internal/postmaster: base URL from config instead of per-domain 127.0.0.1:<port>; the new authority handle label (gap 2); the ErrExists matcher (gap 3).
• internal/verifier/orphan.go: the retiring-ZDS-instance sweep becomes a stale-handle-domain sweep.
• healthz.go: probe vlpds /xrpc/_health once instead of per instance. caddy_ask.go delegates to vlpds /tls-check or is dropped.
• config.go [pds]: url, admin_token_file; drop origins_map, vanity_map, secrets_dir, caddy_config, per-instance env fields.
• internal/authbroker: hosted-PDS login now points at the vlpds origin. Re-run the OAuth broker tests (PAR / PKCE / DPoP, space: scopes) against vlpds.

Deployment (NixOS / Colmena, epic 3beadb2)

• Package vlpds: either a Nix derivation (Rust pinned by its rust-toolchain.toml (1.99), plus the UI npm run build; C deps need cmake/clang, and Nix hardening must not fortify jemalloc’s -O0 configure probes) or the pinned OCI image ghcr.io/jazware/vlpds@sha256:… run via podman. Pin per environment (06-pds-selection non-negotiable).
• 63eb0d9 (cell data plane on NixOS: ZDS, sovrnd, Caddy, did.json): swap ZDS for vlpds.
• 0e32e5a (Litestream + rclone): drop the ZDS DB replica and the zds-blobs/ sync. vlpds’s durability is the bucket (see vlpds docs/operations/backups-and-recovery.md for bucket-level backups/versioning).
• 23e262e (ZDS S3 blob + CDN spike): moot. vlpds stores blobs in the bucket already.
• Recovery runbooks (provision-cell.md, recovery-primary-ip.md, pds-cell-recovery.md): recovery = start vlpds on the same bucket/prefix with the same secrets. No PDS restore step.

Plan

1. Spike on mx99 (decision gate): run vlpds --spaces there. Exercise the domain saga end to end against it (create the authority account via invite under its new label, route records put/get/delete, getServiceAuth, updateSubjectStatus, brokered OAuth login), plus the Spaces contract checks (gap 5). Record results here.
2. ADR: supersede ADR-0009 D30 (ZDS pick) and amend D24 (cell contents), D26 (bucket layout: vlpds/ prefix replaces litestream/zds… + zds-blobs/) and D32 (migration unit, and the authority handle label from gap 2). Update docs/06-pds-selection.md.
3. sovrnd changes as a straight replacement (no zds/vlpds switch): delete the ZDS code paths in the same change that adds the vlpds ones.
4. NixOS packaging + cell module for vlpds.
5. Replace on mx99: tear down the ZDS units and state, deploy vlpds, re-run the domain signup saga for the test domains and recreate test accounts.
6. Remove ZDS leftovers: nix/pkgs/zds.nix, ZDS secrets, Caddy PDS maps, Litestream/rclone ZDS config, and the zds-blobs/ / ZDS litestream/ prefixes in the cell buckets.

Acceptance

• [ ] A newly verified domain is served with no restart and no new process: sign-up via sovrnd → alice.at.<domain> resolves (DNS-less HTTPS well-known) and logs in through the broker.
• [ ] The per-domain authority account (new label, e.g. mailadmin.at.<domain>) is created and authoring route records on vlpds; postmaster@<domain> mail still routes to the domain admin.
• [ ] Spaces flows sovrn depends on pass against vlpds --spaces.
• [ ] updateHandle is blocked on the PDS host; admin renames work.
• [ ] Cell rebuild per recovery-primary-ip.md brings the PDS back from the bucket alone.
• [ ] No zds@ units, ZDS DBs, Caddy PDS maps or ZDS Litestream/rclone config remain.

References

• vlpds: docs/operations/handle-domains.md, docs/spaces/, DESIGN.md “Spaces” (upstream main, ≥ f247960).
• sovrn: ADR-0009, ADR-0006, docs/06-pds-selection.md, internal/pdslifecycle/doc.go; bugs 75966cc, 3beadb2, 63eb0d9, 0e32e5a, 23e262e, 516fcde, 12ce549, 6a79018, 2b097db.