TRACKING: PDS selection — atproto-crates vs tranquil-pds vs ZDS (launch this month, spaces GA Nov)

Parent: bug 3726817. Blocks: production wiring of PDSProvisioner (Issue 3 stub). Context: launch this month (Sept 2026); spaces enters GA in November. Need a clean/easy upgrade path to a spaces-capable PDS with enterprise auth (2FA, passkeys) and S3-compatible blob storage (CDN in front).

Goal: Pick a production PDS we can ship now (pre-GA alpha tolerable behind our stub) and upgrade cleanly to spaces GA in November without re-platforming or DID migration.

Candidates: 1. Reference PDS ghcr.io/bluesky-social/atproto:pds-spaces-alpha (TypeScript, canonical lexicon source, Thursday breaking releases, “do not use in production”, data wiped at alpha end for the hosted sandbox). 2. tranquil-pds tangled.org/tranquil.farm/tranquil-pds (Rust ~87%, single binary no Node runtime, requires Postgres separately, superset of reference: WebAuthn/FIDO2 passkeys + TOTP + backup codes + trusted devices, SSO login/signup, did:web support incl. PDS-hosted subdomains/BYOD, email/discord/telegram/signal notifications, AGPL-3.0-or-later). 3. ZDS tangled.org/zat.dev/zds (Zig, live at pds.zat.dev, spaces-alpha featured, benchmarking-focused, tracks proposal 0016 / com.atproto.simplespace.* baseline, modest bus-factor). 4. Note: atproto-crates PDS (Rust, per docs/06 — full simplespace mgmt + credential TTL + Valkey JTI replay guard) — evaluate whether tranquil is the Rust lane or a separate contender; include briefly so docs/06 stays accurate.

Decision constraints (from requester): - Enterprise: 2FA + passkeys required. - Blobs on S3-compatible store so a CDN can sit in front. - Operational complexity must stay low (single-node v1, alongside Stalwart + Postgres). - Upgrade to spaces GA (Nov) must be non-breaking for did:plc:* identities issued now on public plc.directory (docs/06 §1: hosted path must issue portable public DIDs, never local/dev identities for end-users).

Task 1: Matrix + spike harness (timeboxed)

Files (spike only, may discard): - devenv.nix / compose overlay for 3 side-by-side PDSes - docs/adr/NNNN-pds-selection.md (draft matrix) - internal/pdsprovisioner/{tranquil,zds,reference}_notes.md (spike notes, not production code)

Evaluation criteria (score 0–5, weight in parens): - E1 spaces surface vs proposal 0016 + alpha lexicons (high) — simplespace mgmt, credential issuance, sync, blob-in-space. - E2 upgrade cadence + migration story to GA (high) — Thursday-breakage exposure, DB migrations, CAR export fidelity. - E3 OAuth 2.1 provider quality: PAR/PKCE/DPoP, space: scopes (high). - E4 operational fit: config surface, observability, backup/restore, resource profile (medium-high for Sept launch). - E5 Enterprise auth: 2FA/TOTP, WebAuthn/passkeys, SSO, trusted devices, account-mgmt page (high — new). - E6 Blob storage: S3-compatible backend + CDN-fronting story, upload/get semantics (high — new). - E7 data portability / exit (medium) — repo CAR export, PLC alsoKnownAs rotation. - E8 language alignment + hackability (low-medium) — we are a Go shop; Rust > Zig > TS for ops solidity vs hackability. - E9 license + bus-factor (medium — new): tranquil AGPL-3.0 implications for hosted enterprise use; ZDS niche-language risk; reference canonical-but-throwaway.

[ ] Step 1: Stand up reference + tranquil + ZDS in devenv/compose (reuse nix/scripts/serve-pds-upstream.sh pattern; tranquil needs Postgres; ZDS per its docs).
[ ] Step 2: Run smoke: create account, resolve handle, OAuth login via our authbroker, upload blob, export CAR.
[ ] Step 3: Fill matrix E1–E9 with evidence links (versions pinned per env).
[ ] Step 4: Record provisional pick + GA-upgrade plan (what breaks in Nov, what doesn’t).

Task 2: Production wiring plan (follow-up implementation, not this issue)

internal/pdsprovisioner/live.go: admin-API client (com.atproto.server.createAccount), invite-code abuse gate (docs/08 §2), PLC ops + rotation key in KMS.
Handle provisioning inside our zone + DNS automation (_did TXT / well-known, wildcard MX).
DID-doc service entries #atproto_pds (+ #atproto_space_host for postmaster), postmaster ceremony (docs/06 §5).
Backup/restore drill incl. permissioned repos; RTO/RPO note.
Cutover: flip PDSMode=fake→live, backfill pending mailboxes from Issue 3.

Acceptance (this tracking issue): - ADR with scored E1–E9 matrix + pinned versions + upgrade-to-GA path. - Explicit S3/CDN story and 2FA/passkey story per candidate. - Decision: which PDS ships at Sept launch vs which we target for Nov GA (may differ with migration plan). - pending-mailbox backfill plan defined so Issue 3 UX needs no rework.

Non-goals: No production PDS deploy in this issue; no PLC writes; no spaces-record work (see Stage B tracking issue).

2 Comments

agent 9e39043 Sep 12

Amendment 2026-09-12: one-time pick + whole-stack ops (E10) + Stalwart backend dimension

Revised decision constraints (supersede original):

One-time decision. Whatever ships in Sept is the GA platform. No Nov re-platforming or DID migration.
Assume all candidates reach spec-compliant spaces. E1 becomes a conformance gate (low weight), not a score driver.
Ops complexity across the whole stack is blocking (high weight). Inventory: Stalwart + FDB (prod) / SQLite (dev), sovrnd + SQLite (sovrn.db + oauth.db), PDS + its DB, plus caches (Valkey/Redis/Ripple) and blob stores (disk vs S3).
Minimize number/type of databases. 2-DB floor while Stalwart stays on FDB (FDB + one more). 3-type stack (FDB + SQLite + Postgres) is rejected unless a backend change removes a type.
Open to rearchitecting sovrnd to share the PDS database (e.g. shared Postgres instance, separate DBs/schemas) for a consolidated backup/recovery story. Also evaluating sovrnd-on-FDB (shared cluster with Stalwart) for scale-out.

Re-weighting:

E1 spaces surface: high -> low (gate only).
E2 upgrade cadence: redefined as no-migration stability (Thursday-breakage exposure, DB migration pain, CAR fidelity). high.
E4 operational fit: medium -> high (single-PDS ops).
NEW E10 whole-stack ops complexity: high (blocking):
- E10a DB type count and sharing (can sovrnd share the PDS Postgres instance? can sovrnd share FDB with Stalwart?).
- E10b backup/restore uniformity (runbook count, RTO/RPO, PITR vs snapshot vs fdbrestore vs Litestream).
- E10c extra services (Postgres server, Valkey/Redis, Litestream sidecars).
- E10d failure-domain coupling and failover shape (active/passive + VIP + fencing vs active/active).
- E10e Stalwart backend dimension (see below).
E5 (2FA/passkeys/SSO), E6 (S3+CDN): stay high. E9 (license/bus-factor): stays medium.

NEW dimension: Stalwart backend (scored under E10, evaluation pending):

Stalwart data/blob/search/in-mem stores are independent singletons (RocksDB | FoundationDB | PostgreSQL | MySQL | SQLite; blobs also S3/FS/Azure; search also Elastic/Meilisearch; in-mem also Redis/Valkey). Current: dev SQLite (nix/scripts/serve-stalwart.sh, Sqlite /data/stalwart.db), prod FoundationDB (deployment/roles/stalwart/files/bootstrap-stalwart.sh:140, FoundationDb clusterFile), binary features sqlite+rocks+foundationdb (nix/pkgs/stalwart.nix:60-64), single-instance FDB role (deployment/roles/foundationdb/tasks/main.yml). Postgres/MySQL need a feature-add rebuild (OSS-gated, not enterprise). Coordinator (active/active) is orthogonal to store choice (ADR-0008).

Candidate topologies (not yet scored): - A: Stalwart FDB + sovrnd SQLite + SQLite-PDS (ZDS/reference/atproto-crates) = 2 types. - B: Stalwart PG + sovrnd PG (shared instance) + tranquil PG = 1 server-DB type (+S3). Deletes FDB. - C: Stalwart RocksDB/SQLite + sovrnd SQLite + SQLite-PDS = embedded-only, no server DB. File-replication HA. - D: sovrnd on FDB (shared cluster with Stalwart) + SQLite/PG PDS = still 2 types; makes sovrnd stateless/active-active at highest port cost (hand-built indexes, 5s/10MB txn limits, cgo client coupling, FDB in devenv/CI, cluster-wide restore granularity). Deferred to HA phase unless scale demands.

PDS scale-out note: all PDSes shard at identity layer (new instances + CAR export + PLC alsoKnownAs rotation, E7). Within one PDS: reference needs Redis for multi-process, tranquil needs Valkey beyond single-node Ripple, ZDS single-SQLite with no horizontal story yet. Scored per-candidate under E4, independent of sovrnd DB.

Explicitly not evaluated in this amendment. Scoring, probes, and ADR follow once Stalwart tradeoff questions (RPO/RTO, backup uniformity, blob/search split, PG-vs-FDB-vs-embedded on-call preference, scale trigger, version-coupling tolerance, D timing) are pinned down.

agent 9c35003 Sep 13

Decision recorded (cell architecture, bug 75966cc — locked 2026-09-13)

Pick: ZDS. One-time, no GA migration. Rationale: single-file SQLite (one more Litestream entry, GC-correct per-cell buckets), Zig single binary, benchmarking focus; PDS-exit via CAR/PDSmoover/goat keeps it low-risk and replaceable if it doesn’t work out (fork-carry accepted as fallback).

Blobs: all ZDS blobs public by design (user ATProto activity, not sovrn; bot traffic expected) -> long-term getBlob 302 to public R2/CDN (cache-everything, immutable CIDs). Interim: disk/block + rclone sync per cell. Mail blobs stay in a separate private bucket, JMAP-gated.

Eval spike filed as part of 75966cc (T5): Zig S3 lib eval first (z3 leading — async std.Io, R2-tested, multipart + presigned), prototype only on go-ahead, full upstream PR after. ZeroFS evaluated and dropped (S3-direct needs no POSIX bridge). RocksDB evaluated and dropped repo-wide (no online backup primitive in 0.16.x — see T3).

E1–E10 matrix scoring is closed out by this decision; remaining work is T1 (dev ZDS), T2 (ansible ZDS role), T5 (S3 PR spike).