Domain saga: crash, race and identity-reuse coverage (replaces 516fcde, df700bf)

open
#4e9d7ba opened by agent Oct 8

Consolidates 516fcde (domain signup saga) and df700bf (E2E + race/conflict tests) from their current status, plus the domain-saga half of the identity-reuse comment on a6c4223.

Status (2026-10-08)

Built and verified live on mx99 (staging, NixOS): domain.create -> Stalwart domain (disabled) -> DB row -> ZDS instance (pdslifecycle, health-gated) -> postmaster identity postmaster.at. (did:plc on plc.directory) -> route record -> DNS readiness -> active, with no operator step (test.kilimanjaro.io). Two fixes found live are in: first login (c2db915) and the PDS health gate waiting for ZDS to listen (6efaf8b). Accounts are provisionable on the mail side (858fad1, closed); ATProto user accounts are a6c4223.

Unit tests already cover (internal/appview/provision_saga_test.go): duplicate after success, cross-tenant duplicate, legacy duplicate, handler duplicate status, concurrent duplicate backstop (same and cross tenant), reused-infra DB failure (never destroys reused infra, 5e674f6), compensations surviving cancel, reuse of an existing postmaster on the same instance (ErrExists -> session). Reconciler: verifier orphan/drift tests.

Missing

  1. Crash between steps: kill (not error) after each saga step (Stalwart domain, DB row, ZDS activation, postmaster createAccount/PLC, route record) and show the next run or the reconciler converges, reusing what exists and destroying nothing live. Today’s tests inject errors in-process; nothing restarts sovrnd mid-saga.
  2. Postmaster identity reuse beyond the same instance (from a6c4223’s comment): a postmaster DID already published at plc.directory whose account isn’t on this instance (domain removed and re-added, instance wiped, cell rebuilt without restore, another cell). Today the saga would mint a second DID for the same handle. Wanted: adopt it when sovrn holds its rotation key (PLC op moving the service endpoint, repo restore if needed), refuse a foreign DID, never mint a duplicate. Needs sovrn’s rotation key in every DID it creates (check the postmaster’s current PLC doc).
  3. Ambiguous createAccount/PLC failures: a timeout after the PLC op may have been published; retry must look the handle up on the instance first.
  4. Conflict semantics: “first verified wins” for the same domain across tenants and cells (per-cell DBs; the accepted v1 gap is cross-cell duplicate creation before first login): decide and test what a second cell does.
  5. Domain removal and re-add: retire order (route record retired before teardown), what happens to the postmaster identity (deactivate, keep rotation authority) so a re-add reuses it.
  6. Integration run: the above as just integration tests against a real Stalwart and ZDS (devenv), plus one live rehearsal on mx99 (remove and re-add test.kilimanjaro.io).

Related

a6c4223 (user ATProto accounts; keeps user-handle reuse and the account intent/reconciler design), 6844d4f (crawl + handle DNS), 62aba1d (relay-only DKIM, DNS records), 9d84518 (registry pagination for complete reconciler enumeration).