Parent: bug 75966cc (cell architecture tracking). Supersedes ADR-0008 (FDB active/passive + VIP + leader lease).

Goal

Stalwart runs SQLite everywhere (dev + prod). FoundationDB leaves the stack entirely: no cluster, no client coupling, no FDB roles.

Scope (decompose as 3a/3b/3c)

3a packaging: nix/pkgs/stalwart.nix drops the foundationdb feature, removes the fdb-7_3 pin/postPatch (one-line AGPL fork goes away) and libfdb_c link input; Justfile retires check-fdb-alignment gate and FDB build path; install.yml drops the libfdb_c rpath dance.
3b deploy: site.yml removes the foundationdb role; serve-stalwart.sh and bootstrap-stalwart.sh SQLite-only (Sqlite /data/stalwart.db); group_vars/all/sovrn.yml drops sovrn_fdb_*; foundationdb role deleted.
3c verify + docs: re-run C2 lifecycle / DKIM teardown (cf71374) / pagination (9d84518) suites against SQLite build; ADR-0008 marked superseded by the cell-model ADR (T9).

Acceptance

No FDB .deb, binary feature, role, var, or doc reference remains (grep clean); fresh bootstrap + converge green on SQLite; suites green.

2 Comments

agent a9601b8 Sep 13

T3: Stalwart back to SQLite, remove FoundationDB — Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Stalwart runs SQLite everywhere (dev + prod); FoundationDB leaves the stack entirely.

Architecture: Pure deletion + SQLite-only simplification: strip the foundationdb cargo feature/link inputs, delete the Ansible foundationdb role and all sovrn_fdb_* wiring, make both bootstrap entrypoints emit Sqlite datastore JSON only, then re-verify with grep + Nix build + SQLite integration suites.

Tech Stack: Nix (stalwart.nix, stalwart image), Ansible (site.yml, stalwart/foundationdb/bootstrap/common roles), bash (serve-stalwart.sh, bootstrap-stalwart.sh), Go integration suites (C2 lifecycle, cf71374 DKIM teardown, 9d84518 pagination).

Owner decisions (locked 2026-09-13): - Q1 SQLite path: /var/lib/stalwart/stalwart.db confirmed (matches docs/deployment.md:63). Dev stays /data/stalwart.db. - Q2 common amd64 assert: DELETE entirely (opens arm64 possibility; new T12 tracking issue evaluates amd64+arm64 cells). - Q3 SOVRN_DATASTORE: REMOVE outright (no no-op compat shim). - Q4 stalwart-Cargo.lock: REGENERATE after dropping the feature (don’t leave dead FDB crates vendored).

Scope: 3a packaging / 3b deploy / 3c verify+docs. ADR-0008 already marked SUPERSEDED 2026-09-13 by the cell model (bug 75966cc) — verify only, no edit.

Task 1: 3a — `nix/pkgs/stalwart.nix` SQLite-only build

Files: - Modify: nix/pkgs/stalwart.nix:45-72 - Regen: nix/pkgs/stalwart-Cargo.lock

[ ] Step 1: Remove postPatch + foundationdb feature + libfdb_c input

  # True OSS build: enterprise feature OFF (default features include it).
  # sqlite (data + registry) + rocks (local bootstrap fallback).
  buildNoDefaultFeatures = true;
  buildFeatures = [
    "sqlite"
    "rocks"
  ];

  # The binary lives in the workspace member crates/main.
  buildAndTestSubdir = "crates/main";

  buildInputs = [ ];

  # librocksdb-sys runs bindgen at build time.
  nativeBuildInputs = [
    pkgs.pkg-config
    pkgs.llvmPackages.libclang
  ];

Deletes: the postPatch block substituting fdb-7_4→fdb-7_3, "foundationdb" from buildFeatures, pkgs.foundationdb.lib from buildInputs. Update header comment to drop FDB history.

[ ] Step 2: Regenerate lockfile + build, verify no FDB linkage

Run: nix build .#stalwart --out-link /tmp/stalwart-sqlite --print-out-paths -L Expected: build succeeds (updating cargoLock outputHashes / lockfile if the feature drop changes resolution); then ldd /tmp/stalwart-sqlite/bin/stalwart | grep -i fdb returns nothing.

[ ] Step 3: Commit

jj commit -m "T3a: stalwart.nix SQLite-only, drop fdb feature/pin/libfdb_c" nix/pkgs/stalwart.nix nix/pkgs/stalwart-Cargo.lock

Task 2: 3a — `nix/images/stalwart.nix` drop FDB from dev image

Files: - Modify: nix/images/stalwart.nix:26-27

[ ] Step 1: Remove FDB contents

  contents = [
    stalwart
    serveStalwart
    pkgs.bash
    pkgs.coreutils
    pkgs.curl
    pkgs.jq # bootstrap response parsing
    pkgs.openssl # dev secret generation
  ];

Deletes both pkgs.foundationdb and pkgs.foundationdb.lib lines.

[ ] Step 2: Verify image builds

Run: nix build .#image-stalwart --print-out-paths Expected: PASS.

[ ] Step 3: Commit

jj commit -m "T3a: stalwart dev image drops fdbserver/fdbcli/libfdb_c" nix/images/stalwart.nix

Task 3: 3a — `Justfile` retire `check-fdb-alignment`

Files: - Modify: Justfile:50-85

[ ] Step 1: Delete gate recipe, simplify build-stalwart

Delete check-fdb-alignment recipe + comment; drop the just check-fdb-alignment line; retitle comment to SQLite binary. Keep GLIBC/GLIBCXX ceiling checks.

[ ] Step 2: Verify recipes parse

Run: just --summary Expected: no check-fdb-alignment in list.

[ ] Step 3: Commit

jj commit -m "T3a: retire check-fdb-alignment, SQLite-only build-stalwart" Justfile

Task 4: 3a — `install.yml` drop libfdb_c rpath dance

Files: - Modify: deployment/roles/stalwart/tasks/install.yml:60-75

[ ] Step 1: Keep interpreter fix, drop rpath + ldd assert

- name: Point binary at Debian loader
  ansible.builtin.command: "patchelf --set-interpreter /lib64/ld-linux-x86-64.so.2 {{ sovrn_stalwart_bin }}"
  when: stalwart_interp_check.stdout | default('') | trim != "/lib64/ld-linux-x86-64.so.2"
  notify: Reload systemd and restart stalwart

Replace libfdb_c ldd assert with generic "'not found' in stalwart_ldd.stdout" assert.

[ ] Step 2: Lint

Run: python3 -c "import yaml; yaml.safe_load(open('deployment/roles/stalwart/tasks/install.yml')); print('OK')" Expected: OK.

[ ] Step 3: Commit

jj commit -m "T3a: install.yml drops libfdb_c rpath, keeps loader fix" deployment/roles/stalwart/tasks/install.yml

Task 5: 3b — `serve-stalwart.sh` (dev) SQLite-only, remove SOVRN_DATASTORE

Files: - Modify: nix/scripts/serve-stalwart.sh:22-88 - Modify: devenv.nix:108 (drop -e SOVRN_DATASTORE=... passthrough)

[ ] Step 1: Remove DATASTORE switch + FDB block

CONFIG_PATH="${CONFIG_PATH:-/data/config.json}"
export CONFIG_PATH
...
# --- data store: sqlite only -------------------------------------------
DATASTORE_JSON='{ "@type": "Sqlite", "path": "/data/stalwart.db", "poolMaxConnections": 10 }'

Delete SOVRN_DATASTORE/fdb branches, fdbserver/fdbcli block, FoundationDb JSON. (Q3: remove, no compat shim.)

[ ] Step 2: Shellcheck + grep

Run: shellcheck nix/scripts/serve-stalwart.sh Expected: clean; rg -n "DATASTORE|fdb|FDB|FoundationDb" nix/scripts/serve-stalwart.sh devenv.nix empty.

[ ] Step 3: Commit

jj commit -m "T3b: serve-stalwart.sh SQLite-only, drop SOVRN_DATASTORE" nix/scripts/serve-stalwart.sh devenv.nix

Task 6: 3b — `bootstrap-stalwart.sh` (prod) SQLite-only

Files: - Modify: deployment/roles/stalwart/files/bootstrap-stalwart.sh:7-185 - Modify: deployment/roles/stalwart/tasks/bootstrap.yml:48-95

[ ] Step 1: Switch datastore to Sqlite in script

Required env: DB_PATH replaces CLUSTER_FILE. Payload: dataStore: { "@type": "Sqlite", path: $dbpath, poolMaxConnections: 10 }. Verify block: jq -e --arg db "$DB_PATH" '.["@type"] == "Sqlite" and .path == $db'.

[ ] Step 2: Update bootstrap.yml env (3 occurrences)

Replace each CLUSTER_FILE: "{{ sovrn_fdb_cluster_file }}" with DB_PATH: "{{ sovrn_stalwart_db }}".

[ ] Step 3: Shellcheck

Run: shellcheck deployment/roles/stalwart/files/bootstrap-stalwart.sh Expected: clean.

[ ] Step 4: Commit

jj commit -m "T3b: prod bootstrap SQLite-only (Sqlite /var/lib/stalwart/stalwart.db)" deployment/roles/stalwart/files/bootstrap-stalwart.sh deployment/roles/stalwart/tasks/bootstrap.yml

Task 7: 3b — `sovrn.yml` drop `sovrn_fdb_*`, add SQLite path

Files: - Modify: deployment/inventory/group_vars/all/sovrn.yml:39-58

[ ] Step 1: Replace FDB block with SQLite var

# Stalwart data store (cell model, bug 75966cc): embedded SQLite, replicated
# off-box by Litestream (T4). No shared store.
sovrn_stalwart_db: "/var/lib/stalwart/stalwart.db"
sovrn_backup_target: ""

Deletes sovrn_fdb_mode, sovrn_fdb_cluster_file, sovrn_fdb_version. Reword amd64 packaging comment (FDB rationale gone; arch gate itself deleted in Task 8 per Q2).

[ ] Step 2: Validate YAML

Run: python3 -c "import yaml; d=yaml.safe_load(open('deployment/inventory/group_vars/all/sovrn.yml')); assert 'sovrn_fdb_version' not in d and d['sovrn_stalwart_db'].endswith('stalwart.db'); print('OK')" Expected: OK.

[ ] Step 3: Commit

jj commit -m "T3b: sovrn.yml drops sovrn_fdb_*, adds sovrn_stalwart_db" deployment/inventory/group_vars/all/sovrn.yml

Task 8: 3b — `site.yml` remove role + delete `foundationdb` role + unwire units

Files: - Modify: deployment/playbooks/site.yml:47-50 (delete FDB block) - Delete: deployment/roles/foundationdb/ (entire dir) - Modify: deployment/roles/bootstrap/tasks/main.yml:14-16 (delete Stop foundationdb task) - Modify: deployment/roles/common/tasks/main.yml:1-6 (DELETE amd64 assert entirely per Q2) - Modify: deployment/roles/stalwart/templates/stalwart.service.j2:3-4 → After=network.target only - Modify: deployment/roles/sovrnd/templates/sovrnd.service.j2:3-4 → stalwart-only After/Wants

[ ] Step 1: Apply deletions/edits
[ ] Step 2: Verify

[ ] Step 3: Commit

jj commit -m "T3b: remove foundationdb role, unwire units, drop amd64 gate" deployment/playbooks/site.yml deployment/roles/bootstrap/tasks/main.yml deployment/roles/common/tasks/main.yml deployment/roles/stalwart/templates/stalwart.service.j2 deployment/roles/sovrnd/templates/sovrnd.service.j2

Task 9: 3c — Docs + comment sweep

Files: - Modify: docs/runbooks/provision-cell.md (remove TODO(T3) CLUSTER_FILE line) - Modify: deployment/scripts/persist-bootstrap-secrets:52 (FoundationDB/restic restore → R2 restore) - Modify: deployment/inventory/host_vars/primary.sovrn.example/vault.yml:15 (same reword) - Verify only: docs/deployment.md, docs/09-open-questions.md, docs/adr/0009*, docs/adr/0008* (historical — keep)

[ ] Step 1: Apply three comment edits; final grep gate

Run: rg -n "FDB|fdb|foundationdb|FoundationDB|libfdb|CLUSTER_FILE|clusterFile|fdbserver|fdbcli|fdb\.cluster" --glob '!nix/vendor/*' --glob '!.devenv/*' --glob '!result/*' Expected: only historical (docs/deployment.md former-coupling note, 0008/0009 ADRs) + regenerated stalwart-Cargo.lock internals remain.

[ ] Step 2: Commit

jj commit -m "T3c: docs sweep, CLUSTER_FILE TODO gone" docs/runbooks/provision-cell.md deployment/scripts/persist-bootstrap-secrets deployment/inventory/host_vars/primary.sovrn.example/vault.yml

Task 10: 3c — Verify: rebuild + suites on SQLite

[ ] Step 1: Controller rebuild + grep gate

Run: just build-stalwart Expected: stalwart binary ready ... OK; ldd ... | grep -ci fdb = 0.

[ ] Step 2: Fresh bootstrap + converge

Run: just bootstrap <cell> then just update <cell> Expected: BOOTSTRAP-APPLIED + LISTENERS-OK + cert; second run green/no-changes; config "@type": "Sqlite".

[ ] Step 3: Suites against SQLite build

Run: SOVRN_INTEGRATION=1 go test ./... Expected: green — C2 lifecycle, DKIM teardown (cf71374: dkim.DeleteForDomain → domain.Delete, no orphans), pagination (9d84518: multi-page enumeration, calculateTotal).

Acceptance (from bug): No FDB .deb, binary feature, role, var, or doc reference remains (grep clean); fresh bootstrap + converge green on SQLite; suites green.

agent ac6e1e8 Sep 13

T3 implementation complete — SQLite everywhere, FoundationDB removed

Commits (jj): xnloztru (3a packaging) + ovzuurvy (3b deploy) + e1d590dd (3c docs). No Go code touched by any of them.

What changed

3a packaging: nix/pkgs/stalwart.nix builds sqlite+rocks only (fdb-7_3 postPatch deleted, pkgs.foundationdb.lib link input gone); lockfile needed no regen (feature drop doesn’t alter resolution; consistency validation passed). nix/images/stalwart.nix drops fdbserver/fdbcli/libfdb_c. Justfile retires check-fdb-alignment. install.yml keeps the Debian loader fix, drops --set-rpath /usr/lib + libfdb_c assert (generic not found assert instead).
3b deploy: serve-stalwart.sh + bootstrap-stalwart.sh SQLite-only (Sqlite /data/stalwart.db dev, Sqlite /var/lib/stalwart/stalwart.db prod via new sovrn_stalwart_db var); SOVRN_DATASTORE removed outright (incl. devenv.nix passthrough). site.yml FDB block gone; deployment/roles/foundationdb/ deleted; bootstrap stop task gone; common amd64 assert deleted entirely (arm64 now possible — see T12 ea055d0); both .service units unwired from foundationdb.service; sovrn_fdb_* vars deleted.
3c docs: provision-cell.md T3/CLUSTER_FILE TODO gone; persist-bootstrap-secrets + example vault reworded to R2 restore. ADR-0008 already SUPERSEDED, no edit needed.

Verification

just build-stalwart: stalwart binary ready (... GLIBC_2.39 GLIBCXX_3.4.30 OK); ldd | grep -ci fdb = 0.
Grep gate: zero FDB/CLUSTER_FILE refs outside historical docs (deployment.md, ADR-0008/0009, 09-open-questions.md) + vendored lockfile internals.
ansible-playbook site.yml --syntax-check: clean. just test (SOVRN_INTEGRATION=0): all green.
SOVRN_INTEGRATION=1 vs SQLite dev container: internal/stalwart ⁵²⁄₅₂ pass, internal/dkim + internal/domain green (DKIM teardown cf71374 ordering + pagination 9d84518 suites).

Caveats (not T3 regressions)

internal/integration: 3 failures (AppPasswordXRPC, AppViewXRPCEndToEnd, AppViewUI) from earlier mailcreateAccount DID/handle API drift + appview UI drift — pre-existing, untouched files, needs a follow-up.
Fresh just bootstrap + just update on a real cell is operator-run (no Hetzner host in this environment).

Follow-ups: T12 ea055d0 (amd64+arm64 cell support). Bug stays open for owner review.

T3: Stalwart back to SQLite, remove FoundationDB