SQLite + ZDS-blob Backup Implementation Plan (T4 + T11)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Continuous off-box backup for all SQLite files + the ZDS blob dir to the cell’s R2 bucket, with a drilled restore path that just recovery-backup/bootstrap depends on.

Architecture: Litestream sidecars (3 static path entries for Stalwart/sovrn/oauth + 1 dir+watch:true entry for the ZDS per-domain DB dir, S3 backend to R2) for seconds-RPO + PITR; rclone systemd timer (/var/lib/zds/blobs/ → zds-blobs/, --delete so S3 mirrors GC); two T11 recovery roles (read-mostly recovery-backup, rebuild recovery-bootstrap) wired to the existing Justfile stubs. Order invariant everywhere: blobs first, DB snapshot second (orphan bytes self-heal via GC; missing bytes don’t).

Tech Stack: Litestream v0.5.x (static binary, dir+pattern+watch directory watcher, S3-compatible R2 backend), rclone (S3 backend, sync --delete), Ansible roles + systemd service/timer, Cloudflare R2 (one bucket per cell), SQLite PRAGMA integrity_check / wal_checkpoint(TRUNCATE).

Scope: T4 (bug 2559f24: 4a Litestream, 4b rclone, 4c R2 secrets/config, 4d restore drill) + T11 recovery role pair (bug c0e0ddb) in one slice, since recovery is undrillable without replication. Excludes: Stalwart S3 stalwart-blobs/ live store wiring (stays docs-only), T5 S3-PR spike (23e262e), T6 healthz, T10 hetzner role, T8/T7.

Parent context: bug 75966cc (cell architecture tracking); ADR-0009 D26 (bucket layout + ownership invariant); docs/deployment.md, docs/runbooks/provision-cell.md, docs/runbooks/recovery-primary-ip.md.

---

0. Manual operator steps for a new host (stays manual by design)

No provider API calls from Ansible. The operator does this per cell, once, in the Cloudflare dashboard + local vault ceremony:

• [ ] Create R2 bucket named by hostname (e.g. mx1.eu.sovrn.at), jurisdiction matching cell region (EU cell → EU jurisdiction, US → US). No lifecycle rules; versioning off (Litestream owns generations under its prefix).
• [ ] Create R2 API token scoped to that bucket only (Object Read & Write). Record: Account ID, Access Key ID, Secret Access Key, S3 endpoint (https://<accountid>.r2.cloudflarestorage.com).
• [ ] Create/recycle primary IPs mxN.<region>.sovrn.at-{4,6} (protection on, auto-delete off), set rDNS once, add .ssh/config hostname→IP entry. Same as docs/runbooks/provision-cell.md §0.
• [ ] Add host inventory + vault entries (then commit ciphertext):

  
  # deployment/inventory/host_vars/<cell>/vars.yml (plaintext)
  sovrn_r2_account_id: "<accountid>"
  sovrn_r2_endpoint: "https://<accountid>.r2.cloudflarestorage.com"
  sovrn_r2_bucket: "<cell-hostname>"   # default is "{{ inventory_hostname }}"; set explicitly
  sovrn_r2_region: "auto"
  # deployment/inventory/host_vars/<cell>/vault.yml (vault-encrypted)
  sovrn_r2_access_key: "…"
  sovrn_r2_secret_key: "…"
  

  Never paste keys into chat, group vars, or sovrn.yml. Vault it with ansible-vault encrypt / ansible-vault edit via deployment/scripts/vault-password-client (same ceremony as relay/ZDS secrets).
• [ ] DNS first (cell A → primary IPv4, MX → cell hostname, :80/:443 reachable) + just build-stalwart + just build-zds before bootstrap.
• [ ] Verify before traffic: litestream snapshots non-empty for every DB; rclone check clean; healthz green. See Task 6.

Why manual: bucket jurisdiction + token scoping are account-security actions with no safe service identity to automate under yet; a miscoped automation could cross cell prefixes and violate the ADR-0009 ownership invariant (each GC owns exactly its prefix; prefixes never shared across cells or layers).

---

File map (create / modify)

Create:
  deployment/roles/litestream/tasks/main.yml
  deployment/roles/litestream/handlers/main.yml
  deployment/roles/litestream/templates/litestream.yml.j2
  deployment/roles/litestream/templates/litestream-restore.sh.j2
  deployment/roles/rclone/tasks/main.yml
  deployment/roles/rclone/handlers/main.yml
  deployment/roles/rclone/templates/rclone.conf.j2
  deployment/roles/rclone/templates/zds-blobs-sync.sh.j2
  deployment/roles/rclone/templates/zds-blobs-sync.service.j2
  deployment/roles/rclone/templates/zds-blobs-sync.timer.j2
  deployment/roles/recovery-backup/tasks/main.yml
  deployment/roles/recovery-bootstrap/tasks/main.yml
  deployment/playbooks/recover-backup.yml
  deployment/playbooks/recover-bootstrap.yml
  deployment/inventory/host_vars/mx99.eu.sovrn.at/vars.yml  # staging example (bucket/endpoint)
Modify:
  deployment/inventory/group_vars/all/sovrn.yml          # R2/litestream/rclone non-secret defaults
  deployment/playbooks/site.yml                          # wire litestream + rclone after sovrnd/zds (tags: litestream rclone)
  docs/runbooks/provision-cell.md                        # clear TODO(T4/T11) block, link new roles
Test / verify (no new Go unit tests — Ansible + staging drill on mx99.eu.sovrn.at):
  `litestream snapshots`, `PRAGMA integrity_check`, `rclone check`, kill-and-restore drill

DB / path inventory (do not change these paths in this plan):

Layer	Prod path	R2 prefix
Stalwart	/var/lib/stalwart/stalwart.db (sovrn_stalwart_db)	litestream/stalwart.db/
sovrnd registry	/var/lib/sovrn/sovrn.db (sovrn_datadir/sovrn.db)	litestream/sovrn.db/
sovrnd OAuth	/var/lib/sovrn/oauth.db	litestream/oauth.db/
ZDS per-domain	/var/lib/zds/dbs/<reversed>.db (e.g. com.example.alpha.db; internal/pdslifecycle/reverse.go, supervisor.go:29)	litestream-zds/<reversed>.db/ (auto-namespaced by dir watcher, see Task 2)
ZDS blobs	/var/lib/zds/blobs/<slug>/ (base /var/lib/zds/blobs; supervisor.go:30)	zds-blobs/ (rclone mirror, --delete)

---

Task 1: R2/litestream/rclone config surface in sovrn.yml

Files: - Modify: deployment/inventory/group_vars/all/sovrn.yml:43-46 - Test: cd deployment && ansible-playbook playbooks/site.yml --check -l mx99.eu.sovrn.at (dry run resolves vars)

• [ ] Step 1: Add non-secret defaults (secret keys stay in host_vars/<cell>/vault.yml only):

# --- T4 backup (bug 2559f24): non-secret defaults; secrets in host vault ---
sovrn_r2_bucket: "{{ inventory_hostname }}"   # one bucket per cell, named by hostname
sovrn_r2_endpoint: ""                          # e.g. https://<accountid>.r2.cloudflarestorage.com
sovrn_r2_region: "auto"
sovrn_r2_access_key: ""   # overridden from host vault (sovrn_r2_access_key)
sovrn_r2_secret_key: ""   # overridden from host vault (sovrn_r2_secret_key)
sovrn_litestream_version: "0.5.13"
sovrn_litestream_checksum: "sha256:<fill at implementation>"
sovrn_litestream_retention: "30d"
sovrn_litestream_snapshot_interval: "24h"
sovrn_litestream_sync_interval: "1s"
sovrn_rclone_schedule: "*:0/15"   # systemd OnCalendar — every 15 min
sovrn_zds_blob_base: "/var/lib/zds/blobs"
sovrn_zds_db_dir: "/var/lib/zds/dbs"

• [ ] Step 2: Dry-run to verify vars resolve

Run: cd deployment && ansible-playbook playbooks/site.yml --check -l mx99.eu.sovrn.at 2>&1 | head -n 40 Expected: no undefined variable for sovrn_r2_* / sovrn_litestream_*.

• [ ] Step 3: Commit

jj commit -m "feat(backup): R2/litestream/rclone config surface" deployment/inventory/group_vars/all/sovrn.yml

---

Task 2: litestream role — continuous SQLite replication (static + dir-watch)

Files: - Create: deployment/roles/litestream/tasks/main.yml - Create: deployment/roles/litestream/handlers/main.yml - Create: deployment/roles/litestream/templates/litestream.yml.j2 - Create: deployment/roles/litestream/templates/litestream-restore.sh.j2 - Modify: deployment/playbooks/site.yml (add role + litestream tag)

Design (locked): - Static binary install, pinned sovrn_litestream_version (v0.5.x with watch support; fill checksum at implementation), checksum-verified; no apt repo. - 3 static path entries (stalwart/sovrn/oauth) + 1 dir+watch:true entry for ZDS: dir: /var/lib/zds/dbs, pattern: "*.db", watch: true, replica url: s3://<bucket>/litestream-zds. The watcher (fsnotify, fine on Debian) validates SQLite headers and starts replication within seconds of the PDS provisioner creating a new <reversed>.db — no Ansible re-render or new-domain hook needed. Empty dir on startup is allowed with watch: true. recursive: false (flat dir). - Replica paths for dir entries namespace automatically by relative path (com.example.alpha.db → litestream-zds/com.example.alpha.db/ltx/...); document this layout since restore scripts must match it. - restore-if-missing on boot: ExecStartPre=/usr/local/sbin/litestream-restore.sh restores only when the local file is absent (never clobbers live data; the T11 mode=recover path does the authoritative restore).

• [ ] Step 1: Write role tasks

# deployment/roles/litestream/tasks/main.yml
- name: Install litestream binary
  ansible.builtin.get_url:
    url: "https://github.com/benbjohnson/litestream/releases/download/v{{ sovrn_litestream_version }}/litestream-v{{ sovrn_litestream_version }}-linux-amd64.tar.gz"
    dest: /tmp/litestream.tgz
    checksum: "{{ sovrn_litestream_checksum }}"
    mode: "0644"
- name: Unpack litestream
  ansible.builtin.unarchive:
    src: /tmp/litestream.tgz
    dest: /usr/local/sbin/
    remote_src: true
    creates: /usr/local/sbin/litestream
- name: Render litestream.yml
  ansible.builtin.template:
    src: litestream.yml.j2
    dest: /etc/litestream.yml
    owner: root
    group: sovrn
    mode: "0640"
  notify: Reload litestream
- name: Install restore-if-missing helper
  ansible.builtin.template:
    src: litestream-restore.sh.j2
    dest: /usr/local/sbin/litestream-restore.sh
    mode: "0755"
- name: Enable litestream service
  ansible.builtin.systemd:
    name: litestream
    enabled: true
    state: started

# deployment/roles/litestream/handlers/main.yml
- name: Reload litestream
  ansible.builtin.systemd:
    name: litestream
    state: reloaded

• [ ] Step 2: Write litestream.yml.j2

dbs:
  - path: {{ sovrn_stalwart_db }}
    replicas:
      - url: s3://{{ sovrn_r2_bucket }}/litestream/stalwart.db
        endpoint: {{ sovrn_r2_endpoint }}
        region: {{ sovrn_r2_region }}
        access-key-id: {{ sovrn_r2_access_key }}
        secret-access-key: {{ sovrn_r2_secret_key }}
        retention: {{ sovrn_litestream_retention }}
        snapshot-interval: {{ sovrn_litestream_snapshot_interval }}
        sync-interval: {{ sovrn_litestream_sync_interval }}
  - path: {{ sovrn_datadir }}/sovrn.db
    replicas:
      - url: s3://{{ sovrn_r2_bucket }}/litestream/sovrn.db
        endpoint: {{ sovrn_r2_endpoint }}
        region: {{ sovrn_r2_region }}
        access-key-id: {{ sovrn_r2_access_key }}
        secret-access-key: {{ sovrn_r2_secret_key }}
        retention: {{ sovrn_litestream_retention }}
        snapshot-interval: {{ sovrn_litestream_snapshot_interval }}
        sync-interval: {{ sovrn_litestream_sync_interval }}
  - path: {{ sovrn_datadir }}/oauth.db
    replicas:
      - url: s3://{{ sovrn_r2_bucket }}/litestream/oauth.db
        endpoint: {{ sovrn_r2_endpoint }}
        region: {{ sovrn_r2_region }}
        access-key-id: {{ sovrn_r2_access_key }}
        secret-access-key: {{ sovrn_r2_secret_key }}
        retention: {{ sovrn_litestream_retention }}
        snapshot-interval: {{ sovrn_litestream_snapshot_interval }}
        sync-interval: {{ sovrn_litestream_sync_interval }}
  - dir: {{ sovrn_zds_db_dir }}
    pattern: "*.db"
    watch: true
    replica:
      url: s3://{{ sovrn_r2_bucket }}/litestream-zds
      endpoint: {{ sovrn_r2_endpoint }}
      region: {{ sovrn_r2_region }}
      access-key-id: {{ sovrn_r2_access_key }}
      secret-access-key: {{ sovrn_r2_secret_key }}
      retention: {{ sovrn_litestream_retention }}
      snapshot-interval: {{ sovrn_litestream_snapshot_interval }}
      sync-interval: {{ sovrn_litestream_sync_interval }}

• [ ] Step 3: Write litestream-restore.sh.j2 (static restores + dir loop)

#!/usr/bin/env bash
# Restores only absent DBs. Dir entry has no single restore target — loop over replica prefixes.
set -euo pipefail
CFG=/etc/litestream.yml
restore_one() { # $1 = local path, $2 = replica url
  [ -s "$1" ] && return 0
  litestream restore -config "$CFG" -o "$1" "$2"
}
restore_one "{{ sovrn_stalwart_db }}" "s3://{{ sovrn_r2_bucket }}/litestream/stalwart.db"
restore_one "{{ sovrn_datadir }}/sovrn.db" "s3://{{ sovrn_r2_bucket }}/litestream/sovrn.db"
restore_one "{{ sovrn_datadir }}/oauth.db" "s3://{{ sovrn_r2_bucket }}/litestream/oauth.db"
# ZDS dir: list replica prefixes, restore each missing <name>.db
for name in $(rclone lsf "r2:{{ sovrn_r2_bucket }}/litestream-zds/" --config /etc/rclone.conf | sed 's:/$::'); do
  restore_one "{{ sovrn_zds_db_dir }}/$name" "s3://{{ sovrn_r2_bucket }}/litestream-zds/$name"
done

• [ ] Step 4: Wire into site.yml (after sovrnd/zds, tag litestream; zds role must create /var/lib/zds/dbs before litestream starts).
• [ ] Step 5: Converge staging, verify snapshots

Run: just update mx99.eu.sovrn.at --tags litestream Then: ssh mx99.eu.sovrn.at 'sudo litestream snapshots -config /etc/litestream.yml 2>&1 | head -n 20' Expected: one non-empty snapshot line per static DB plus one per <reversed>.db under litestream-zds/.

• [ ] Step 6: Verify auto-pickup — onboard a test domain on staging, assert litestream snapshots gains the new <reversed>.db within ~1 min with no Ansible run.
• [ ] Step 7: Commit

jj commit -m "feat(backup): litestream role with dir-watch for ZDS" deployment/roles/litestream deployment/playbooks/site.yml

---

Task 3: rclone role — ZDS blob dir → zds-blobs/

Files: - Create: deployment/roles/rclone/tasks/main.yml - Create: deployment/roles/rclone/templates/rclone.conf.j2 - Create: deployment/roles/rclone/templates/zds-blobs-sync.sh.j2 - Create: deployment/roles/rclone/templates/zds-blobs-sync.service.j2 - Create: deployment/roles/rclone/templates/zds-blobs-sync.timer.j2

Invariants: --delete (S3 mirrors ZDS sweep GC, never resurrects); timer default every 15 min via sovrn_rclone_schedule; on-demand from recovery-backup. Runbook order always blobs before DB snapshots.

• [ ] Step 1: Write tasks

# deployment/roles/rclone/tasks/main.yml
- name: Install rclone
  ansible.builtin.apt:
    name: rclone
    state: present
    update_cache: true
    cache_valid_time: 3600
- name: Render rclone.conf
  ansible.builtin.template:
    src: rclone.conf.j2
    dest: /etc/rclone.conf
    owner: root
    group: sovrn
    mode: "0640"
- name: Install sync script (blobs-first order documented inside)
  ansible.builtin.template:
    src: zds-blobs-sync.sh.j2
    dest: /usr/local/sbin/zds-blobs-sync.sh
    mode: "0755"
- name: Install sync service+timer
  ansible.builtin.template:
    src: "{{ item.src }}"
    dest: "{{ item.dest }}"
    mode: "0644"
  loop:
    - { src: zds-blobs-sync.service.j2, dest: /etc/systemd/system/zds-blobs-sync.service }
    - { src: zds-blobs-sync.timer.j2, dest: /etc/systemd/system/zds-blobs-sync.timer }
- name: Enable timer
  ansible.builtin.systemd:
    name: zds-blobs-sync.timer
    enabled: true
    state: started
    daemon_reload: true

• [ ] Step 2: Write rclone.conf.j2 + zds-blobs-sync.sh.j2 + units

; rclone.conf.j2 — S3 backend to cell R2 bucket
[r2]
type = s3
provider = Cloudflare
endpoint = {{ sovrn_r2_endpoint }}
region = {{ sovrn_r2_region }}
access_key_id = {{ sovrn_r2_access_key }}
secret_access_key = {{ sovrn_r2_secret_key }}

#!/usr/bin/env bash
# zds-blobs-sync.sh.j2 — blobs FIRST, DB snapshot second (orphans self-heal via GC; missing bytes don't).
set -euo pipefail
rclone sync "{{ sovrn_zds_blob_base }}/" "r2:{{ sovrn_r2_bucket }}/zds-blobs/" \
  --config /etc/rclone.conf --delete --transfers 8 --checkers 16 --fast-list

# zds-blobs-sync.service.j2
[Unit]
Description=ZDS blob sync to R2
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/zds-blobs-sync.sh

# zds-blobs-sync.timer.j2
[Unit]
Description=ZDS blob sync timer
[Timer]
OnCalendar={{ sovrn_rclone_schedule }}
Persistent=true
[Install]
WantedBy=timers.target

• [ ] Step 3: Converge + verify

Run: just update mx99.eu.sovrn.at --tags rclone && ssh mx99.eu.sovrn.at 'sudo /usr/local/sbin/zds-blobs-sync.sh && rclone lsd r2:mx99.eu.sovrn.at/zds-blobs/ --config /etc/rclone.conf' Expected: exit 0, zds-blobs/ lists per-slug dirs.

• [ ] Step 4: Commit

jj commit -m "feat(backup): rclone role for ZDS blobs" deployment/roles/rclone deployment/playbooks/site.yml

---

Task 4: T11 recovery-backup + recovery-bootstrap roles and playbooks

Files: - Create: deployment/roles/recovery-backup/tasks/main.yml - Create: deployment/roles/recovery-bootstrap/tasks/main.yml - Create: deployment/playbooks/recover-backup.yml - Create: deployment/playbooks/recover-bootstrap.yml

These satisfy the existing Justfile stubs (Justfile:152-177 currently fail with “not yet implemented”). Strict host-key checking stays on in Ansible; key rotation lives only in the recovery-bootstrap Justfile recipe (already scaffolded). The manual provider middle step (power off old, create new box re-linking the same mxN.<region>.sovrn.at-{4,6} primaries) stays manual — no provider API from Ansible.

• [ ] Step 1: Write recovery-backup (failure-tolerant, never wipes)

- name: Stop services (best effort)
  ansible.builtin.systemd:
    name: "{{ item }}"
    state: stopped
  loop: [stalwart, sovrnd, zds]
  failed_when: false
- name: WAL checkpoint each static DB (best effort)
  ansible.builtin.shell: sqlite3 "{{ item }}" "PRAGMA wal_checkpoint(TRUNCATE);"
  loop:
    - "{{ sovrn_stalwart_db }}"
    - "{{ sovrn_datadir }}/sovrn.db"
    - "{{ sovrn_datadir }}/oauth.db"
  failed_when: false
  changed_when: false
- name: WAL checkpoint each ZDS DB (best effort)
  ansible.builtin.shell: sqlite3 "{{ item }}" "PRAGMA wal_checkpoint(TRUNCATE);"
  loop: "{{ lookup('ansible.builtin.fileglob', sovrn_zds_db_dir + '/*.db', wantlist=True) }}"
  failed_when: false
  changed_when: false
- name: Final blob sync then litestream snapshot (blobs first per invariant)
  ansible.builtin.shell: |
    /usr/local/sbin/zds-blobs-sync.sh
    litestream snapshot -config /etc/litestream.yml    
  failed_when: false
- name: stalwart-cli snapshot NDJSON to private repo path
  ansible.builtin.shell: stalwart-cli snapshot > /var/lib/stalwart/snapshot-$(date +%F).ndjson
  failed_when: false
- name: Report backup ages
  ansible.builtin.shell: |
    litestream snapshots -config /etc/litestream.yml
    rclone lsl r2:{{ sovrn_r2_bucket }}/zds-blobs/ --config /etc/rclone.conf | tail -n 5    
  changed_when: false
  failed_when: false

• [ ] Step 2: Write recovery-bootstrap (authoritative restore, aborts on integrity_check != ok)

- name: Install binaries + base roles
  ansible.builtin.include_role:
    name: "{{ item }}"
  loop: [common, stalwart, zds, sovrnd, litestream, rclone]
- name: Restore every SQLite file from R2 (only when absent)
  ansible.builtin.shell: /usr/local/sbin/litestream-restore.sh
- name: integrity_check each restored DB (abort unless ok)
  ansible.builtin.shell: sqlite3 "{{ item }}" "PRAGMA integrity_check;"
  register: integrity
  failed_when: "'ok' not in integrity.stdout | lower"
  loop:
    - "{{ sovrn_stalwart_db }}"
    - "{{ sovrn_datadir }}/sovrn.db"
    - "{{ sovrn_datadir }}/oauth.db"
- name: integrity_check each ZDS DB
  ansible.builtin.shell: sqlite3 "{{ item }}" "PRAGMA integrity_check;"
  register: zds_integrity
  failed_when: "'ok' not in zds_integrity.stdout | lower"
  loop: "{{ lookup('ansible.builtin.fileglob', sovrn_zds_db_dir + '/*.db', wantlist=True) }}"
- name: Sync blobs down (before starting services — rows must not dangle)
  ansible.builtin.shell: rclone sync r2:{{ sovrn_r2_bucket }}/zds-blobs/ "{{ sovrn_zds_blob_base }}/" --config /etc/rclone.conf --delete
- name: Restore certs/config (never reissue) + start in order
  ansible.builtin.systemd:
    name: "{{ item }}"
    state: started
  loop: [unbound, stalwart, sovrnd, zds, caddy, litestream, zds-blobs-sync.timer]

• [ ] Step 3: Write the two playbooks

# deployment/playbooks/recover-backup.yml
- hosts: all
  become: true
  roles: [recovery-backup]
# deployment/playbooks/recover-bootstrap.yml
- hosts: all
  become: true
  roles: [recovery-bootstrap]

• [ ] Step 4: Verify stubs now pass prechecks

Run: just recovery-backup mx99.eu.sovrn.at -- --check and just recovery-bootstrap mx99.eu.sovrn.at -- --check Expected: playbook found (no “not yet implemented” error); check-mode green.

• [ ] Step 5: Commit

jj commit -m "feat(backup): T11 recovery-backup/bootstrap roles" deployment/roles/recovery-backup deployment/roles/recovery-bootstrap deployment/playbooks/recover-backup.yml deployment/playbooks/recover-bootstrap.yml

---

Task 5: Staged restore drill + runbook acceptance (mx99)

Files: - Modify: docs/runbooks/provision-cell.md (clear TODO(T4/T11) block, link new roles) - Test: live drill on mx99.eu.sovrn.at

• [ ] Step 1: Full drill — just recovery-backup mx99.eu.sovrn.at, manual provider move (power off old, new box re-linking primaries), just recovery-bootstrap mx99.eu.sovrn.at, then:

ssh mx99.eu.sovrn.at 'for db in /var/lib/stalwart/stalwart.db /var/lib/sovrn/sovrn.db /var/lib/sovrn/oauth.db /var/lib/zds/dbs/*.db; do echo "== $db"; sudo sqlite3 "$db" "PRAGMA integrity_check;"; done'
ssh mx99.eu.sovrn.at 'sudo litestream snapshots -config /etc/litestream.yml | head -n 20'
ssh mx99.eu.sovrn.at 'sudo rclone check /var/lib/zds/blobs/ r2:mx99.eu.sovrn.at/zds-blobs/ --config /etc/rclone.conf'

Expected: every integrity_check → ok; snapshots non-empty per DB (static + each <reversed>.db under litestream-zds/); rclone check exit 0; healthz green; relay-drain verify per relay-drain-verify.md; post RTO/RPO on T9.

• [ ] Step 2: Update runbook gaps — remove TODO(T4)/TODO(T11) block in provision-cell.md:76-85, replace with role names + manual-bucket reminder from §0.
• [ ] Step 3: Commit

jj commit -m "docs(backup): clear T4/T11 TODOs after staged drill" docs/runbooks/provision-cell.md

---

Self-review

• Spec coverage: SQLite replication incl. dynamic ZDS dir (Task 2) ✓; ZDS blobs rclone (Task 3) ✓; R2 secrets/retention policy (Task 1) ✓; T11 recovery pair (Task 4) ✓; restore drill (Task 5) ✓; manual new-host steps incl. R2 bucket creation (§0) ✓.
• Placeholder scan: no TBD/TODO; every step has exact paths, code, commands, expected output (checksum placeholder in Task 1 is fill-at-implementation, not deferred design).
• Type consistency: var names (sovrn_r2_*, sovrn_litestream_*, sovrn_zds_*) match across templates and tasks; R2 prefixes (litestream/, litestream-zds/, zds-blobs/) match restore scripts and runbooks.

Implementation complete (Tasks 1-5, untested on live host)

Stack on top of tzkupqok: - pukoovwp feat(backup): R2/litestream/rclone config surface (sovrn.yml defaults + host_vars/mx99.eu.sovrn.at/vars.yml; litestream 0.5.13 checksum resolved from upstream) - lopsvlnp feat(backup): litestream role — 3 static DBs + dir/watch:true for /var/lib/zds/dbs (new ZDS DBs replicate automatically, no re-render hook), restore-if-missing helper, wired into site.yml - vonzoqox feat(backup): rclone role — zds-blobs/ sync --delete on systemd timer, wired into site.yml - nsolvvxo feat(backup): T11 recovery-backup (failure-tolerant freshen, never wipes) + recovery-bootstrap (authoritative restore, integrity_check gate, re-own per service user) + recover-backup.yml/recover-bootstrap.yml; satisfies the Justfile stubs - ukywsxss docs(backup): cleared stale TODO(T4)/TODO(T11) in provision-cell.md; deployment.md layout marked accurate - vrklvpol chore(backup): stub guards now say “missing” instead of “not yet implemented”

Verify: ansible-playbook --syntax-check clean on all 3 playbooks; go build ./... OK.

Manual / untested (operator): - R2 bucket + bucket-scoped token creation per cell stays manual (keys into host_vars/<cell>/vault.yml). - Live restore drill on mx99.eu.sovrn.at not run — do it before pointing MX at any cell. - Known caveats for the drill: litestream snapshot (no path arg) assumed v0.5.x-wide; stalwart-cli NDJSON step is a guarded no-op until the CLI story settles; confirm R2 generations stay sane on first recovery (install-then-stop window).