Auto-provision vault secrets in bootstrap (fold provision-*-secrets) - Issue #3418287

Goal

Fold provision-*-secrets into just bootstrap so an operator never has to remember a provisioning recipe or paste anything via ansible-vault edit: derive the role from the domain, detect missing vault keys, ask ONE Y/n, then generate/pull/merge/encrypt them. Auto-persist the Stalwart credential after a green bootstrap. just update is fail-only (never provisions).

Implemented

deployment/scripts/_vault_lib.py: merge-aware primitives (ruamel round-trip), load_vault/save_vault, encrypt-then-shred, shared+host key registry, store access (secrets decrypt), INI/key-value parsers, role classification, required-key matrix, value validators, and a fix to encrypt_in_place (subprocess, not execvp).
provision-shared-secrets: owns all shared keys; generates OIDC/OAuth, pulls relay (smtp2go_api_key + smtp2go_smtp_credentials.toml), resend_api_key, and the fleet-consistent sovrn_metrics_password, computes the bcrypt hash, merges (preserves foreign keys). Store is authoritative for the metrics password.
provision-zds-secrets → provision-cell-secrets (renamed, old name removed): owns ZDS×5 + account-level R2×2 from rclone.conf [r2].
persist-bootstrap-secrets: merge-safe (preserves ZDS/R2), refuses a conflicting pinned credential.
ensure-bootstrap-secrets (new): role→required-key matrix, --check (fail-only), --dry-run, --yes; pre-checks store sources; single prompt; never hangs on non-TTY.
Justfile: bootstrap accepts cell/relay/infra.sovrn.at (metrics = ensure secrets then converge in update mode), consumes --yes, auto-runs persist-bootstrap-secrets after a green cell/relay play; update runs ensure-bootstrap-secrets --check first; provision-metrics-secret is now a thin wrapper.
New-in-bootstrap prerequisite guard for cells, role comments/fail_msgs, docs (deployment.md, provision-cell.md, metrics-box.md), and the resend_pds_api_key→resend_api_key name fix.
deployment/scripts/test-deploy (23 tests: vault merge/idempotency, store sourcing, ensure matrix, dry-run redaction, Justfile gates), wired into just test.

Verification

just test-deploy: ²³⁄₂₃ pass.
Real repo provision-shared-secrets --dry-run: succeeds, fully REDACTED.
ansible-playbook playbooks/site.yml --syntax-check: ok.
py_compile + YAML sweep: clean.

SECURITY: leaked secrets during verification (READ FIRST)

A real provision-shared-secrets --dry-run leaked the EXISTING fleet sovrn_oidc_key_pem and sovrn_oauth_attestation_key_b64 to the terminal: the old redaction scrubbed by string-replacing only NEW values, and multi-line/indented PEM body lines defeated replacement. The store-backed values (relay/resend/metrics) were masked by the shell filter and did NOT leak.

Fixed by redacting by KEY (all secret keys rendered as REDACTED before dump; no real value ever reaches stdout) in all three provisioners, with regression tests. Action: rotate both keys with just provision-shared-secrets --force and commit (OIDC rotation invalidates active sessions — pick a window). Treat the previously printed key material as compromised.

Note (unrelated)

deployment/inventory/host_vars/mx99.eu.sovrn.at/vault.yml is currently a working-copy deletion, unrelated to this work.

4 Comments

agent 3941198 Sep 19

Correction: the metrics-box domain is infra.sovrn.at, not infra.rtw.run — an earlier mix-up. All Justfile/script/doc literals plus the issue body have been corrected; the ‘pending inventory rename’ follow-up is void because inventory/hosts.yml already lists infra.sovrn.at. [email protected] is an unrelated pre-existing alert recipient and is unchanged.

agent 3b4f138 Oct 6

Superseded (2026-10-06): the vault scripts this built go away with Ansible (3beadb2, c3b0d04). Secrets now come from the secrets store as Colmena keys.

The security note above is still open: nothing records that sovrn_oidc_key_pem and sovrn_oauth_attestation_key_b64 were rotated after leaking. Their store copies (sovrn/shared/oidc-key.pem, sovrn/shared/oauth-attestation.key) are what the NixOS cell uses. Rotation is task 1 of ae42c89 (no live hosts, so it costs nothing now). Close this once that’s done.

agent 37431a8 Oct 6

Keys rotated (2026-10-06, ae42c89 task 1): - New sovrn/shared/oidc-key.pem and sovrn/shared/oauth-attestation.key, generated with go run ./cmd/gensecrets into a private temp dir (removed), encrypted into the store. Hashes changed for both. - The leaked copies are kept, renamed out of the store’s view: ~/projects/data/sovrn/shared/{oidc-key.pem,oauth-attestation.key}.age.leaked-2026-10-06. Delete them when you like; nothing reads them. - No host uses either key yet (no sovrn hosts are live), and both public halves are derived by sovrnd at runtime (client metadata inline JWKS), so nothing else needed updating. - The Ansible vault (deployment/inventory/group_vars/all/vault-shared.yml) still holds the old leaked values; it is deleted with Ansible (c3b0d04).

Ready to close when you’re satisfied.

agent 3542158 Oct 8

Closing (user, 2026-10-08): superseded. Ansible and its vaults are gone; secrets come from the age-encrypted secrets store via Colmena keys, generated per host by the fleet’s gen-host-secrets.