Your control-plane database holds everything you cannot regenerate: provider credentials sealed under the KEK, virtual keys, routes, budgets, RBAC assignments and the audit log.
Backing up PostgreSQL is a solved problem. The part that catches people out is that ROLTER_KEK is not in the dump — and a restore onto the wrong one fails silently.
Restoring a dump onto a deployment with a different ROLTER_KEK produces a database full of provider secrets that cannot be decrypted. Nothing fails at restore time. The process boots, the dashboard renders every provider, and the first symptom is a 401 from an upstream hours later — which looks like a provider problem, not a restore problem.
What has to be in the backup set#
| Item | Where it lives | What its loss costs |
|---|---|---|
| The control-plane database | PostgreSQL | Everything below, plus routes, budgets, RBAC and the audit log |
ROLTER_KEK | Your secret manager — not the database | Every sealed secret in that database, permanently |
ROLTER_KEY_PEPPER | Your secret manager | Every virtual key hash stops matching; keys must be reissued |
ROLTER_SESSION_PEPPER | Your secret manager | Everyone is signed out; open invitations, SCIM tokens and MFA recovery codes stop matching and must be reissued |
The gateway’s rolter.toml, if it has one | Config management | Reconstructible, but not from the dump |
A backup of the database alone is not a backup. The KEK is stored separately by design — that is what encryption at rest means — so the one thing pg_dump cannot carry is the one thing a restore needs most.
Back up#
--format=custom compresses and lets a restore be parallelised and selective. Nothing in the schema needs special dump options.
Store the KEK under a name that ties it to this database, not to “production” in general. Two deployments, two KEKs and one label is the most common way to restore onto the wrong key.
Restore#
Restore into a database migrated from scratch
Restoring over a live database can resurrect a schema the binary no longer expects.
Prove the KEK matches, before starting anything
rolter check --connect runs the same audit as part of pre-boot validation, so a deployment that already gates on rolter check gets this for free.
Start the control plane
Only once the verify passes. A store the control plane cannot read is not a store it should boot into.
When the KEK does not match#
There is no recovery path that does not involve the original KEK — AES-256-GCM is doing exactly what it was chosen to do. In order:
Find the original KEK
Check your secret manager’s version history, the deployment that produced the dump, and any sealed-secret manifest in the cluster. This is the only option that recovers the credentials themselves.
Re-enter the credentials
The rest of the database is unaffected — routes, virtual keys, budgets, RBAC and the audit log all survive. Clear the unreadable rows and add each provider credential again through the dashboard.
Do not start it anyway
Booting a control plane against a store it cannot read turns an unreadable credential into an upstream outage that gets blamed on the provider. Fix it before it starts.
Rotate the KEK#
Rotation reseals every stored secret from the old KEK to a new one. It is all-or-nothing: a partially rotated store is readable by neither key, so the whole pass runs in one transaction and a single row the old KEK cannot open aborts it.
Back up first
Rotation is not reversible without the old KEK.
Stop the writers
The control plane, and any rolter-seed run.
Dry run
The dry run reports what would be resealed and confirms the old KEK still opens all of it. Both values are read from the environment rather than from flags, because a KEK on a command line lands in shell history, in ps output, and in whatever collects a container’s arguments.
Apply, then publish the new KEK
Between these two commands a control-plane replica still holding the old value can no longer read a stored credential, so keep the window short. Gateways hold no KEK and need no restart: they keep serving their last snapshot and pick up the next one, which a control plane on the new KEK unseals.
Rotation does not touch ROLTER_KEY_PEPPER or ROLTER_SESSION_PEPPER. The digests under them are not reversible, so changing a pepper is a reissue, not a rotation: every virtual key for the first, and for the second every session, open invitation, SCIM token and MFA recovery code.
What gets audited#
rolter kek verify samples every sealed column in the schema:
| Table | Holds |
|---|---|
provider_keys | Upstream provider credentials |
sso_providers | SSO client secrets |
ldap_directories | LDAP service-account bind passwords |
alert_channels | Alert channel webhook secrets |
security_settings | A dashboard credential sealed before removal |
observability_connectors | Observability connector credentials |
invitations | Pending invitation links, for a resend |
user_totp_factors | TOTP second-factor shared secrets |
mcp_servers | MCP OAuth client secrets |
mcp_servers | MCP static bearer tokens and header api keys |
mcp_oauth_login_states | In-flight MCP OAuth PKCE verifiers |
mcp_oauth_sessions | MCP access and refresh tokens |
Run restore drills#
A backup nobody has restored is a hypothesis. Restore into a scratch database on whatever cadence your recovery objective implies, and make “does rolter kek verify pass” one of the steps.