> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Backup, Restore and KEK Rotation > How to back up the Rolter control-plane database, restore it safely, verify that ROLTER_KEK still opens your stored credentials, and rotate the key-encryption key without splitting the fleet. Your control-plane database holds everything you cannot regenerate: provider credentials sealed under the KEK, virtual keys, routes, budgets, RBAC assignments and the audit log. Backing up PostgreSQL is a solved problem. The part that catches people out is that **`ROLTER_KEK` is not in the dump** — and a restore onto the wrong one fails silently. Restoring a dump onto a deployment with a different `ROLTER_KEK` produces a database full of provider secrets that cannot be decrypted. Nothing fails at restore time. The process boots, the dashboard renders every provider, and the first symptom is a `401` from an upstream hours later — which looks like a provider problem, not a restore problem. ## What has to be in the backup set | Item | Where it lives | What its loss costs | | ------------------------------------------ | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------- | | The control-plane database | PostgreSQL | Everything below, plus routes, budgets, RBAC and the audit log | | `ROLTER_KEK` | Your secret manager — **not** the database | Every sealed secret in that database, permanently | | `ROLTER_KEY_PEPPER` | Your secret manager | Every virtual key hash stops matching; keys must be reissued | | `ROLTER_SESSION_PEPPER` | Your secret manager | Everyone is signed out; open invitations, SCIM tokens and MFA recovery codes stop matching and must be reissued | | The gateway's `rolter.toml`, if it has one | Config management | Reconstructible, but not from the dump | A backup of the database alone is not a backup. The KEK is stored separately by design — that is what encryption at rest means — so the one thing `pg_dump` cannot carry is the one thing a restore needs most. ## Back up ```bash pg_dump "$ROLTER_DATABASE_URL" --format=custom --file rolter-$(date +%F).dump ``` `--format=custom` compresses and lets a restore be parallelised and selective. Nothing in the schema needs special dump options. Store the KEK under a name that ties it to **this** database, not to "production" in general. Two deployments, two KEKs and one label is the most common way to restore onto the wrong key. ## Restore Restoring over a live database can resurrect a schema the binary no longer expects. ```bash createdb rolter_restored pg_restore --dbname "$RESTORED_URL" --no-owner --no-privileges rolter-2026-09-02.dump ``` ```bash ROLTER_DATABASE_URL=$RESTORED_URL ROLTER_KEK=... rolter kek verify ``` ```text ROLTER_KEK opens all 12 sampled secrets across 3 columns ``` `rolter check --connect` runs the same audit as part of pre-boot validation, so a deployment that already gates on `rolter check` gets this for free. Only once the verify passes. A store the control plane cannot read is not a store it should boot into. ### When the KEK does not match ```text error ROLTER_KEK does not match the one this store was sealed with - provider_keys.ciphertext holds upstream provider credentials — 3 of 3 sampled rows could not be decrypted ``` There is no recovery path that does not involve the original KEK — AES-256-GCM is doing exactly what it was chosen to do. In order: Check your secret manager's version history, the deployment that produced the dump, and any sealed-secret manifest in the cluster. This is the only option that recovers the credentials themselves. The rest of the database is unaffected — routes, virtual keys, budgets, RBAC and the audit log all survive. Clear the unreadable rows and add each provider credential again through the dashboard. Booting a control plane against a store it cannot read turns an unreadable credential into an upstream outage that gets blamed on the provider. Fix it before it starts. ## Rotate the KEK Rotation reseals every stored secret from the old KEK to a new one. It is all-or-nothing: a partially rotated store is readable by neither key, so the whole pass runs in one transaction and a single row the old KEK cannot open aborts it. ```bash pg_dump "$ROLTER_DATABASE_URL" --format=custom --file pre-rotation.dump ``` Rotation is not reversible without the old KEK. The control plane, and any `rolter-seed` run. ```bash kubectl scale deployment/rolter-control --replicas 0 ``` ```bash export ROLTER_KEK_OLD="" export ROLTER_KEK="$(openssl rand -hex 32)" rolter kek rotate ``` The dry run reports what would be resealed and confirms the old KEK still opens all of it. Both values are read from the environment rather than from flags, because a KEK on a command line lands in shell history, in `ps` output, and in whatever collects a container's arguments. ```bash rolter kek rotate --apply kubectl scale deployment/rolter-control --replicas 3 ``` Between these two commands a control-plane replica still holding the old value can no longer read a stored credential, so keep the window short. Gateways hold no KEK and need no restart: they keep serving their last snapshot and pick up the next one, which a control plane on the new KEK unseals. Rotation does not touch `ROLTER_KEY_PEPPER` or `ROLTER_SESSION_PEPPER`. The digests under them are not reversible, so changing a pepper is a reissue, not a rotation: every virtual key for the first, and for the second every session, open invitation, SCIM token and MFA recovery code. ## What gets audited `rolter kek verify` samples every sealed column in the schema: | Table | Holds | | -------------------------- | -------------------------------------------- | | `provider_keys` | Upstream provider credentials | | `sso_providers` | SSO client secrets | | `ldap_directories` | LDAP service-account bind passwords | | `alert_channels` | Alert channel webhook secrets | | `security_settings` | A dashboard credential sealed before removal | | `observability_connectors` | Observability connector credentials | | `invitations` | Pending invitation links, for a resend | | `user_totp_factors` | TOTP second-factor shared secrets | | `mcp_servers` | MCP OAuth client secrets | | `mcp_servers` | MCP static bearer tokens and header api keys | | `mcp_oauth_login_states` | In-flight MCP OAuth PKCE verifiers | | `mcp_oauth_sessions` | MCP access and refresh tokens | ## Run restore drills A backup nobody has restored is a hypothesis. Restore into a scratch database on whatever cadence your recovery objective implies, and make "does `rolter kek verify` pass" one of the steps.