> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Rolter Production Deployment: Security and Ops Checklist > Checklist and guidance for running Rolter in production: TLS termination, secrets management, horizontal scaling, health probes, and database backups. Running Rolter in production requires a few deliberate decisions around network exposure, secret management, and operational hygiene. This page walks you through the key concerns so you can deploy with confidence and keep your gateway secure, observable, and easy to operate at scale. Never expose the control plane (`rolter-control`, port 4001) to the public internet. It has full write access to your entire gateway configuration — providers, virtual keys, routes, and stored secrets. Place it on a private network and restrict access to trusted operators only. ## Checklist Work through these steps before sending production traffic to your gateway: Do not invent the secrets by hand. `rolter init` writes a production `rolter.toml` and a `.env` holding five distinct 256-bit secrets: the KEK, the admin token, the internal token, the virtual-key pepper and the session pepper: ```bash rolter init --profile production \ --database-url postgres://rolter@db.internal:5432/rolter \ --redis-url redis://cache.internal:6379 ``` The generated `rolter.toml` has no `[[virtual_keys]]` section: the example's public `sk-rolter-dev` key is left out on purpose. Create keys in the dashboard or API. `rolter check` fails if a config file still declares it. It is non-interactive, so it also runs inside a Dockerfile, a Helm hook or a CI job. Pipe it straight into a secret manager instead of leaving it on disk: ```bash rolter init --print --profile production | your-secret-tool import - ``` `rolter init` refuses to overwrite an existing file without `--force`. Regenerating `ROLTER_KEK` strands every credential encrypted under the old one — that failure is total, delayed, and looks like data corruption. Both peppers belong in place before the first user signs in. Adding `ROLTER_KEY_PEPPER` later invalidates every virtual key already issued, and adding `ROLTER_SESSION_PEPPER` later signs everyone out and voids open invitations, SCIM tokens and MFA recovery codes. The `--profile local` variant exists for development and deliberately does **not** pass the checks in the next step. ```bash rolter check --strict --config rolter.toml ``` This is a gate, not a linter: it exits non-zero on anything that would produce a silently-degraded process. The failure it was built for is the one that produces no error at all — a missing `ROLTER_KEK` makes the control plane store provider credentials **unsealed** with only a log warning, so the deployment looks completely healthy while a credential you believed was encrypted at rest is not. Add `--connect` to also probe that the database and Redis accept a connection, and, when `ROLTER_SMTP_URL` is set, that the mail relay accepts a connection, negotiates TLS and takes the credential (a warning that names the stage, never the address or the password). The Helm chart runs exactly this as an init container by default (`preflight.enabled`), against the same environment block the workload gets. See [Cluster deployment](/deployment/cluster). Rolter's gateway listens on plain HTTP by default. Put a TLS-terminating ingress controller, reverse proxy (e.g. nginx, Caddy, Traefik), or cloud load balancer in front of port 4000. Your clients connect over HTTPS to the ingress; the ingress forwards cleartext HTTP to the gateway over your private network. Keep the control plane (port 4001) off public-facing load balancers entirely. Route admin traffic through a VPN or SSH tunnel instead. The master key is the AES-256-GCM key-encryption key (KEK) used to protect all provider API secrets at rest. Generate a cryptographically random 32-byte value and store it in your secrets manager: ```bash openssl rand -hex 32 ``` Inject it into the control plane container as the `ROLTER_KEK` environment variable. Gateways never read it: they receive provider keys already unsealed over the token-guarded `/internal/snapshot`, so keep the KEK off them. Back it up securely — losing this key means losing access to all stored provider credentials, and a database dump does not contain it. See [Backup, restore and KEK rotation](/deployment/backup-and-restore). Never write API keys or passwords inline in `rolter.toml` or into container images. Instead: - Reference provider keys with `api_key_env` in your config — Rolter reads the key from the named environment variable at runtime. - Supply `ROLTER_DATABASE_URL`, `ROLTER_REDIS_URL`, and `CLICKHOUSE_URL` as environment variables or inject them from a secrets manager (AWS Secrets Manager, HashiCorp Vault, Kubernetes Secrets, etc.). ```toml [[providers]] name = "openai-prod" kind = "openai" api_base = "https://api.openai.com" api_key_env = "OPENAI_API_KEY" # never use api_key = "sk-..." ``` The gateway is stateless — it reads its working config from an in-memory snapshot that is refreshed from Redis. You can run as many gateway replicas as you need behind a load balancer. When you push a config change through the control plane, all replicas receive the hot-reload event via Redis and swap in the new config simultaneously without restarting. PostgreSQL and ClickHouse are shared across all replicas, so request logs and config changes are always consistent regardless of which replica handles a request. The gateway can run independently from the control plane using just a `rolter.toml` file — no Redis, PostgreSQL, or ClickHouse required. This is ideal for zero-dependency edge deployments or resource-constrained environments where you manage config changes through file updates instead of the dashboard. Both binaries serve two endpoints that answer two different questions. Wire each to its own probe — never point both probes at the same path. ```text GET /healthz → liveness (process is up; checks no dependency) GET /readyz → readiness (this instance can actually serve right now) ``` A failing **liveness** probe restarts the pod, so `/healthz` deliberately checks nothing outside the process — otherwise a PostgreSQL blip would restart your whole control-plane deployment instead of draining it. A failing **readiness** probe only removes the pod from the Service. On the gateway, `/readyz` returns `503` while the instance is draining, and while it has never received a config snapshot in a format it reads (see [Upgrading](/deployment/upgrading#upgrade-order)). On the control plane, it returns `503` until the PostgreSQL pool hands out a connection, every migration is applied, and `ROLTER_KEK` parses when set; Redis and ClickHouse are treated as degraded, not unready. ```yaml # Example Kubernetes probe config livenessProbe: httpGet: path: /healthz port: 4000 initialDelaySeconds: 5 periodSeconds: 10 readinessProbe: httpGet: path: /readyz port: 4000 initialDelaySeconds: 3 periodSeconds: 5 # control plane only: hold liveness off while first-boot migrations run # against a cold database startupProbe: httpGet: path: /healthz port: 4001 periodSeconds: 5 failureThreshold: 60 ``` The bundled Helm chart already wires all three probes this way. Alert on `/readyz` staying down past a database failover, and on `/healthz` flapping. PostgreSQL is the single source of truth for all Rolter configuration — providers, routes, virtual keys, RBAC policies, and budgets. Schedule regular automated backups using `pg_dump` or your cloud provider's managed backup tooling. Test restore procedures before you need them. ClickHouse holds request logs and analytics data. Back it up according to your own retention and compliance requirements, but losing ClickHouse data does not affect gateway operation or config. Use `RUST_LOG=info` (or `warn`) in production. The `info` level gives you startup messages, config reload events, and error conditions without overwhelming your log aggregation system. ```bash RUST_LOG=info ``` Switch to `debug` temporarily when diagnosing a specific issue, then revert. You can also scope debug logging to a single crate to reduce noise: ```bash RUST_LOG=info,rolter_gateway=debug ``` If you are running in Kubernetes or another container orchestrator, set explicit CPU and memory requests and limits on gateway containers. The gateway is a Rust process with predictable memory usage under steady load — baseline it in staging and set limits with appropriate headroom. ```yaml resources: requests: cpu: "250m" memory: "128Mi" limits: cpu: "1000m" memory: "512Mi" ``` ## Wire transparency Rolter is invisible to your upstream providers. Outbound requests carry no gateway-identifying marks — no `User-Agent`, no added `X-*` or `Via` headers, and no metadata injected into the request body. The only headers sent upstream are the functionally required ones: `content-type`, the provider's own auth header, and `anthropic-version` for Anthropic. Responses returned to your clients likewise gain no Rolter-added headers. No configuration is needed — this is the default and only behavior, locked in by wire-level tests. ## Secrets reference Here is a consolidated view of every secret and credential the stack requires: | Variable | Used by | Notes | | ----------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ROLTER_KEK` | control | 32-byte hex; AES-256-GCM KEK. Rotate with `rolter kek rotate` — see [Backup & restore](/deployment/backup-and-restore) | | `ROLTER_KEK_OLD` | `rolter kek rotate` only | the KEK being rotated away from; never set on a running process | | `ROLTER_ADMIN_TOKEN` | gateway, control | bearer token for the management API and `/internal/snapshot`; at least 32 characters. Generated by `rolter init` | | `ROLTER_INTERNAL_TOKEN` | gateway, control | shared secret the gateways present on snapshot polls. Generated by `rolter init` | | `ROLTER_KEY_PEPPER` | gateway, control | mixed into virtual-key digests. Identical everywhere; changing it invalidates every virtual key. Generated by `rolter init` | | `ROLTER_SESSION_PEPPER` | control | mixed into the digests of sign-in sessions, invitation links, SCIM tokens and MFA recovery codes. Identical on every control-plane replica; changing it signs everyone out and voids those. Generated by `rolter init` | | `ROLTER_DATABASE_URL` | control | Postgres connection string, e.g. `postgres://user:pass@host:5432/rolter` | | `ROLTER_REDIS_URL` | gateway, control | e.g. `redis://host:6379` | | `CLICKHOUSE_URL` | gateway, control | e.g. `http://host:8123` | | `OPENAI_API_KEY` | gateway (via `api_key_env`) | Referenced by provider config, not stored in TOML | | `ANTHROPIC_API_KEY` | gateway (via `api_key_env`) | Referenced by provider config, not stored in TOML | You can reference any arbitrarily named environment variable in `api_key_env`. If you manage provider keys in a secrets manager that injects them under custom names (e.g. `PROD_OPENAI_KEY`), set `api_key_env = "PROD_OPENAI_KEY"` in your provider config — Rolter will read from that variable at startup. ## Rotate provider API keys periodically Rotate your upstream provider API keys on a schedule (e.g. every 90 days) or immediately if a key is suspected to be compromised. Because keys are referenced by environment variable name rather than stored inline, rotation is a two-step operation: 1. Update the secret value in your secrets manager or environment. 2. Restart or re-deploy the gateway containers to pick up the new value. If you use the control plane to manage providers, you can update the encrypted secret through the dashboard without touching the TOML file or restarting containers. Running without public internet? See the [air-gapped installation & operation guide](/deployment/air-gapped) for mirror-based install paths and every outbound path, including the release check you turn off with `ROLTER_UPDATE_CHECK=false`.