Rolter Production Deployment: Security and Ops Checklist

Do not invent the secrets by hand. rolter init writes a production rolter.toml and a .env holding five distinct 256-bit secrets: the KEK, the admin token, the internal token, the virtual-key pepper and the session pepper:

rolter init --profile production \            --database-url postgres://rolter@db.internal:5432/rolter \            --redis-url redis://cache.internal:6379

The generated rolter.toml has no [[virtual_keys]] section: the example’s public sk-rolter-dev key is left out on purpose. Create keys in the dashboard or API. rolter check fails if a config file still declares it.

It is non-interactive, so it also runs inside a Dockerfile, a Helm hook or a CI job. Pipe it straight into a secret manager instead of leaving it on disk:

rolter init --print --profile production | your-secret-tool import -

Both peppers belong in place before the first user signs in. Adding ROLTER_KEY_PEPPER later invalidates every virtual key already issued, and adding ROLTER_SESSION_PEPPER later signs everyone out and voids open invitations, SCIM tokens and MFA recovery codes.

The --profile local variant exists for development and deliberately does not pass the checks in the next step.

rolter check --strict --config rolter.toml

This is a gate, not a linter: it exits non-zero on anything that would produce a silently-degraded process. The failure it was built for is the one that produces no error at all — a missing ROLTER_KEK makes the control plane store provider credentials unsealed with only a log warning, so the deployment looks completely healthy while a credential you believed was encrypted at rest is not.

Add --connect to also probe that the database and Redis accept a connection, and, when ROLTER_SMTP_URL is set, that the mail relay accepts a connection, negotiates TLS and takes the credential (a warning that names the stage, never the address or the password).

The Helm chart runs exactly this as an init container by default (preflight.enabled), against the same environment block the workload gets. See Cluster deployment.

Rolter’s gateway listens on plain HTTP by default. Put a TLS-terminating ingress controller, reverse proxy (e.g. nginx, Caddy, Traefik), or cloud load balancer in front of port 4000. Your clients connect over HTTPS to the ingress; the ingress forwards cleartext HTTP to the gateway over your private network.

Keep the control plane (port 4001) off public-facing load balancers entirely. Route admin traffic through a VPN or SSH tunnel instead.

The master key is the AES-256-GCM key-encryption key (KEK) used to protect all provider API secrets at rest. Generate a cryptographically random 32-byte value and store it in your secrets manager:

openssl rand -hex 32

Inject it into the control plane container as the ROLTER_KEK environment variable. Gateways never read it: they receive provider keys already unsealed over the token-guarded /internal/snapshot, so keep the KEK off them. Back it up securely — losing this key means losing access to all stored provider credentials, and a database dump does not contain it. See Backup, restore and KEK rotation.

Never write API keys or passwords inline in rolter.toml or into container images. Instead:

• Reference provider keys with api_key_env in your config — Rolter reads the key from the named environment variable at runtime.
• Supply ROLTER_DATABASE_URL, ROLTER_REDIS_URL, and CLICKHOUSE_URL as environment variables or inject them from a secrets manager (AWS Secrets Manager, HashiCorp Vault, Kubernetes Secrets, etc.).

[[providers]]name = "openai-prod"kind = "openai"api_base = "https://api.openai.com"api_key_env = "OPENAI_API_KEY"   # never use api_key = "sk-..."

The gateway is stateless — it reads its working config from an in-memory snapshot that is refreshed from Redis. You can run as many gateway replicas as you need behind a load balancer. When you push a config change through the control plane, all replicas receive the hot-reload event via Redis and swap in the new config simultaneously without restarting.

PostgreSQL and ClickHouse are shared across all replicas, so request logs and config changes are always consistent regardless of which replica handles a request.

Both binaries serve two endpoints that answer two different questions. Wire each to its own probe — never point both probes at the same path.

GET /healthz  →  liveness   (process is up; checks no dependency)GET /readyz   →  readiness  (this instance can actually serve right now)

A failing liveness probe restarts the pod, so /healthz deliberately checks nothing outside the process — otherwise a PostgreSQL blip would restart your whole control-plane deployment instead of draining it. A failing readiness probe only removes the pod from the Service.

On the gateway, /readyz returns 503 while the instance is draining, and while it has never received a config snapshot in a format it reads (see Upgrading). On the control plane, it returns 503 until the PostgreSQL pool hands out a connection, every migration is applied, and ROLTER_KEK parses when set; Redis and ClickHouse are treated as degraded, not unready.

# Example Kubernetes probe configlivenessProbe:  httpGet:    path: /healthz    port: 4000  initialDelaySeconds: 5  periodSeconds: 10readinessProbe:  httpGet:    path: /readyz    port: 4000  initialDelaySeconds: 3  periodSeconds: 5# control plane only: hold liveness off while first-boot migrations run# against a cold databasestartupProbe:  httpGet:    path: /healthz    port: 4001  periodSeconds: 5  failureThreshold: 60

PostgreSQL is the single source of truth for all Rolter configuration — providers, routes, virtual keys, RBAC policies, and budgets. Schedule regular automated backups using pg_dump or your cloud provider’s managed backup tooling. Test restore procedures before you need them.

ClickHouse holds request logs and analytics data. Back it up according to your own retention and compliance requirements, but losing ClickHouse data does not affect gateway operation or config.

Use RUST_LOG=info (or warn) in production. The info level gives you startup messages, config reload events, and error conditions without overwhelming your log aggregation system.

RUST_LOG=info

Switch to debug temporarily when diagnosing a specific issue, then revert. You can also scope debug logging to a single crate to reduce noise:

RUST_LOG=info,rolter_gateway=debug

If you are running in Kubernetes or another container orchestrator, set explicit CPU and memory requests and limits on gateway containers. The gateway is a Rust process with predictable memory usage under steady load — baseline it in staging and set limits with appropriate headroom.

resources:  requests:    cpu: "250m"    memory: "128Mi"  limits:    cpu: "1000m"    memory: "512Mi"