Rolter reads its configuration from a TOML file named rolter.toml by default. The repository ships a fully annotated example file you can copy as a starting point. Once you have the file, open it in your editor and work through each section before you start the gateway.
Unknown keys are ignored — and reported#
Rolter never rejects a rolter.toml over a key it does not recognise. That is what keeps the file compatible in both directions: a config written for a newer release still loads on an older binary, so rolling a release back is not an outage.
The trade-off is that a typo would otherwise be silent. connect_sesc under [timeouts] produces no error — the default simply applies, and a misspelled api_key_env leaves a provider with no credential until the first upstream 401.
So rolter names the key instead, without changing what it accepts. Every process that reads the file reports it, as a warning in the startup log:
rolter gateway --config rolter.tomlrolter control --config rolter.tomlrolter easy-uprolter-seed --import rolter.toml, before it writes anything to the database
rolter check reports the same keys without starting anything, which is the form to run before a deploy:
Either way you get the full path to every unrecognised key, including keys nested inside tables and inside arrays of tables, plus the nearest real key when one is close enough to be a likely typo.
These are always warnings. The file is valid, the gateway starts, and an import still runs — an unknown key never stops a process. The one exception is deliberate: --strict turns them into a non-zero exit, which is what you want in CI when a config change is under review.
A config served from the database (the control plane’s snapshot, which is what rolter easy-up --database-url ... and any DB-backed deployment runs on) has no unknown keys to report: it is built from validated rows, not from a file you typed.
Which settings a control plane owns#
rolter.toml is a gateway’s whole configuration only when the gateway reads it itself (rolter gateway --config rolter.toml, with no --snapshot-url). A gateway that polls a control plane applies that control plane’s snapshot instead, and a control plane with a database builds the snapshot from the database. For the tables below the database is then the source of truth: what an admin saves on the dashboard reaches every gateway, and the table in rolter.toml is not what the snapshot carries.
Table in rolter.toml | Edited in the dashboard under | Page |
|---|---|---|
[compatibility] | Settings → Compatibility | Compatibility |
[client] | Settings → Client Settings | Client settings |
[model_defaults] | Settings → Model Settings | Model settings |
[adaptive_routing] | Adaptive Routing → Settings | Adaptive routing |
[logging]: sample_rate, payload_capture, ui_events | Observability → Logs Settings | Log settings |
[retry], [timeouts], [queue] | Settings → Performance Tuning | Performance tuning |
[feature_flags] | Settings → Feature Flags | Feature flags |
[security] | the Security screen |
rolter-seed --import does not carry these tables into the database. The only parts of them it reads from the file are [logging.payload_capture] and ui_events, and only when the file writes them. So in a database-backed deployment a value written in rolter.toml changes nothing for these tables, and each of them takes effect from the file only for a gateway that reads it directly. The exception is a control plane that loads the file itself, described under the list. The sections for these tables repeat the caveat.
A control plane started with --config as well as a database (rolter easy-up runs it this way) starts its snapshot from the file and lets the database take over table by table. For every table in the list above except [feature_flags], gateways get the file’s table until something other than the defaults is saved on its screen, and the saved table from then on. Saving the defaults back over a file value that differs hands the file’s value back, so clear a value like that by editing the file. [feature_flags] always comes from the database in that setup. In [logging], clickhouse_url, batch_max, flush_ms and queue_capacity stay the file’s either way, since the database holds no copy of them.
Two tables work the other way round. [currency] and [egress] have no stored copy, so a control plane reads them from the file it was started with (rolter control --config rolter.toml), and it checks the URLs and prices written through its API against them.
schema_version#
A top-level key, written before the first table, that stamps which spelling of the config the file uses. You do not have to write it.
The schema version the file was written for. The current version is 3. A file without the key is read as the oldest version, which is right for every config written before the stamp existed. A file stamped with a version newer than the running build understands still loads unchanged, and the keys that build does not know are ignored. rolter config export writes the stamp, and rolter check --migrations reports a file that is behind. See Upgrading.
[server]#
The [server] section controls how the gateway process binds and exposes its built-in endpoints.
The address the gateway listens on. Use 127.0.0.1 to bind only to localhost.
Default: "0.0.0.0"
The path Rolter serves its Prometheus metrics on. Change this when another app or sidecar already owns /metrics behind the same reverse proxy. The value must start with / and must not collide with any built-in gateway route (/healthz, /v1/chat/completions, /v1/completions, /v1/messages, /v1/models).
Default: "/metrics"
A deployment-wide secret mixed into virtual-key digests. Keeping a pepper means a leaked digest is useless without it. Omit this field and set the ROLTER_KEY_PEPPER environment variable instead — that keeps the secret out of your config file entirely.
Maximum accepted request body size in bytes (defaults to 32 MiB). Raises Axum’s default 2 MiB limit so large prompts, base64 images, or audio uploads are not rejected. An oversized body gets an OpenAI-style HTTP 413 error.
Default: "33554432"
Enforces virtual-key authentication when the effective key set is empty. When true, requests are always denied if no keys exist; when false, keyless access is allowed; when omitted, managed gateways fail closed while static local configurations stay keyless for development.
[tls]#
Outbound TLS trust for upstream providers. Rolter verifies HTTPS with the host’s system trust store; this table adds private certificate authorities on top of it, which an air-gapped or self-hosted deployment with its own CA needs.
Paths of PEM files whose certificates are added to the system trust roots for every provider’s upstream TLS. Certificate-chain and hostname verification stay on. The ROLTER_CA_BUNDLE environment variable replaces this list with the one path it names, which suits a container that mounts a single bundle. A provider’s own ca_bundles replaces the result for that provider, and ca_bundles = [] on a provider trusts the system roots only. Every file is checked while the config loads, so a path that is missing, unreadable, empty or not valid PEM stops startup with an error that names it.
Default: "[]"
[[providers]]#
Each [[providers]] block registers an upstream provider the gateway can forward to. You can define as many as you need; each one is identified by its unique name. The flat [[providers]] array is the deprecated spelling of the readonly tier; see “Config tiers: readonly and default” below for [[providers.readonly]] and [[providers.default]].
A unique identifier for this provider. Routes reference providers by this name.
A stable, URL-safe identity for provider-slug/model addressing. When omitted, Rolter automatically derives a slug from name.
The wire protocol this provider speaks. One of openai, anthropic, openai_compatible, ollama, ollama_cloud, llama_cpp, openrouter, tei, azure_openai, bedrock, vertex, gemini, gemini_native, gemini_interactions, mistral, groq, xai, meta_llama_api, cohere, perplexity, together, fireworks, databricks, aleph_alpha, nebius, ovhcloud, scaleway, deepseek, qwen, zhipu, kimi, ernie, doubao, hunyuan, yi, minimax, baichuan, gigachat, yandex_gpt, cloud_ru, mts_ai, naver, upstage, rinna, rakuten, sarvam, krutrim, falcon, system_one, or infinity. Use openai_compatible for vLLM, TGI, Ollama, or any other OpenAI-schema endpoint.
The provider’s base URL with no trailing slash, e.g. https://api.openai.com.
Whether to include /v1 depends on the provider kind. For openai-shaped kinds (openai, openai_compatible, ollama, ollama_cloud, llama_cpp, anthropic) Rolter appends the full /v1/... path itself, so the base must stop at the host: https://api.openai.com, not https://api.openai.com/v1. Including it produces /v1/v1/chat/completions and every request 404s.
For the remaining kinds — mistral, groq, openrouter, azure_openai and the other hosted OpenAI-compatible clouds — Rolter strips the /v1 it was asked for, so the base has to supply one: https://api.mistral.ai/v1.
A base that would double the prefix is rejected on write rather than stored. The dashboard states the rule for the selected kind and previews the resolved upstream URL as you type, naming the endpoint that kind is called at: /v1/messages for anthropic, /v1/embeddings for tei, a generateContent path for gemini_native, /v1/chat/completions for the rest. GET /api/v1/provider-kinds returns the same rule, the endpoint and the header the key is sent in per kind if you are automating provider creation.
The name of an environment variable to read the API key from. Preferred over api_key because it keeps secrets out of your config file. The gateway reads it from its own environment for every request, and the control plane reads it from its own for Test connection, so set it in both when they run as separate processes. Written as the single api_key_env it is deprecated in favour of an api_keys entry with env; it keeps working in every file and Rolter logs a warning on load.
An inline API key. Use api_key_env instead wherever possible. Deprecated in favour of an api_keys entry with key; it keeps working in every file and Rolter logs a warning on load.
An outbound proxy URL to route traffic through, e.g. http://corp-proxy:3128. Supports http://, https://, socks5://, and socks5h:// schemes. Deprecated in favour of a one-element egress_proxies; it keeps working in every file and Rolter logs a warning on load.
A pool of outbound proxy URLs that requests rotate through round-robin. When the list is empty, a legacy egress_proxy is used as a one-entry pool. Write an authenticated proxy as ${ENV_VAR} so its credentials never appear in serialized configuration or database rows.
Let this provider point at an api_base its kind would otherwise refuse. A hosted kind names one operator, so Rolter pins its base URL: an openrouter provider must be https://openrouter.ai/api/v1. That pin is what stops a config edit from sending an OpenRouter key to a host of the editor’s choosing.
Set this when the endpoint really is somewhere else — a corporate egress proxy or internal gateway in front of the vendor, an endpoint that re-implements the vendor’s API, or a local fake you use to exercise the dialect in tests. It relaxes the host pin and nothing else: the credential must still come from api_key_env, never an inline api_key.
rolter check reports every provider that sets it, and rolter check --strict fails on it, so the opt-out stays visible rather than becoming a copied default.
A provider created in the dashboard has the same opt-out: the provider sheet shows an Allow a custom base URL switch for a pinned kind, and the providers API takes allow_custom_api_base. A custom base without it is refused when the provider is saved. rolter-seed --import keeps the flag, and rolter config export writes it back out.
Default: "false"
Keep this provider to one project. By default a provider is org-wide: every project of the organization can use it, which is how projects share fallbacks. With project_scoped = true, only virtual keys minted in the project can reach it, whether by a route or as provider-slug/model, and a key from another project gets 404 model_not_found.
A file names no projects, so the flag means “the project this file is imported into”: it takes effect through rolter-seed --import and rolter config export writes it back. A gateway that loads the file directly ignores it, because the keys in a file carry no project. It logs a startup warning naming each flagged provider or group, and rolter check reports the same as a warning; rolter-seed --import never warns. The same key exists on [[provider_groups]]. A project-scoped group may hold that project’s providers and org-wide ones; an org-wide group may hold only org-wide providers, and the import stops with an error naming the group when it does not.
In the dashboard and API, the provider’s project_id is the same scope. See Project-scoped providers.
Default: "false"
Make active health checks send a real max_tokens = 1 completion to this provider instead of the free /v1/models liveness probe, so a healthy result proves end-to-end inference. Costs a few tokens on every sweep — leave it off unless you need inference-level health. Recorded as source = llm_call in provider_health_events and fed through the same failure/recovery thresholds as free probes. The request goes where a forwarded one would, in the provider kind’s own format, and unlike the free probe it counts a 404/405 or a 401/403 answer as a failure. See Health checks.
Default: "false"
The upstream model id the also_track_via_llm_call completion targets (e.g. gpt-4o-mini). Required when the flag is on; otherwise the checker warns and falls back to the free probe.
An optional statuspage.io-style status JSON URL (e.g. https://status.anthropic.com/api/v2/status.json). When set, a background poll records the provider’s public status as a secondary health signal.
Provider-specific list of PEM CA bundle file paths. When set, overrides the global [tls].ca_bundles setting for tls verification against this provider.
How this provider handles instruction roles: openai keeps developer and system messages distinct, system_only lowers developer to system, and anthropic sends top-level system blocks. When omitted, native OpenAI keeps both roles, Anthropic uses top-level system blocks, and OpenAI-compatible providers use system_only.
Per-upstream-model overrides of role_profile, keyed by the upstream model id, e.g. model_role_profiles = { "my-model" = "openai" }. Use it for a custom chat template that explicitly supports the developer role.
[[providers.api_keys]]#
Multiple weighted API keys for one provider. When present, this list takes precedence over the single api_key/api_key_env pair. Providers cap throughput per key, so rotating across keys multiplies your effective RPM/TPM and isolates a single revoked or throttled key from taking the whole provider down. On every request the gateway picks one key from the pool with probability proportional to its weight. A key that gets 429 or 401 upstream is parked on a cooldown and the request retries immediately on a sibling key; the target itself stays in rotation, and key cooldown trips are visible as rolter_key_cooldowns_tripped_total in /metrics.
An inline API key. Use env instead wherever possible.
The name of an environment variable to read the API key from.
The Postgres-backed control plane stores one credential per provider. rolter-seed --import therefore keeps the first entry that names an env and logs a warning naming how many it dropped, and rolter config export writes that one back as api_key_env. Keep a multi-key pool in the file the gateway loads directly, not in a file you round-trip through the control plane.
[providers.kv_events]#
vLLM ZeroMQ KV-event subscription settings for the precise_cache_aware strategy. See cache-aware routing.
Subscriber endpoint, e.g. tcp://vllm-0:5557.
[providers.lmcache]#
LMCache controller signal settings for the lmcache_aware strategy. See cache-aware routing.
HTTP URL returning { "occupancy": 0..1, "cache_available": bool }.
[[provider_groups]]#
A provider group lets a client send group-slug/model and have Rolter spread the request across several providers with a load-balancing strategy, without a [[routes]] entry per model. The slug shares the provider slug namespace, so one slug names at most one provider or group, and a provider wins a tie. A group with no members never resolves.
The group’s display name.
A stable, URL-safe identity for group-slug/model addressing. When omitted, Rolter derives one from name.
How requests are spread across the members. Accepts the same values as a route’s strategy.
Default: "round_robin"
The providers the group spreads requests across, written as [[provider_groups.members]] entries. See below.
Keep this group to one project, exactly as for a provider. A project-scoped group may hold that project’s providers and org-wide ones; an org-wide group may hold only org-wide providers, and rolter-seed --import stops with an error naming the group when it does not. As on a provider, the flag means “the project this file is imported into”, and a gateway that reads the file directly ignores it.
Default: "false"
[[provider_groups.members]]#
The name of a provider defined in [[providers]].
The upstream model id to send. When omitted, the model the client wrote after group-slug/ is forwarded as-is.
The relative traffic weight for the weighted and pipeline strategies. Ignored by other strategies.
Default: "1"
A client then asks for the model vllm-cluster/meta-llama/Llama-3.1-8B-Instruct.
Config tiers: readonly and default#
[[providers]], [[provider_groups]] and the model presets under [models] each come in two tiers, so one file can both pin a row and merely seed one (ADR-0022). The tiers are written [[providers.readonly]] and [[providers.default]], [[provider_groups.readonly]] and [[provider_groups.default]], and [[models.readonly]] and [[models.default]].
| Tier | Owned by | Behaviour |
|---|---|---|
readonly | the file | Served as written and immutable at runtime: the CRUD API and the dashboard refuse to edit or delete it. A readonly model joins [[routes]], so its name must not repeat a route’s. |
default | the database, once seeded | A control plane with a database inserts each entry into the default project of the default organization when it starts, unless the row exists or its name or slug is already in use, and never overwrites it afterwards. From then on it is edited like any row created on the dashboard, and one an admin deletes returns at the next restart. |
The bare [[providers]] and [[provider_groups]] arrays are the deprecated spelling of the readonly tier: every entry in them is readonly, and they cannot express a default. rolter config export writes the tiered spelling.
A default provider or group whose slug matches a readonly one, and a default model whose name matches a route, is a config error at load, since the readonly row is immutable and would shadow the seeded one.
Only a control plane that has a database and was started with this file (rolter control --config) acts on the default tier. It seeds into the default organization that rolter-seed creates, and without that organization it logs a warning and seeds nothing. A gateway that reads the file itself ignores the default tier.
[[routes]]#
Each [[routes]] block maps a public model name that clients request to one or more upstream targets and a load-balancing strategy.
The public model name clients send in their request body, e.g. gpt-4o. Must be unique across all routes.
The load-balancing strategy used to pick a target. One of:
| Value | Behaviour |
|---|---|
round_robin | Cycles through targets in order |
random | Picks a target uniformly at random |
power_of_two | Samples two targets and picks the one with fewer in-flight requests |
consistent_hash | Routes the same request consistently to the same target |
cache_aware | Steers toward the target most likely to have the request prefix cached |
weighted | Smooth weighted round-robin proportional to each target’s weight |
pipeline | Composable scorer pipeline: static weight, in-flight load and prefix-cache affinity |
cheapest | Lowest catalog price for the target’s upstream model; load breaks ties |
fastest | Lowest observed request latency; load breaks ties and carries cold routes |
precise_cache_aware | Exact resident-prefix score from vLLM KV-cache events; load fallback |
lmcache_aware | Cache availability and free-capacity score from LMCache controller occupancy |
adaptive | Weighted blend of latency, cost and load, governed by [adaptive_routing] |
lora_aware | Prefers a target that already holds the requested LoRA adapter |
predicted_latency | Ranks by what this request is modelled to cost, from queue depth and prompt size |
See Load balancing for when to pick each.
Default: "round_robin"
[[routes.targets]]#
The name of a provider defined in [[providers]].
The upstream model ID to send to the provider. When omitted, the requested model name is forwarded as-is.
The relative traffic weight for weighted and pipeline strategies. Ignored by other strategies.
Default: "1"
The failures this target is a fallback for: any of throttled, timeout, transport, server_error, refusal and refusal:<category>. A target with classes is never dialled first and takes no ordinary traffic; it is tried only after another target failed with one of them. Empty, the default, makes it an ordinary member of the pool. A route whose every target sets it is rejected (all_targets_restricted). See Fallback targets. Also valid on a variant’s targets.
Default: "[]"
[routes.params]#
An optional table of admin-set default inference parameters injected into the request body. Any key accepted by the upstream provider is valid here (temperature, max_tokens, stop, etc.). A parameter you do not set here is passed through from the client untouched.
[routes.param_policy]#
Controls whether clients may override the values set in [routes.params].
The baseline override policy. allow lets clients override any param that is not in deny. deny pins every param except those in allow. When an override is denied, the admin default silently wins.
Default: "allow"
[[routes.variants]]#
Variants replace the top-level targets/strategy pool when you want A/B, canary, or traffic-split routing under a single model name. A request randomly samples one variant by weight (the primary) and falls back to the remaining variants in declared order if the primary fails. Within a variant, the route’s strategy picks which target is tried first; the remaining targets follow in declared order.
A stable identifier for this variant. Appears in request logs under the variant column for attribution and A/B analysis.
Ordered list of upstream targets tried within this variant. Same shape as [[routes.targets]].
Variant-scoped default inference params layered over [routes.params]. The variant value wins on any key collision.
[routes.advanced]#
Catalog metadata and model-scoped policy for one route. The dashboard’s model sheet writes it; Per-model advanced settings explains what each part means and the ranges the control plane accepts.
The catalog family of the model, such as chat or embedding. A label the dashboard reads to choose which parameters and prices its model sheet offers, and GET /v1/models serves on the route’s entry. The gateway does not enforce it.
Capability labels recorded for the model, such as vision. A label for the dashboard and for GET /v1/models: the gateway forwards a request whether or not a flag is set.
Default: "[]"
The model’s context window in tokens, at least 1. Served on GET /v1/models as context_length; advertised, not enforced. Unset by default.
A note about the model for other operators. The dashboard and the routes API return it; it is never sent to API clients and never listed on GET /v1/models.
Headers sent to the upstream on every call this route makes, none by default. A name must be a valid header name and a value visible ASCII on one line. authorization, x-api-key, api-key, x-goog-api-key and the headers that frame a request (host, content-length, content-type) are refused, because credentials belong on the provider. The routes API answers 400 for one, rolter check fails a file that sets one, and the gateway leaves it out and logs a config problem at startup.
Names from headers whose value replaces a caller’s header of the same name. A caller’s header reaches the upstream only if [client] forwarded_headers lists it. An unlocked route header is a default that a forwarded caller header replaces.
Default: "[]"
Per-route selection of guardrail rules, written as [routes.advanced.guardrails]. See below.
[routes.advanced.limits]
Per-route request limits. An omitted key inherits the gateway-wide default, except rpm, tpm and concurrency where omitted means no cap. See Route limits. A request over an rpm, tpm or concurrency cap is refused with 429 and Retry-After. Without redis_url, rpm and tpm are not enforced and concurrency still is. A zero fails config validation, and a stored row that carries one is served without that cap and reported by GET /api/v1/config/problems.
Response headers timeout in seconds (1 and up); replaces [timeouts] request_secs for this route. A streamed body is not cut short.
Extra attempts after the first (0 to 10); replaces [retry] max_retries for this route. 0 turns retries off.
Refuses a request whose max_tokens, max_completion_tokens or max_output_tokens is above this limit.
Requests per minute cap across all gateway replicas via Redis (1 and up).
Tokens per minute cap across all gateway replicas via Redis (1 and up).
Maximum concurrent requests in flight per gateway replica (1 to 10000000). Works without Redis.
[routes.advanced.visibility]
Who may see and call the model. Keys from a config file carry no organization and are not narrowed.
Minimum role required to access or see the model.
When true, serves the route only to keys minted in its own project.
Default: "false"
Earlier releases also accepted base_url, insecure_tls, [routes.advanced.pricing] and a context_window limit, and stored them, but the gateway never read them (#2924). They are gone, and a file that still sets one loads with the key ignored and reported as unknown. A second endpoint for a model is a second provider with its own api_base; to trust a private certificate authority use [tls] ca_bundles or the provider’s ca_bundles; token rates belong in [[model_prices]]. The rpm, tpm and concurrency limits went the same way and came back with a reader (#2936); a limit on a caller rather than a model is set under [[rate_limits]].
[routes.advanced.guardrails]
Switches rules of the global [guardrails] set on or off for one route. Rules are named, so adding a rule globally still reaches every route that has not opted out of it by name. A name that matches no configured rule fails config validation instead of being ignored, because a typo in disable would otherwise read as “this rule is off here” while the rule kept running.
[routes.cache]#
Opts a route into the exact-match response cache. A route caches only when this table is present with enabled = true and the global [cache] switch is on; a virtual key’s cache setting can override the route. See Response cache.
Whether this route caches. Writing the table opts the route in; set false to opt it out again.
Default: "true"
Entry lifetime in seconds for this route. When omitted, default_ttl_secs from [cache] applies.
Isolate cached entries by virtual key so responses are not shared across keys. When false, every caller of the route shares the cache.
Default: "false"
Adds a semantic lookup after the exact-match one, written as [routes.cache.semantic]. See below.
[routes.cache.semantic]
A semantic hit replays a reply only to a request that matches it exactly on everything but the user and assistant wording. Multimodal and tool-call conversations skip the semantic lookup.
The name of the configured provider used for /v1/embeddings. It reuses that provider’s authentication, proxy and CA policy.
The embeddings model sent upstream.
How many of the most recent semantic entries are compared per request, which bounds the cost of a lookup.
Default: "256"
[models]#
Model presets that behave like routes but come in the two tiers described under “Config tiers” above. Each entry takes the same keys as a [[routes]] entry, including its [[models.readonly.targets]] tables.
Routes served directly from the file and rejected by the CRUD API, written as [[models.readonly]]. A readonly model is added to [[routes]], and its model must not repeat the name of a route.
Default: "[]"
Routes seeded once into the default project of a database-backed control plane, written as [[models.default]], and then owned and edited in the dashboard. A model that conflicts with a readonly model or a route is a config error.
Default: "[]"
[[virtual_keys]]#
Virtual keys are the bearer tokens clients present to the gateway. Remove the entire [[virtual_keys]] section to run in open (unauthenticated) mode, which is suitable only for local development.
The bearer token value. Clients send this in the Authorization: Bearer <key> header.
A human-readable label for this key, used in logs and the dashboard.
Model allow-list. An empty list grants access to all models. When non-empty, requests for models not on the list are rejected.
Default: "[]"
Upstream provider allow-list. An empty list allows selecting any provider on an allowed route. When non-empty, requests authenticated with this key can only route to the named providers.
Default: "[]"
Set to true to revoke the key without deleting it from the config file. Useful for auditing.
Default: "false"
An RFC3339 expiry timestamp. The key stops authenticating at or after this instant, e.g. "2027-01-01T00:00:00Z".
Per-key response-cache override, independent of a route’s own opt-in. Omit to inherit the route decision, false to force responses to bypass the cache, or true to cache responses even on a route that did not opt in. The global [cache] switch is still required.
[[db_virtual_keys]]#
Not a rolter.toml table. Virtual keys created on the dashboard or through the API live in the database, and the control plane puts them in the snapshot as db_virtual_keys. Each entry carries the key’s peppered digest, never the plaintext, with its id, the scope its usage is attributed to (organization, team, project, creator, business unit and customer) and the same models, providers, disabled, expires_at and cache settings as a [[virtual_keys]] entry. While a rotation’s overlap is running (see Virtual keys), an entry also carries previous: the digest of the replaced secret and the valid_until instant up to which it still authenticates as the same key. A gateway matches a presented key against those digests alongside the keys of [[virtual_keys]], which stay the file’s own list. See Virtual keys.
[[mcp_servers]]#
Not a rolter.toml table. MCP servers belong to an organization and are registered on the dashboard or through the management API; the control plane unseals their credentials and ships them in the snapshot. Each entry carries id, org_id, slug, url, transport, required_scopes, auth_kind, auth_header_name, credential, connect_timeout_ms, request_timeout_ms, max_retries and tools. With a database, the snapshot carries the control plane’s own rows and replaces whatever a bootstrap file holds for this table. See MCP credentials and transport settings for what each field means.
[[mcp_tool_groups]]#
Not a rolter.toml table. Each entry is an enabled MCP tool group, which the control plane adds to the snapshot and the MCP proxy enforces: a server that at least one group names serves only the tools those groups list. Entries carry id, org_id, slug and tools, a list of server_id and tool pairs. Groups are managed on the dashboard or through the management API; see MCP.
[[mcp_settings]]#
Not a rolter.toml table. Each entry is one organization’s saved MCP settings, which the MCP proxy applies per request: org_id, connect_timeout_ms, request_timeout_ms, max_retries, fail_open and allow_unlisted_tools. An organization that never saved any has no entry and keeps the deployment defaults. They are managed on the dashboard or through the management API; see MCP.
[[mcp_oauth_sessions]]#
Not a rolter.toml table. Each entry is a live OAuth session, one user’s access token for one MCP server, that the control plane adds to the snapshot after the user completes the consent flow and drops when the session expires. The tokens cross only the protected /internal/snapshot channel and are never shown on the dashboard or through the public API. Entries carry id, server_id, user_id, scopes, expires_at and access_token. There is nothing to write by hand; see MCP OAuth for the consent flow.
[[model_prices]]#
Define per-model token pricing so Rolter can compute a cost_usd value on every request log. Models without a price entry log a cost of 0. Rates are in USD per million tokens, matching the convention used by major providers. Every rate must be zero or more: a negative rate is a config error, because it would turn a request into a credit against its budget.
The public model name this price applies to. Must match a route’s model field.
Prices and budget limits are written as numbers but computed in exact decimal
arithmetic, so a rate like 0.1 is a tenth rather than its nearest binary
approximation. See Budgets & Rate Limits
for what that does and does not cover.
USD per million input (prompt) tokens.
USD per million output (completion) tokens.
USD per million cache-hit input tokens. When omitted, falls back to input_per_mtok.
USD per million input tokens written to the provider’s prompt cache, which Anthropic bills above the input rate. When omitted, falls back to input_per_mtok. See Cache writes.
USD per million input tokens written to the provider’s 1 hour prompt cache, for a provider that reports the split and bills it apart from the 5 minute cache (Anthropic). When omitted, falls back to cache_write_per_mtok, and from there to input_per_mtok. See Cache writes.
Price of one generated image on /v1/images/generations, in currency, for a model the provider bills per image. The count is the length of the response’s data array, or the request’s n when the response has none. The cost is added to the token cost of the same request. When omitted, an image request that reports no tokens is logged unpriced. See Images, audio and speech.
Price of one minute of input audio on /v1/audio/transcriptions and /v1/audio/translations, in currency. The minutes are the duration the response states: duration in a verbose_json answer, or a usage object of type duration. A response that states none is logged with usage_unknown = 1 and no cost. When omitted, such a request that reports no tokens is logged unpriced.
Price per million characters of text sent to /v1/audio/speech, in currency. The characters are counted from the request’s input, because the audio that comes back states no length. When omitted, a speech request is logged unpriced.
Price per million audio input tokens, for a realtime or audio model that bills the audio it is sent apart from the text. The share of the prompt that is audio is the one the response’s usage names (see Audio and text tokens). Defaults to input_per_mtok.
Price per million audio output tokens. Defaults to output_per_mtok.
Price per million cached audio input tokens, for a provider that says how many of its cache hits were audio (a realtime session does). Defaults to cached_input_per_mtok, and from there to the audio input rate.
The currency the rates above are written in. It defaults to USD, not to the base of [currency], so on a deployment whose base is another currency set it on every price or add a USD rate. A currency with no rate in [currency] is a config error.
Default: "USD"
[currency]#
The settlement currency and a static rate table. Budgets and accumulated spend are denominated in base; a model priced in any other currency is converted through rates before it reaches a budget counter, so pricing in euros while budgeting in dollars is exact. Rates are written by hand, which keeps this working air-gapped.
The dashboard’s currency chooser offers base followed by every code in rates, so adding a row here is the whole of making a currency selectable.
The currency budgets and spend are denominated in. Codes are open strings (ISO 4217 today), compared without regard to case or surrounding whitespace.
Default: "USD"
How many units of base one unit of the keyed currency is worth: with a base of USD, EUR = 1.09 means one euro costs 1.09 dollars. The base itself is always 1.0 and needs no row. Every rate must be positive, and the table is empty by default. A [[model_prices]] entry in a currency with no rate is a config error, because charging face value bills the wrong number and charging zero bills nothing.
The database has no currency table, so a control plane reads this one from the file it was started with (rolter control --config rolter.toml) and checks the currency of every price written through its API against it.
[[budgets]]#
Spend caps block requests with HTTP 402 once a scope’s cumulative cost reaches its limit for the current period. Budgets are enforced using Redis-tracked costs; without a Redis connection, budgets fail open. The most restrictive matching budget across the scope chain wins.
The id field matches the UUID of an entity defined in the database (org, team, project, or virtual key). Config-file virtual keys carry no scope IDs, so [[budgets]] entries in rolter.toml are most useful when combined with the full control-plane stack.
The scope level this budget applies to. One of org, team, project, key, business_unit, customer or user. A user budget counts every key that person minted, whatever project it lives in.
The UUID of the scoped entity. In a file that rolter-seed --import reads, the entity’s portable name instead: an org slug, org/team, org/team/project, or, for a user budget, the account’s email (Config export).
The spend cap in USD for the current window. Must be a positive number.
The window the budget resets on. One of monthly, daily, or total. total is a lifetime cap that never resets.
Default: "monthly"
This budget’s own answer to traffic the gateway cannot price: ignore, warn or block. Omit it to inherit the deployment-wide unpriced_policy. The most restrictive answer across the scope chain and the deployment setting wins, so a budget can tighten what the deployment asked for but never loosen it. See Per-budget override.
Narrows a user budget to the keys the person minted in one org, { org = "org-uuid" }, or one project, { project = "project-uuid" }; in an import file the same places by slug path, { org = "acme" } or { project = "acme/team/project" }. Omit it to count all of their keys. Has no meaning on any other scope, and an import refuses it there.
unpriced_policy#
A top-level key deciding what happens to traffic the gateway cannot price. A model with no [[model_prices]] entry accrues zero spend, so it can never exhaust a budget however much is served — without this setting, budgets fail open on exactly the models you added most recently.
Unlike budgets, this needs no Redis: being unpriced is a property of the config, not of a counter, so it is enforced even while budget counters are failing open.
One of ignore, warn, or block. ignore serves the request and accrues nothing. warn serves it but logs the model once per hour. block refuses at admission with HTTP 402 and code model_unpriced, before any upstream call. Defaults to ignore so that upgrading cannot start refusing traffic on a deployment that has not priced its models yet.
Default: "ignore"
The built-in fake-llm model is served before any price lookup, so block never makes it unreachable.
[[rate_limits]]#
Throughput caps reject excess requests with HTTP 429 and a Retry-After header. Limits use a Redis sliding-window counter; without Redis they fail open. At least one of rpm or tpm must be set — a limit with neither never blocks anything.
The scope level this limit applies to. One of org, team, project, key, business_unit, customer or user. A user limit counts every key that person minted.
The UUID of the scoped entity. In a file that rolter-seed --import reads, the entity’s portable name instead: an org slug, org/team, org/team/project, or, for a user limit, the account’s email (Config export).
Requests per minute cap. Omit to leave request rate uncapped.
Tokens per minute cap (prompt + completion combined). Omit to leave token rate uncapped.
Narrows a user limit to the keys the person minted in one org, { org = "org-uuid" }, or one project, { project = "project-uuid" }; in an import file the same places by slug path, { org = "acme" } or { project = "acme/team/project" }. Omit it to count all of their keys. Has no meaning on any other scope, and an import refuses it there.
[retry]#
On a transient upstream failure (HTTP 408, 429, 5xx, or a connection error), Rolter re-picks a different target and retries. Retries only happen before any response bytes have been streamed to the client, so no partial response is ever duplicated.
In a database-backed deployment the store owns this table: the snapshot carries what is saved under Settings → Performance Tuning, see Performance tuning. It takes effect from the file for a gateway that reads rolter.toml itself.
The base delay in milliseconds for the first retry. Actual delay uses exponential backoff with full jitter. A Retry-After header from the upstream overrides the computed delay.
Default: "100"
[cooldown]#
After a target returns a transient failure, Rolter parks it for base_secs and the balancer skips it, shifting load to healthy siblings. When every sibling is parked, the gateway fails open rather than rejecting the request.
How long a failing target is parked, in seconds. Set to 0 to disable cooldowns entirely.
Default: "5"
The ceiling for a cooldown duration derived from a 429 Retry-After header.
Default: "300"
[cache]#
Global exact-match response-cache settings. Backed by Redis; off by default. Individual routes must also opt in or carry a virtual-key cache override.
Redis key prefix for cache entries to isolate and flush entries easily.
Default: "rolter:cache"
Largest response body in bytes that may be cached (defaults to 1 MiB).
Default: "1048576"
[responses]#
Routing records for the OpenAI Responses lifecycle endpoints. When a native OpenAI provider creates a response, the gateway remembers which provider, target and credential served it, so GET and DELETE /v1/responses/{id}, cancellation and input-item retrieval go back to the same place. Records are isolated by virtual key and removed after a successful delete.
The registry lives in the gateway process. Records do not survive a restart, and a deployment with several gateway replicas has to keep a response’s lifecycle requests on the replica that created it. An unknown, expired, deleted or cross-key record returns 404 response_not_found.
How long a record lives after the response is created, in seconds (24 hours). 0 disables registration.
Default: "86400"
The most live records one gateway process keeps. 0 disables registration too. Expired and overflowing records are cleaned up lazily, on lookup and insert.
Default: "100000"
[timeouts]#
Timeouts bound how long the gateway waits for an upstream before treating the request as a transient failure and feeding it into the retry and cooldown machinery. request_secs applies to time-to-response-headers only, so legitimately long SSE streams are never cut short.
In a database-backed deployment the store owns this table: the snapshot carries what is saved under Settings → Performance Tuning, see Performance tuning. It takes effect from the file for a gateway that reads rolter.toml itself.
[compatibility]#
Cross-dialect compatibility settings, applied when Rolter translates a request between the OpenAI and Anthropic wire formats. Both values have no equivalent on the other side of the translation, so they have to come from somewhere.
The anthropic-version header sent to Anthropic upstreams on a translated request. It is forwarded verbatim, so it must be a dated release string; the dashboard and the API refuse anything else.
Default: "2023-06-01"
The max_tokens Rolter fills in when it translates an OpenAI request that set neither max_tokens nor max_completion_tokens, since the Anthropic Messages API requires one. A request that sets its own cap always wins. The dashboard accepts 1 to 1000000.
Default: "1024"
Database-backed deployments. Behind a control plane with a database, the compatibility policy is owned by the store: the snapshot carries what is saved under Settings → Compatibility, and rolter-seed --import does not read this table from the file. Edit it there, see Compatibility. This table takes effect for a gateway that reads rolter.toml itself.
[client]#
Header handling on the upstream leg of a request, and the base URL the dashboard advertises to clients.
The address clients point their SDK at, such as https://gateway.example.com, without /v1. The dashboard builds its connection snippets from it. Advisory only: the gateway never reads it, and setting it does not change what the gateway listens on. When unset, the dashboard shows no connection snippet.
Inbound client headers passed through to the upstream provider, written in lowercase. Anything not listed is dropped. Trace-context headers (traceparent, tracestate, b3) and every anthropic-* header to an Anthropic-dialect upstream propagate whether or not they are listed. Never list authorization, x-api-key, api-key, host, content-length, content-type, transfer-encoding, connection or cookie: the dashboard and the API refuse them, because forwarding one would hand a caller the gateway’s own credentials or corrupt the upstream request, and the gateway does not check a file for them.
Default: "[]"
Static header names and values added to every upstream request, none by default and written with lowercase names: a partner id, a routing tag, a billing dimension your provider expects. A trace header of the same name still wins. The reserved names listed under forwarded_headers apply here too.
The header carrying the end-to-end request id. The gateway reads the caller’s id from it, mints a UUID when it is missing or blank, echoes the id in it on the response and stores it as the request-log row’s request_id. With another name set, x-request-id is an ordinary header the gateway neither reads nor sets. The name cannot be one that carries a credential, session state or message framing (authorization, x-api-key, api-key, x-goog-api-key, proxy-authorization, cookie, set-cookie, host, content-length, content-type, transfer-encoding, connection and the other hop-by-hop headers): a file naming one fails validation. See Client settings.
Default: "x-request-id"
Database-backed deployments. A control plane with a database owns this table: the snapshot carries what is saved under Settings → Client Settings, and rolter-seed --import does not read it from the file. See Client settings. The table takes effect for a gateway that reads rolter.toml itself. A control plane started with --config as well as a database serves the file’s table until something other than the defaults is saved on the screen, and the saved one after that.
[model_defaults]#
Inference parameters Rolter fills in when a client leaves them out. Every field only ever fills a gap: a value the client sent, an explicit null included, is never overwritten, so turning this on cannot change the meaning of a request that was already explicit.
Master switch. While false, request bodies reach the provider untouched, whatever the other fields hold.
Default: "false"
The model used when a request omits one or sends an empty one. It can name a routed model or a group-slug/model address. Leave it unset to keep rejecting such requests.
The sampling temperature applied when the client sent none. The dashboard accepts 0 to 2.
The nucleus-sampling value applied when the client sent none. The dashboard accepts 0 to 1.
The completion-length cap applied when the client sent none, written under the key the upstream dialect reads (max_tokens, or max_output_tokens for the Responses API). A request that already set max_completion_tokens on OpenAI chat has stated its cap, so none is added. The dashboard accepts 1 to 1000000.
These apply to /v1/chat/completions, /v1/messages and /v1/responses. Embeddings, audio and image endpoints take no sampling parameters and are left alone, and Gemini, which nests them under generationConfig, does not take defaults from here.
Database-backed deployments. A control plane with a database owns this table: the snapshot carries what is saved under Settings → Model Settings, and rolter-seed --import does not read it from the file. See Model settings, whose API names the fields default_temperature, default_top_p and default_max_tokens. The table takes effect for a gateway that reads rolter.toml itself. A control plane started with --config as well as a database serves the file’s table until something other than the defaults is saved on the screen, and the saved one after that.
[adaptive_routing]#
The deployment-wide policy for routes on the adaptive strategy, which scores each target from live latency, catalog cost and in-flight load instead of using static weights. The whole feature is off by default: with enabled off, or before a route has gathered enough evidence to rank its targets, an adaptive route behaves exactly like the pipeline strategy, so moving a route onto it shifts no traffic until you say so.
The master kill switch. While false, every adaptive route is served by the pipeline fallback stack.
Default: "false"
The share of picks made uniformly at random, so a target the blend would otherwise starve keeps producing fresh latency samples. Values above 0.5 are clamped to 0.5.
Default: "0.05"
The number of requests a route must have served before the blend engages, counted across all its targets. While latency_weight is above 0 the blend also needs latency samples for at least two targets. Until then the route is served by the fallback stack.
Default: "50"
Only the ratio between the three weights matters, and a negative weight is treated as 0. If all three are 0 the blend has nothing to rank by, and Rolter treats the policy as not configured. See Adaptive routing for how to tune the blend and read the telemetry.
Database-backed deployments. A control plane with a database owns this table: the snapshot carries what is saved under Adaptive Routing → Settings, and rolter-seed --import does not read it from the file. The table takes effect for a gateway that reads rolter.toml itself.
[queue]#
Rolter keeps a bounded dispatch queue and a worker set for each provider. A provider that stalls before sending response headers therefore backs up only its own queue; requests selected for other providers continue independently. Once headers arrive, response bodies (including SSE) stream directly to the caller. The settings are applied to subsequent requests immediately after a config snapshot reload.
In a database-backed deployment the store owns this table: the snapshot carries what is saved under Settings → Performance Tuning, see Performance tuning. It takes effect from the file for a gateway that reads rolter.toml itself.
Maximum waiting requests per provider. Active workers do not count against this limit.
Default: "256"
Concurrent upstream requests waiting for response headers for each provider:
a provider with workers = 8 has up to eight such calls outstanding, and the
next one waits for a worker. A worker is released when the headers arrive, so
a streamed response keeps generating upstream after its worker has moved on
and does not count against this limit.
Default: "8"
drop sheds a full queue with a 503; error returns a 429 queue_full response. block waits for capacity up to block_timeout_ms and then returns 429 queue_timeout.
Default: "error"
Maximum capacity wait when backpressure = "block". It must be greater than zero for that policy.
Default: "1000"
A [queue] change takes effect for requests that start after the reload. Requests already buffered in the replaced queue are not dropped: its workers finish them, so for that short window a provider can see the old and the new workers at the same time.
Queue shedding is exposed through rolter_provider_queue_rejections_total; timed-out blocking admissions use rolter_provider_queue_timeouts_total. Depth, in-flight calls and the time requests wait for a worker are per-provider series described under Provider queue.
[health]#
When enabled, a background task periodically probes each provider with a lightweight GET request. A provider that times out, fails to connect, or returns 5xx is marked unhealthy and its targets are skipped. Any non-5xx response (including 401 or 404) counts as healthy. Disabled by default.
The request path appended to each provider’s api_base when probing. Left at the default /, Rolter probes each provider kind’s free non-inference liveness endpoint instead (/v1/models for openai/openai_compatible, and /v1/models with an anthropic-version header for anthropic) — no token cost. Set an explicit path (e.g. /health) to override this for all providers.
Default: "/"
Maximum probes in flight at once during a sweep. Each provider also starts at a stable offset of up to a quarter of the interval (at most two seconds), so a sweep never hits every upstream at the same moment. A value of 0 is treated as 1.
Default: "2"
Consecutive probe failures before a provider is marked unhealthy and its targets are skipped. One failed probe never removes a provider on its own. A value of 0 is treated as 1.
Default: "3"
Consecutive probe successes before an unhealthy provider returns to rotation, so a flapping upstream does not bounce in and out. A value of 0 is treated as 1.
Default: "2"
Seconds between polls of provider status pages. Only providers with a status_page_url are polled, and the poller runs even with enabled = false. A status page is a secondary signal: it records a status_page event and never changes routing on its own. See Health checks.
Default: "60"
[breaker]#
The circuit breaker complements the short-lived cooldown with a longer-lived state machine. After failure_threshold consecutive transient failures a target trips open and is skipped for open_secs. The first request after that window probes the target in a half-open state, and other requests skip the target until that probe answers; a success closes the breaker, a failure re-opens it. Disabled by default.
Consecutive transient failures required to trip a target open. Setting this to 0 also disables the breaker.
Default: "5"
How long a tripped target stays open before a half-open probe, in seconds.
Default: "30"
[metrics_scrape]#
When enabled, a background task periodically pulls each provider’s Prometheus /metrics endpoint and extracts scheduler queue depth (vLLM num_requests_waiting, with SGLang/TGI fallbacks). That depth feeds into the balancer’s in-flight load view so load-aware strategies (power_of_two, pipeline) steer away from backed-up engines. Disabled by default.
[realtime]#
Resource limits and metering cadence for /v1/realtime WebSocket sessions. The three limits are per gateway process and guard the gateway’s own resources; tenant policy comes from [[budgets]] and [[rate_limits]], which a session meets at admission and while it runs. Set a limit to 0 to disable it.
Closes a session when neither side has sent a frame for this many seconds.
Default: "300"
How often a live session’s finished responses are written to the request log, charged to budgets and tpm windows, and followed by a budget check that closes the session once a budget is spent. It bounds how far a session can overspend. 0 flushes after every response instead of on a timer, and re-checks the budgets every second while the session is quiet; metering cannot be switched off.
Default: "15"
[usage_recording]#
After a response completes, Rolter adds the request’s cost to its applicable budget counters and its tokens to its applicable rate-limit windows. Both write to Redis, so both run off the response path — through a bounded queue rather than one detached task per request, so a slow counter store sheds records instead of accumulating work without limit.
Dropping is the deliberate failure mode: a lost record under-counts spend for a single request, while an unbounded queue grows worst exactly when the gateway is already under stress. Overflow is never silent — watch rolter_usage_records_dropped_total, and rolter_usage_records_queued for pressure before it turns into drops.
This section is inert without a Redis URL: with no counter store there is nothing to record.
Records waiting to be written. New records are dropped (and counted) once the queue is full; the response path never blocks on it.
Default: "10000"
Concurrent writers draining the queue. Each record is one Redis round trip, so a single consumer would serialise them. Raise this if rolter_usage_records_queued sits high against a healthy Redis.
Default: "4"
[security]#
Ingress rules applied to every request on the /v1/* surface. A gateway managed by the control plane takes these from the dashboard’s Security screen; a file-configured gateway reads them from this section. Both keys are optional and the section adds no rules when omitted.
Lowercase header name to the exact value every request must carry. A request that is missing one,
or carries a different value, is refused with 403 before authentication. This is an ingress
filter (a mesh identity header, a WAF marker), not an authorization decision. The /healthz and /readyz
probes are exempt.
Exact /v1/... paths served without authentication. Matching is exact: no prefixes and no
wildcards.
[logging]#
Request and cost logs are written asynchronously in batches, off the hot request path. Logging is disabled entirely when clickhouse_url is unset. When the internal queue fills, records are dropped and counted rather than blocking request processing.
The base URL of the ClickHouse HTTP interface, e.g. http://localhost:8123. Logging is disabled when this is not set.
The maximum number of log records buffered in-flight. Records are dropped when this queue is full.
Default: "10000"
The fraction of requests written to the request log, from 0 to 1, applied before a row is written. 1 keeps every record and 0 writes no request-log rows. The decision is made per request from a hash of the request id, and each kept row records its weight so the analytics endpoints scale counts back up. Budgets, rate limits and the Prometheus metrics still count every request. See Log settings.
Default: "1.0"
Opt-in capture of raw prompt and completion bodies next to the log metadata, written as [logging.payload_capture]. Off by default, because payloads hold whatever the caller sent. See below.
Whether the control plane records the dashboard’s own usage events: screen views, load times, abandoned forms and error states. They carry structure only (a screen, an action, a duration and ids), never a form value or a prompt, which is why this is on by default where payload_capture is not. false drops them at the ingest endpoint. The stream is inert anyway without clickhouse_url.
Default: "true"
[logging.payload_capture]#
Captured bodies go to ClickHouse’s short-retention request_payloads table. Redaction happens before the write, so a redacted value never reaches storage, and turning a field on later does not clean what is already stored.
The most UTF-8 bytes kept of each request or response body; anything longer is truncated.
Default: "32768"
JSON object keys whose values are replaced before storage. Matching ignores case and reaches nested objects, tool arguments included.
Default: "[\"api_key\", \"authorization\", \"password\", \"secret\", \"token\"]"
Built-in detectors that scan the text of every captured string and replace a match with a typed marker such as [redacted:aws_key]: private_key, anthropic_key, openai_key, google_api_key, github_token, slack_token, stripe_key, aws_key, jwt, bearer_token and connection_string (only the password of a scheme://user:password@host string goes). An empty list turns the built-ins off. An unknown name is rejected when saved through the dashboard.
Default: "all built-in detectors"
Your own value patterns, each a table with a name (lowercase letters, digits or _, up to 32 characters, which becomes the marker [redacted:<name>]) and a pattern (a regular expression of at most 256 bytes). Written as [[logging.payload_capture.redact_patterns]]. Matching uses a linear-time engine with no look-around or backreferences, at most 16 patterns are allowed, and a pattern that does not compile is rejected when saved.
Default: "[]"
Public model names to capture. Empty captures every route; a non-empty list limits capture to the routes named.
Default: "[]"
Database virtual-key ids to capture. Empty captures every key that models allows; a non-empty list makes capture key-specific.
Default: "[]"
Database-backed deployments. Behind a control plane with a database, sample_rate, payload_capture and ui_events are owned by the store: the snapshot carries what is saved under Observability → Logs Settings, see Log settings. rolter-seed --import copies [logging.payload_capture] and ui_events into the database when the file writes them, and does not read sample_rate. All three take effect from the file only for a gateway that reads rolter.toml itself. clickhouse_url follows a different rule: a gateway that names none inherits the control plane’s own ClickHouse URL from the snapshot, and one that names its own keeps it.
[feature_flags]#
Deployment-wide switches for the subsystems that are safe to turn on and off at runtime. They belong to the control plane: an admin edits them under Settings → Feature Flags, the control plane stores them and applies them to the snapshot every gateway polls, and a gateway picks the change up without a restart. A gateway that reads rolter.toml itself does not apply this table, so writing it there does nothing. See Feature flags for what each flag gates and when one is reported unavailable.
All six default to true.
Serving repeated completions from the shared response cache. In the snapshot it sets [cache] enabled.
Default: "true"
Scoring targets by KV-cache affinity. While false, a route on cache_aware, precise_cache_aware or lmcache_aware is served by power_of_two instead.
Default: "true"
Tripping a failing target out of the pool and probing it before restoring traffic. In the snapshot it sets [breaker] enabled.
Default: "true"
Polling targets in the background so an unhealthy one is parked before a live request finds it. In the snapshot it sets [health] enabled.
Default: "true"
Estimating prompt complexity and routing cheap requests to smaller models. While false, a route’s complexity policy is dropped.
Default: "true"
Running the configured guardrail rules. In the snapshot it sets [guardrails] enabled, so false pauses every built-in, custom and config-file rule.
Default: "true"
[guardrails]#
Built-in regex guardrails and PII redaction, evaluated inside the gateway with no external service and no network hop. Disabled by default, and a disabled or empty block adds no cost to a request. It complements the custom guardrail webhook and the external PII sanitizer, and never replaces them. Guardrails describes the dashboard screens and how rules defined here interact with the ones added there.
The master switch. While false no rule runs. On a control-plane-managed gateway the guardrails feature flag is the kill switch on top of it.
Default: "false"
The most request text scanned per request, in bytes (256 KiB). Content past the cap is forwarded unscanned, which keeps the work bounded.
Default: "262144"
What a request with "stream": true does on a route that has post_call rules. Output masking needs the whole completion, because a match can straddle any number of token boundaries, so it cannot run on a stream. reject refuses the request with 400 guardrail_streaming_unsupported, which fails closed. passthrough serves the stream with the output rules not applied; pre_call rules still run. The setting only matters when a post_call rule exists. The dashboard shows the effective value beside each response rule but cannot change it.
Default: "reject"
The rules, written as [[guardrails.rules]] entries and run in the order given. See below.
[[guardrails.rules]]#
Provide exactly one of builtin or pattern. Patterns use a linear-time, RE2-style engine with no catastrophic backtracking, and are compiled under a bounded program size while the config is validated, so an invalid or unbounded pattern fails at startup and never on the request path. The gateway never logs the text a rule matched; metrics only count blocks and redactions.
A stable, unique name for the rule. It appears in telemetry, never alongside the matched text, and is how a route’s enable and disable lists refer to it.
A starter detector: email, phone, api_token or payment_card. Mutually exclusive with pattern. The card detector matches 13 to 19 digit candidates and does not check the Luhn digit.
A custom regular expression. Mutually exclusive with builtin.
When the rule runs. pre_call scans the request before it goes upstream. post_call masks a non-streamed response body before the client sees it, and a streamed one follows streaming_post_call.
Default: "pre_call"
What a match does. annotate counts it and forwards the content unchanged. block rejects the request with an OpenAI-compatible 403 guardrail_blocked error naming the rule. redact replaces each match with replacement.
Default: "annotate"
The token that replaces a match under redact, up to 64 bytes. When omitted, empty or only whitespace, the detector’s default token is used ([REDACTED:EMAIL], [REDACTED:PHONE], [REDACTED:API_TOKEN] or [REDACTED:CARD]), or [REDACTED] for a custom pattern, so a match is never deleted outright.
Also scan operator-authored system and developer messages. They are trusted and skipped unless you opt in.
Default: "false"
Request rules scan OpenAI /v1/chat/completions and /v1/responses (messages and input), /v1/completions (prompt) and Anthropic /v1/messages (system and messages), including string, string-array and typed text parts. Output rules mask choices[].message.content and choices[].text, the top-level content parts of an Anthropic reply, and output[].content[].text and output_text of a Responses reply, plus the arguments of tool calls. A block on a response rule runs after the provider has generated and billed the answer, so the request is still billed.
To switch a rule off for one route, name it in that route’s [routes.advanced.guardrails] disable list, described under [[routes]] above. The old default_on key was removed: it documented a per-request opt-in that was never implemented, and a config that still sets it loads with the key ignored.
[egress]#
Where the gateway may send upstream traffic. A provider’s api_base and egress_proxy are written by an operator, so without a destination policy a crafted or compromised provider row turns the gateway into a way to reach whatever its network position can reach, and the prize on a cloud host is instance metadata at 169.254.169.254. Egress policy lists every URL the policy covers.
Self-hosted and air-gapped upstreams live on loopback and private networks, so only the link-local range is denied by default.
Deny link-local destinations (169.254.0.0/16 and fe80::/10), which is where cloud instance-metadata services live. Leave it on unless you have a concrete reason.
Default: "true"
Deny loopback destinations (127.0.0.0/8 and ::1) and the unspecified addresses 0.0.0.0 and ::, which reach the local host when connected to. Off by default so sidecar and single-host deployments can serve models on localhost.
Default: "false"
Deny private and unique-local destinations (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and fc00::/7). Off by default because on-premises clusters are the common case.
Default: "false"
Hosts exempt from every check above: an IP literal or a hostname, matched against the host the URL reaches whatever its spelling, so 169.254.169.254 also covers http://0xa9fea9fe/. The escape hatch for a deployment that must reach one otherwise-denied address.
Default: "[]"
IP literals are classified when the config is validated, so a bad api_base is rejected at startup or when the control plane writes it. A hostname cannot be classified without DNS, so it is checked at connect time against the address the lookup actually returned, which also makes the policy safe against DNS rebinding. A denied address fails the request and is not retried elsewhere.
The database has no egress policy, so a control plane reads this table from the file it was started with (rolter control --config rolter.toml) and applies it to every URL written through its API.
[guardrail_webhook]#
A vendor-neutral hook to a self-hosted semantic guardrail service, such as Guardrails AI or LLM Guard. Before it proxies a request, the gateway POSTs a JSON envelope to your endpoint, and the service answers with a decision. Disabled by default, and it complements the built-in [guardrails] rather than replacing them. Guardrails describes registering providers on the dashboard instead. Since semantic guard policies this hook is one provider among those of [semantic_guards] and the dashboard: it runs beside them rather than displacing one, and a policy can list it by name.
The master switch. A disabled block is never a problem, whatever else is unset.
Default: "false"
A label recorded with this hook’s outcomes in LLM Logs (provider_blocked:<name>). Optional; empty records the bare provider_blocked / provider_transformed. A provider registered on the dashboard supplies its own name.
Default: ""
The http:// or https:// endpoint the envelope is POSTed to. Required when enabled is true. The URL is checked against the egress policy when the config loads and again before every call, and Rolter does not follow a redirect from it.
When the service is consulted. pre_call inspects the request before it goes upstream; post_call inspects the response before it is delivered. A streamed response is refused or buffered for a post_call provider, per the policy’s streaming.
Default: "pre_call"
The per-call timeout in milliseconds. It must be greater than zero.
Default: "2000"
Extra attempts after a transient failure: a connect error, a timeout, an unreadable body, or a 429 / 503. Another error status and an answer that is not a decision are not retried.
Default: "0"
What happens when the service gives no verdict: it is unreachable, times out, is overloaded, errors, or answers something that is not a decision. fail_open forwards the content unchanged, favouring availability. fail_closed rejects the request with an OpenAI-compatible error (or withholds the response), favouring enforcement.
Default: "fail_open"
The most content forwarded to the service, in bytes. Oversized content is sent as a truncated preview with truncated: true. It must be greater than zero.
Default: "65536"
An optional credential, resolved from the environment at call time and never written inline. { bearer = { token_env = "GUARD_TOKEN" } } sends Authorization: Bearer <value of GUARD_TOKEN>, and { shared_secret = { secret_env = "GUARD_SECRET" } } sends the value in an X-Rolter-Guardrail-Secret header.
The contract. The envelope is { direction, stage, model, route, trace_id, tenant: { org, team, project, key }, truncated, content, policy?, streamed? }, and only those fields are sent; the gateway never logs the prompt. policy names the policy that selected the provider, and streamed: true marks a buffered stream, sent as content: { "stream": true, "text": … }. The service answers { "action": "allow" | "block" | "transform" | "annotate", "content"?, "reason"?, "annotations"?, "categories"? }, where categories maps category names to scores from 0 to 1; a body with categories and no action is an allow carrying scores. An answer that is not a decision follows failure_mode, like a transport failure. The trace id travels in the X-Rolter-Trace-Id header.
[semantic_guards]#
Several guardrail providers side by side, and the scoped policies that pick which of them judge a request (#459). Empty by default. Semantic guard policies explains the precedence, thresholds, failure handling and streaming in full. The dashboard’s Guardrails → Providers and Guardrails → Policies add to what the file sets; the file wins a name, and an enabled file policy wins its scope.
The providers, each a [[semantic_guards.providers]] table below.
The policies, each a [[semantic_guards.policies]] table below.
[[semantic_guards.providers]]#
The same keys as [guardrail_webhook], one table per provider. Only enabled providers run.
Required and unique, including against [guardrail_webhook]’s name. Policies list providers by it, and it labels the provider’s metrics and log events.
The http:// or https:// endpoint, checked against the egress policy at load and before every call.
fail_open or fail_closed, applied when this provider gives no verdict.
Default: "fail_open"
An optional credential, in the same two forms as [guardrail_webhook].
[[semantic_guards.policies]]#
Required and unique. It labels the policy’s metrics and log events.
A paused policy matches nothing, but its providers stay confined to policies.
Default: "true"
deployment, org, project, route or key. The most specific enabled match wins, in the order key, route, project, org, deployment, and replaces the broader ones.
Default: "deployment"
The org, project or key id, or the route’s public model name. Required for every scope but deployment, which takes none. One enabled policy per scope.
Category name to a score from 0 to 1; a score at or above it violates the policy. "*" covers every category not named.
Default: "{}"
block refuses the request or withholds the response; annotate only records the violation.
Default: "block"
What a streamed request gets when the policy has a post_call provider: reject refuses it with 400 guardrail_streaming_unsupported, buffer holds the whole stream, checks it once and then replays or withholds it. No value skips the check.
Default: "reject"
[pii_sanitizer]#
An external PII sanitizer: a self-hosted de-identification service, Microsoft Presidio or anything that speaks the same small JSON contract, that replaces each detected entity with a placeholder before the content reaches a provider and can reverse the substitution on the way back. Unlike the guardrail webhook, which returns a decision, it returns transformed content. The placeholder-to-plaintext mapping stays on the sanitizer, so Rolter never holds the values. Disabled by default. PII sanitizer explains the contract, restoration, streaming and the response cache in full.
The http:// or https:// endpoint that sanitizes content and returns placeholders. Required when enabled is true.
The endpoint that turns placeholders back into plaintext. Required unless restoration is never.
Which legs the sanitizer sees: request keeps personal data away from a third-party model, response scans what the model generated, and both does each with a separate call.
Default: "request"
When placeholders may be turned back into plaintext. never leaves them in the reply. trusted_downstream restores every response, for callers that supplied the data in the first place. caller_authorized restores only when the request carries x-rolter-pii-restore: true.
Default: "never"
What a streamed request does while the response leg is active. reject refuses it with 400 pii_streaming_unsupported; passthrough serves the stream with the response leg skipped; restore restores the stream through restore_stream_url. restore covers restoration only: when direction also covers the response, a streamed request is refused as under reject. It needs restore_stream_url and a restoration other than never. A request-only setup streams normally.
Default: "reject"
The endpoint that restores a stream over one full-duplex NDJSON exchange per streamed response. Required when streaming is restore. PII sanitizer describes the protocol.
The per-call timeout in milliseconds. It must be greater than zero.
Default: "5000"
fail_open forwards the unsanitized body when the sanitizer is unreachable, which sends untouched personal data upstream during an outage. fail_closed refuses the request with 502 pii_sanitizer_unavailable. Choose fail_closed when the sanitizer is a compliance control.
Default: "fail_open"
The most content sent to the sanitizer, in bytes. Oversized content is sent as a truncated preview with truncated: true. It must be greater than zero.
Default: "262144"
The entity types to detect, such as EMAIL_ADDRESS or PERSON. Empty means whatever the service is configured for.
Default: "[]"
An optional credential, in the same two forms as [guardrail_webhook], resolved from the environment at call time.
All three URLs are checked against the egress policy when the config loads and again before every call.
[prompt_templates]#
Centrally managed, versioned prompt templates. A template adds system, assistant or user messages around a caller’s own, so applications reuse approved instructions without being granted arbitrary prompt-authoring rights. Disabled by default, and a disabled or empty block adds no cost to a request. Prompt templates covers the dashboard repository, scopes and the request-time errors.
The templates, written as [[prompt_templates.templates]] entries. Each is one immutable version, and you list exactly the versions to activate. See below.
[[prompt_templates.templates]]#
A stable identifier. It appears in safe metadata, never in content logs. The (id, version) pair must be unique.
The immutable version, at least 1.
The named variables a decorator may reference as {{ name }}, written as [[prompt_templates.templates.variables]].
The messages injected around the caller’s own, written as [[prompt_templates.templates.decorators]]. A template needs at least one.
[[prompt_templates.templates.variables]]
The variable name, matching [A-Za-z_][A-Za-z0-9_]*.
The caller must supply the variable on every request the template applies to, whether or not a decorator references it. Mutually exclusive with default.
Default: "false"
The value used when the caller omits the variable. An optional variable with no default renders as an empty string.
[[prompt_templates.templates.decorators]]
The role of the injected message: system, assistant or user. For Anthropic, a system decorator folds into the top-level system field.
Default: "system"
prepend places the message before the caller’s messages and append places it after, both in the order declared.
Default: "prepend"
The message text, which may contain {{ variable }} placeholders. Every placeholder is checked at config load to name a declared variable.
Callers pass values in a rolter_template_vars object on the request body, which is always stripped before the request goes upstream. A caller’s value overrides the declared default. An unknown variable, a missing required variable, a value over 4 KiB or a rendered message over 16 KiB is refused with invalid_prompt_template. Substitution is structural: each rendered message is emitted as a JSON string through the serializer, never concatenated into raw JSON, so a value cannot break out of its string or inject extra messages.
Templates are applied to /v1/chat/completions, /v1/responses and Anthropic /v1/messages. Surfaces without a message array, such as /v1/completions, are not decorated.
In a database-backed deployment, templates are authored on the Prompt Repository screen, and rolter-seed --import seeds the templates in this table into the database, creating missing versions and refusing to overwrite an existing version with different content.
[plugins]#
Not a rolter.toml table. It holds the enabled webhook plugin instances that the control plane builds from the plugin registry (Plugins on the dashboard) and ships in the snapshot, for the gateway’s pre_route, pre_upstream and post_response stages. The list is empty by default and costs nothing on the request path. Each instance carries its slug, org_id, an optional project_id, stage, position, failure_mode, endpoint and an optional auth. See Plugins, which is experimental, for scope, ordering and failure policy.
The repository ships a fully annotated rolter.example.toml that shows every table a rolter.toml can set, with inline comments explaining each field; most blocks are commented out with their defaults. Copy it to rolter.toml to get started, then remove or uncomment the sections you need. It leaves out the tables the control plane owns and a file does not set: feature_flags, db_virtual_keys, mcp_servers, mcp_oauth_sessions, mcp_tool_groups, mcp_settings and plugins.