Providers & Routes: Configure LLM Routing in Rolter

Rolter sits between your application and one or more AI backends. You teach it where to forward traffic by declaring providers (the upstream services) and routes (the rules that decide which provider handles each model request). Together they give you a clean public API surface — your code always calls the same model name — while the gateway handles routing, failover, and load balancing behind the scenes.

Providers#

A provider is an upstream AI service that Rolter can forward requests to. Each provider entry names the service, tells Rolter which wire protocol it speaks (kind), and supplies the base URL and API key.

Project-scoped providers#

A provider or provider group is org-wide by default: every project of the organization can use it, so projects can share one as a fallback or an option. Give it a project and it becomes project-scoped: only virtual keys minted in that project can reach it, through a route or as provider-slug/model or group-slug/model. A key from another project of the organization, or a key with no project, gets 404 model_not_found, as if the provider did not exist, and GET /v1/models does not list it. Existing providers stay org-wide.

In the dashboard, the provider and provider group sheets have a Scope field: Whole organization (the default) or one project of the organization. The Providers and Provider Groups lists show it as a Scope column (Project: Gateway), and a route’s target picker offers only its own project’s providers and the org-wide ones. Someone who is not an organization admin can only pick their own project, cannot change the scope of an existing provider or group, and cannot set api_key_env on a project-scoped provider.

Set it with project_id on POST /api/v1/orgs/{org_id}/providers and /provider-groups, or change it later with PUT; send "project_id": null to make it org-wide again. The project must belong to the organization. A project admin can create and delete their own project’s providers; making one org-wide or moving it, and reading a credential from an environment variable (api_key_env), take an organization admin. Project members see their project’s providers and the org-wide ones when they list. A project that still owns a provider or group cannot be deleted until they are removed or made org-wide, so deleting a project never widens access silently.

Provider kinds#

Provider config example#

[[providers]]name = "openai"kind = "openai"api_base = "https://api.openai.com"api_key_env = "OPENAI_API_KEY"[[providers]]name = "anthropic"kind = "anthropic"api_base = "https://api.anthropic.com"api_key_env = "ANTHROPIC_API_KEY"[[providers]]name = "vllm-1"kind = "openai_compatible"api_base = "http://localhost:8001"# no api_key needed for local instances[[providers]]name = "vllm-2"kind = "openai_compatible"api_base = "http://localhost:8002"egress_proxy = "http://corp-proxy:3128"   # optional outbound proxy

Routes#

A route maps a public model name — what your application actually sends in the "model" field — to one or more upstream targets plus a load-balancing strategy. When a request arrives for "gpt-4o", Rolter looks up the matching route and forwards it to the appropriate provider.

Basic route example#

[[routes]]model = "gpt-4o"strategy = "round_robin"[[routes.targets]]provider = "openai"model = "gpt-4o"[[routes]]model = "claude"strategy = "round_robin"[[routes.targets]]provider = "anthropic"model = "claude-sonnet-4-20250514"

Inference params#

You can inject admin-controlled default inference parameters into every request on a route using [routes.params]. Any parameter key the upstream model accepts is valid — temperature, max_tokens, stop, top_p, and so on.

When a caller omits a parameter, the admin default is injected. When a caller sends a value, the param policy decides whether their override stands or the admin default silently wins.

Param policy#

[routes.param_policy] has two modes:

• allow (default) — callers may override any default except those listed in deny
• deny — callers may override no default except those explicitly listed in allow

[[routes]]model = "gpt-4o"strategy = "round_robin"[routes.params]temperature = 0.0    # force deterministic outputmax_tokens = 1024    # cap output length[routes.param_policy]mode = "deny"allow = ["max_tokens"]   # callers may still lower max_tokens, but not change temperature[[routes.targets]]provider = "openai"model = "gpt-4o"

Per-model advanced settings#

Beyond params, a route carries an advanced block: catalog metadata, request settings the gateway applies to every call the route makes, and catalog visibility. In the dashboard these are the “Limits & network”, “Custom request headers” and “Access & permissions” sections of a model’s sheet, and saving the sheet writes them back to the route. In a rolter.toml they sit under [routes.advanced].

What the gateway applies. Each of these changes what the gateway does for the route, and nothing else in the block does:

timeout_secs, retries, headers, rpm, tpm and concurrency apply to a request that names the route, on /v1/chat/completions, /v1/responses, the other JSON endpoints and the multipart audio uploads alike. They do not apply to a request addressed as provider-slug/model or group-slug/model, which has no route, nor to health probes. A Realtime session is one request for rpm, holds a concurrency slot until it closes, and its turns count towards tpm; timeout_secs, retries and headers do not apply to it.

Catalog metadata. model_type, capabilities and context_length (the context window in tokens, at least 1) are labels the dashboard reads back and GET /v1/models serves on the route’s entry when they are set, so an API client can learn what a model is for and how large its context is. description is a note for your team that the routes API returns; it is never sent to API clients. The gateway enforces none of them, so a request is forwarded whether or not a capability flag is set or the prompt fits context_length.

Route limits#

limits.rpm, limits.tpm and limits.concurrency protect one expensive model without limiting the keys that call it. They belong to the route, whoever calls: every key, team and project shares them, and they sit on top of the limits on a key, team, project or organization, which are checked first. A request over either is refused with the same 429.

• A refusal is a 429 with code rate_limit_exceeded and a Retry-After header, in the same error envelope on /v1/chat/completions, /v1/messages and the other endpoints. rpm and tpm say how long until the window frees capacity; concurrency says 1. A refused request never reaches an upstream. Nothing is queued: a client that wants to wait retries.
• rpm and tpm are shared by the fleet. The counters live in the Redis the other throughput limits use, on a key of the route’s own organization and name, so two replicas count one total and the same route name in two organizations is two counters. They use the same sliding one-minute window as every other limit.
• tpm is recorded after the response, as for the other scopes. A request is refused when the window already holds the cap, so requests in flight together can take it over by their own usage.
• concurrency is per replica. Each gateway counts the requests it is serving, so three replicas with concurrency = 4 serve up to twelve at once. It needs no Redis. A request holds its slot until its response has been delivered, which for a stream is the last event, and a client that disconnects mid-stream frees it.
• Without Redis (no redis_url, or Redis unreachable) rpm and tpm are not enforced, the same fail-open every other throughput limit has, and the rolter_fail_open_total counter says so. concurrency keeps holding.
• A refusal costs the caller a request. The caller’s own rpm is charged before the route is known, so a request the route refuses has still taken a slot of the key’s rpm.
• A route addressed through a complexity tier is limited by the route that serves it, not by the one the client named.
• Refusals are counted in rolter_route_limit_blocks_total{route,kind} (see Metrics) as well as in rolter_rate_limit_blocks_total.

The built-in fake-llm model has no route and no limits.

Route headers#

headers is a map of header name to value, sent with every upstream call the route makes. A header the deployment forwards from the caller (forwarded_headers under Client settings; a caller cannot send any other header upstream) is the caller’s to choose unless you list its name in locked_headers:

• Unlocked (the default): the route’s value is a default, and a caller’s header of the same name replaces it.
• Locked: the route’s value replaces a caller’s header of the same name.

A route header also replaces a deployment-wide injected header of the same name.

Credentials belong to the provider. A route may not set authorization, proxy-authorization, x-api-key, api-key or x-goog-api-key, nor the headers that frame the request (host, content-length, content-type, transfer-encoding, connection and the other hop-by-hop headers). The routes API refuses one with 400, a rolter.toml that sets one fails rolter check, and a stored row that carries one anyway (written around the API) has the header left out of the snapshot while the route keeps serving; GET /api/v1/config/problems names it. The same applies to a name that is not a valid header name or a value that is not visible ASCII on one line.

Settings that never took effect#

Earlier releases stored, validated and displayed several more settings that the gateway did not read, and the dashboard and these docs described them as working. They are gone from the model sheet and the API, and Rolter now ignores them:

A route that still carries one of these keeps loading, and requests are served exactly as they always were. Nothing is migrated. The routes API accepts them from an older client and drops them instead of storing them, saving a model’s sheet in the dashboard sheds them from the route, and a rolter.toml that sets one is reported as an unknown key at startup and by rolter check. To find the routes that still carry one before you rely on any of this, see Upgrading.

A route is only ever served to keys of the organization it belongs to. For a key from another organization the route does not exist: the key gets 404 with code model_not_found, as for a model nobody configured, and does not see it in GET /v1/models. The route does not hide anything of that key’s own either, so the key’s own edge provider still answers edge/gpt-4o when another organization has a route by that name. Within the organization, the sheet’s Access & permissions → Visibility control narrows it further:

• Whole organization (the default) — any key in the organization.
• This project — only keys minted in the project the route lives in (visibility.project_only). This narrows the route, not the provider behind it. To keep a provider to one project, scope the provider itself, see Project-scoped providers.
• Restricted — only the teams, virtual keys and users you list.

The gateway enforces the organization boundary, project_only, allowed_team_ids and allowed_key_ids on every endpoint that names a model, including audio uploads and Realtime sessions. A key outside the organization or outside a This project route gets 404 with code model_not_found; a key the allowed teams or keys leave out gets 403 with code model_not_allowed. allowed_user_ids and minimum_role apply in the dashboard, since a virtual key carries no user. Keys defined in the gateway’s own config file belong to the operator rather than to an organization and are not narrowed. See the same checks on every endpoint.

The control plane validates the whole block before storing it and refuses the save with the offending field named, so an out-of-range limit or a malformed header name never reaches the gateway. The dashboard reports that refusal on the sheet, with the rest of the model already saved.

The block used to carry an additional_fields map. It has been removed: nothing ever read it, and it could not do what its name suggested either — a request field Rolter does not model is already forwarded to the upstream as-is, so there was nothing for it to gate. The “Allow additional fields” switch that stood in for an editor went first, in #1271, because it had no field of its own on the wire.

If a route of yours still has the key, nothing breaks: the stored block loads as before, every other setting in it is unaffected, and the key is simply ignored. Rolter never writes it back out itself. In a rolter.toml it is now reported by the unrecognised-key warning at startup, which is a warning and not a refusal to start — delete the line to silence it.

Variants#

Variants let you run A/B tests, canary deployments, and per-key traffic splits under a single route. Instead of defining a flat targets pool, you define named variant buckets, each with its own weight, targets, and optional param overrides.

Each variant can carry its own params table. Variant params layer over the route-level params (variant wins on collision), and the route’s param_policy governs both.

[[routes]]model = "chat"[[routes.variants]]name = "control"weight = 9[[routes.variants.targets]]provider = "openai"model = "gpt-4o"[[routes.variants]]name = "canary"weight = 1[[routes.variants.targets]]provider = "anthropic"model = "claude-sonnet-4-20250514"

A route created in the dashboard or through the API can carry variants too: open the route’s sheet and use its Variants section, which previews the traffic split as you set the weights, or call PUT /api/v1/routes/{id}/variants. rolter-seed --import stores a file route’s variants in the database and rolter config export writes them back, so a route keeps its split when it moves between the file and the control plane.

Variant-level param overrides#

[[routes]]model = "chat"[routes.params]temperature = 0.7    # route-level default[[routes.variants]]name = "control"weight = 9[[routes.variants.targets]]provider = "openai"model = "gpt-4o"[[routes.variants]]name = "experimental"weight = 1[routes.variants.params]temperature = 1.2    # overrides the route default for this variant only[[routes.variants.targets]]provider = "openai"model = "gpt-4o"