Rolter connects models in two moves: a provider says how to reach an upstream (its kind, base URL and credential), and a route maps the public model name your clients send to one or more provider targets plus a strategy for choosing between them. Everything below is those two moves, repeated in different shapes. See Providers & Routes for the full reference.
Which shape are you?#
| You have | Balancing | Start at |
|---|---|---|
| one hosted provider, one or two models | none — one target per route | One provider |
| the same model from two providers | failover — retries move traffic off a failing target | Failover |
| several vendors | none or failover, per route | Several vendors |
| a self-hosted fleet of replicas | load-balanced — a strategy per route | A fleet |
| any of these, and you want it in git | — | As code |
Before any provider exists, the built-in fake-llm model answers in the Playground and through the API. Use it to prove the path from client to log first; then every step below changes only where the answer comes from.
One provider#
Add the provider
Models → Model Providers → + Add provider: pick the kind (OpenAI, Anthropic, Gemini, Mistral, OpenRouter, a self-hosted vLLM, Ollama or llama.cpp, and many more), the base URL, and the API key. The key is sealed with the deployment’s key-encryption key and never shown again.
Test the connection
Test connection asks the upstream for its model list and says so if it gets anything else. Fix a wrong key or URL here, not after a client fails.
Add a route
Models → Routing Rules → + Add route (Model Catalog has the same button and opens the same sheet): the public name clients will send (say gpt-4o), a strategy, and the provider target it lands on. Pick the provider yourself when you have more than one; the Upstream model field then suggests what that provider lists, and takes any id you type. The magnifier button beside it checks that the provider lists the id (it needs the same permission as Test connection), and says plainly when the provider has no model list to check against. It appears in Model Catalog, and in GET /v1/models for every key allowed to reach it.
Try it
Send a message from the Playground, then find the request in Observability → LLM Logs with its provider, latency and cost.
Failover#
Give a route two targets that serve the same model — two regions of one vendor, or two vendors. In the route sheet, + Add target adds the second line. A transient failure on one (a 5xx, a timeout) is retried on the other before the client sees it, a target that keeps failing is put in cooldown, and a circuit breaker stops sending it traffic until it recovers.
To split a route’s traffic between groups of targets instead, for an A/B test or a canary, use the route sheet’s Variants section: each variant has a name, a weight and its own targets in the order they are tried, and the sheet previews the resulting split as you type. See Variants.
- Retries, backoff and timeouts: Settings → Performance Tuning, and Retries & cooldowns.
- Uptime and SLA state per provider and target: Models → Provider Health, and Provider health in the dashboard. Breaker trips are counted in
/metrics; see Circuit breakers. - Probe targets before a request fails: Health checks.
Several vendors#
One provider per vendor, one route per public model name. Clients ask for gpt-4o or claude-sonnet-4 and never learn which vendor answered — the OpenAI and Anthropic dialects are both served, and requests are translated between them when a route crosses over.
On each route you can pin parameter defaults (a default temperature, a capped max_tokens) and a parameter policy deciding which of them callers may override. Deployment-wide fallbacks for anything a caller omits live under Models → Model Settings. See Model settings.
A fleet#
For many replicas of self-hosted engines (vLLM, SGLang, TGI):
- Provider groups name a set of providers once, and
group/modeladdresses the whole set — Models → Provider Groups. - Pick a strategy per route, matched to the workload. It is chosen when the route is created, in the route sheet or in
rolter.toml. To change it later, along with the route’s targets and weights, usePUT /api/v1/routes/{id}or re-importrolter.toml; the route sheet cannot change it yet (#2134).
| workload | strategy |
|---|---|
| identical replicas, nothing special | round_robin |
| a canary or an uneven fleet | weighted |
| latency matters most | fastest, predicted_latency |
| multi-turn chat or long shared prefixes | cache_aware (keeps KV caches warm) |
| spread load with little coordination | power_of_two |
| let observed latency, cost and load decide | adaptive (turn it on under Adaptive Routing → Settings) |
See Load balancing, Cache-aware routing and Adaptive routing.
As code#
Keep providers, provider groups, routes, prices and prompt templates in a rolter.toml and apply it as the desired state:
Credentials never live in the file — a provider names the environment variable its key is read from. Keys, budgets, SSO and deployment policy are not part of the file; manage those in the dashboard or through the API. See Config file and Config export.