Rolter is a high-performance AI gateway written in Rust that sits in front of your upstream model providers. Any SDK or HTTP client that already speaks the OpenAI or Anthropic API works with Rolter unchanged — point it at the gateway base URL and supply a Rolter virtual key. Rolter handles authentication, load balancing, retries, rate limiting, and spend enforcement before the request ever reaches an upstream.
What this surface guarantees across upgrades — and what happens when OpenAI or Anthropic change their own dialect — is stated in Versioning & compatibility.
Base URL#
The default port is 4000, configurable via [server] port in rolter.toml.
Authentication#
Every inference endpoint requires a virtual key. You can pass it in either of two ways, matching the convention of the SDK you are using:
| Style | Header |
|---|---|
| OpenAI-style | Authorization: Bearer <virtual-key> |
| Anthropic-style | x-api-key: <virtual-key> |
When no virtual keys are configured in rolter.toml the gateway runs open, which is convenient for local development. In production, define at least one key under [[virtual_keys]].
Virtual keys are stored as peppered digests — the plaintext key never persists in gateway memory after the initial handshake. Revoke a key instantly by setting disabled = true in your config, or set expires_at for time-bounded access.
Endpoints#
| Method | Path | Description |
|---|---|---|
POST | /v1/chat/completions | OpenAI Chat Completions; streaming via "stream": true (SSE) |
POST | /v1/completions | OpenAI legacy text completions |
POST | /v1/messages | Anthropic Messages API; streaming supported |
GET | /v1/models | List configured model route names your key can access |
GET | /healthz | Liveness check — returns 200 OK when the gateway is running |
GET | /readyz | Readiness check — 503 while the instance is draining |
GET | /metrics | Prometheus metrics in exposition format |
Routing via the model field#
The model field in your request body selects a route defined in rolter.toml, not necessarily a literal upstream model name. Rolter looks up the route, applies any admin-configured param defaults, picks a target according to the route’s balancing strategy, rewrites model to the target’s upstream model ID, and forwards the request with the provider’s credentials.
If the requested model name does not match any configured route, Rolter returns 404 with "code": "model_not_found".
Request headers#
Alternative to Authorization. Accepted on all /v1/* endpoints; preferred by Anthropic SDKs.
Pass a stable session identifier (e.g. a user or conversation ID) to enable session affinity. When present, the consistent-hash and cache-aware balancers pin the request to the same upstream replica, improving KV-cache hit rates across multi-turn conversations.
An optional client-supplied request identifier. Rolter echoes it in response headers and includes it in request logs for correlation. A deployment can carry the id in another header, such as x-correlation-id, by setting the request ID header; x-request-id is then neither read nor echoed.
Response headers#
Every proxied response carries x-rolter-* headers describing the routing
decision, so you can see which upstream served a request without enabling
ClickHouse logging.
The end-to-end request id — the value you sent, or one Rolter generated. Returned under the configured request ID header when the deployment sets one.
The provider whose target served the request (e.g. openai-prod).
The resolved upstream model the request was forwarded as.
The requested model name that was routed.
The A/B variant that served the request. Omitted on the classic single-pool path.
HIT when the response came from the cache, otherwise MISS.
The request parameters Rolter did not forward as you sent them, because the upstream that served the request speaks another dialect, as <parameter>=<disposition> entries separated by , . Omitted when every parameter went through. See Adjusted parameters.
Adjusted parameters#
When a route sends an OpenAI or Anthropic request to an upstream that speaks another dialect (an Anthropic, Gemini or Gemini Interactions provider), a parameter with no equivalent there cannot be forwarded. Rolter still answers the request, and names each such parameter in x-rolter-param-adjustments:
| Disposition | Meaning |
|---|---|
dropped | The upstream has no equivalent, so the parameter was not sent. top_k to Gemini, presence_penalty to Anthropic. |
clamped | The value was sent, limited to the range the upstream accepts: a temperature above 1 sent to Anthropic, which accepts 0 to 1, is sent as 1. |
translated | The parameter was sent as an approximation that loses part of its meaning: a tool_choice forcing one function, sent to Gemini as “call any function”. |
- Names, never values. The header carries parameter names from Rolter’s fixed list of dialect parameters and the disposition, nothing you sent. A field Rolter does not recognise is not echoed.
- Absent when nothing changed. Its presence is the signal. A parameter that is only renamed on the way (
max_completion_tokensto Anthropic’smax_tokens,stoptostop_sequences) keeps its meaning and is not listed, and neither aremodel,streamandstream_options, which Rolter carries itself. - Streams too. It is decided before the request is sent upstream, so a streamed response carries it in its headers like any other.
- Cache hits too. A response served from Rolter’s response cache carries the header of the request that produced the cached answer, so a hit discloses what its miss did. An entry cached by a Rolter older than this carries none until it expires.
- Recorded. The same entries are kept on the request’s log row, in the
param_adjustmentscolumn ofrequest_logsas<disposition>:<parameter>(dropped:top_k), onceclickhouse/022_param_adjustments.sqlis applied (see Upgrading). The LLM Logs drawer lists them under Parameters not sent as given, and the Compatibility screen andGET /api/v1/analytics/param-adjustmentsroll them up per route and provider, with the share of its traffic each one touched. - Informational. The header’s name and its
<parameter>=<disposition>shape are stable. The parameters it can name and the dispositions it uses may grow in any release, so treat an unknown disposition as “not forwarded as sent” rather than failing on it.
Error responses#
Rolter returns OpenAI-style JSON error bodies on every error path, so existing error-handling code works unchanged.
| Status | Meaning |
|---|---|
401 Unauthorized | Missing or invalid virtual key (type authentication_error, code invalid_api_key) |
403 Forbidden | Key is valid but not permitted to use the requested model |
402 Payment Required | A spend budget for the key, project, team, or org has been exceeded |
429 Too Many Requests | A rate limit (RPM or TPM) has been reached, or every upstream target answered 429 (code upstream_rate_limited); check the Retry-After response header |
404 Not Found | No route matches the requested model name |
502 Bad Gateway | An upstream target was unreachable on the last attempt (connection refused, reset or timed out). Credentials in the provider’s address are masked in the message |
503 Service Unavailable | Route has no targets, or every upstream target failed with a retryable status and none is left to try (code upstream_unavailable; the message names the last upstream status) |
When your route’s retry budget runs out on a retryable upstream status, you get the upstream’s own answer, status and body as it sent them. When rolter runs out of targets first, it answers itself: a 429 from the provider stays a 429 (code upstream_rate_limited, with the provider’s Retry-After when it sent one), and any other retryable failure is a 503 (code upstream_unavailable) whose message names the last upstream status. Both are logged with the upstream’s own reason; see What gets logged.