By default, Rolter discovers that an upstream is unhealthy only when a real request fails. Active health checks flip this around: a background task periodically issues a lightweight GET request to each provider, and any provider that fails several consecutive probes is marked unhealthy and pulled from rotation — before it causes a client-visible error. After enough consecutive successful probes, the provider is quietly restored.
What counts as healthy#
Rolter probes each provider with a lightweight GET. Any response that is not a 5xx status is treated as healthy, including 401 Unauthorized and 404 Not Found — a timeout, connection failure, or 5xx counts as a probe failure. A single failed probe never flips a provider: it takes consecutive_failure_threshold failures in a row (default 3) to mark it unhealthy, and recovery_success_threshold successes in a row (default 2) to restore it, so one transient blip neither removes a provider nor bounces it in and out of rotation.
By default (path = "/"), the probe targets each provider kind’s free, non-inference liveness endpoint so a healthy result means the API itself is up, not merely that the host answers TCP — and it burns no tokens:
| Provider kind | Probe |
|---|---|
openai, openai_compatible | GET {api_base}/v1/models |
anthropic | GET {api_base}/v1/models with the required anthropic-version header |
system_one | GET {api_base}/v1/models |
Set an explicit path (e.g. /healthz) to override this for every provider and probe that exact route instead.
The “any non-5xx is healthy” rule belongs to this free probe. The end-to-end check below is judged more strictly, because it is aimed at the endpoint that has to work.
When the provider has a key, the probe sends it the way that provider’s API reads it (x-api-key for Anthropic, api-key for Azure OpenAI, x-goog-api-key for native Gemini, a bearer token for the rest), the same as a forwarded request. A 401 still counts as healthy, but only an authenticated model list can be read, so a provider whose probe is rejected contributes no models to /v1/models.
Fail-open guarantee: when every target of a route is marked unhealthy, Rolter does not return an error. It fails open and still forwards the request, preferring a potential upstream error over a guaranteed 503 from the gateway.
Configuration#
Health checks are disabled by default because probing adds background traffic to every configured provider. Add a [health] section to rolter.toml and set enabled = true to activate them.
Seconds between probe sweeps. Every provider is probed once per sweep.
Default: "10"
Per-probe timeout in seconds. A probe that does not receive a response within this window is treated as a failure.
Default: "2"
Request path appended to each provider’s api_base when probing. Change this if your upstreams expose a dedicated health route.
Default: "/"
Maximum probes in flight at once during a sweep. Probing is bounded and jittered (each provider gets a stable offset within the first quarter of the interval), so a sweep never hits all upstreams simultaneously.
Default: "2"
Consecutive probe failures required before a provider is marked unhealthy.
Default: "3"
Consecutive probe successes required before an unhealthy provider returns to rotation.
Default: "2"
End-to-end (LLM-call) checks#
By default the probe hits a free liveness endpoint that burns no tokens. If you need a signal that real inference works — not just that the API answers — enable also_track_via_llm_call on a provider. Each sweep then sends a minimal max_tokens = 1 completion instead of the free probe.
Send a real max_tokens = 1 completion instead of the free /v1/models probe. Costs a few tokens on every sweep (interval_secs) — leave it off unless you need inference-level health. Recorded as source = llm_call in provider_health_events; it feeds the same failure/recovery thresholds as the free probe.
Default: "false"
The upstream model id the check calls (e.g. gpt-4o-mini). Required when the flag is on; without it (or a resolvable api key) the checker logs a warning and falls back to the free probe.
The check calls the same endpoint a forwarded request would reach, in that provider’s own format: the Messages API for anthropic, generateContent for gemini_native, an embedding for a tei server, one yes/no question for a system_one decision server, and a chat completion for the rest. It follows your api_base exactly as routing does, so a base that already ends in /v1 is not given a second one.
What counts as healthy for an LLM-call check#
Because this check exists to prove that inference works, it does not give the free probe’s benefit of the doubt:
| Answer | Counts as | Why |
|---|---|---|
404 or 405 | failure | The check is pointed at the wrong place: check api_base and the model id. |
401 or 403 | failure | The configured key cannot call the model. |
429 | healthy | The model answered; probing backs off as described below. |
any other status below 500 (200, 400) | healthy | The model answered, or declined this prompt for its own reasons. |
5xx, timeout, connection failure | failure | The provider is not serving. |
A failure of the first two kinds is logged as a warning naming the provider, the status and what to check, and is recorded in provider_health_events with error_kind set to wrong_endpoint or key_refused. It counts toward consecutive_failure_threshold like any other failed probe. The free probe keeps its own rule and is unaffected.
The tokens spent scale with your sweep frequency: one completion per provider per interval_secs. On a 10s interval that is ~8,640 tiny calls per provider per day. Prefer a longer interval_secs for LLM-call checks.
Status-page signal (secondary)#
Set a provider’s status_page_url to a statuspage.io-style status.json and rolter polls it on a slow cadence (health.status_page_interval_secs, default 60s). A non-operational indicator is recorded as a status_page event in provider_health_events and increments rolter_status_page_degraded_total — so a provider-wide incident surfaces on your dashboard before rolter’s own probes start failing.
The URL is checked against the egress policy when the configuration loads and before every poll; a denied URL is skipped with a warning. Rolter does not follow redirects from it.
This is a secondary signal only. It never marks a provider unhealthy or changes routing on its own — it exists to give early visibility. Transport or JSON-parse failures are logged and skipped, never counted as a provider error. The poller runs whenever any provider sets status_page_url, even with health.enabled = false.
Activation walkthrough#
Enable health checks
Add the [health] section to your rolter.toml with enabled = true. The background task starts when the gateway process starts.
Set a probe interval
Choose interval_secs based on how quickly you need to detect an outage versus how much extra background traffic you are willing to generate. For self-hosted pools, 10–30 seconds is a reasonable starting point.
Set a probe path
If your upstreams expose a dedicated health endpoint (e.g. /health, /healthz, /ping), set path to that route so the probe gets a meaningful response. If not, leave path = "/" — a 404 still counts as healthy.
Tune the timeout
Set timeout_secs to a value lower than your interval_secs. A timeout_secs = 2 with interval_secs = 15 means a slow-responding provider is flagged promptly and the next sweep happens well before the previous probe would have hung the worker.
If a probe itself receives 429 Too Many Requests, the provider still counts as healthy (the API is up), but Rolter pauses probing that provider with exponential backoff — 1, 2, 4, then 8 sweeps — so the prober never contributes to tripping your provider’s rate limits.
Active health checks are most valuable for self-hosted vLLM, TGI, or SGLang pools where instances can crash or restart independently. For managed APIs like OpenAI and Anthropic, cooldowns and retries are usually sufficient — those providers are highly available and probing them generates billable traffic.
Provider health in the dashboard#
Models → Provider Health reads provider_health_events over the last 7 days (probes, status-page polls and real traffic) and measures each provider against a 99% SLA. One card per provider, with the targets that carried its traffic listed inside it.
The screen does not show circuit-breaker state. The gateway keeps its breakers in memory and does not report them to the control plane; trips are counted as rolter_breaker_trip_total in /metrics. See Circuit breakers.
Each card, and each target inside it, is in one of three states:
| State | What it means |
|---|---|
| below SLA | More than 1% of the last 7 days’ events failed. The state holds until those failures leave the window, even if the provider has since recovered. |
| SLA at risk | The 7 days still meet the SLA, but more than 1% of events in the last two hours failed, so the error budget is being spent faster than the SLA allows. |
| within SLA | The 7 days meet the SLA, and so do the last two hours. |
The last two hours are the two newest hourly buckets in the timeline. A probed provider is judged on its own probes and status-page polls. A provider with no probes has no events of its own, so its card is rolled up from its targets and shows the worst of their states.
Models reads the same three states for the targets of each route: the catalog row counts the targets below SLA, or at risk when none are below, and the expanded list says “SLA at risk” in words beside the uptime. Where the timeline cannot be read, a target is judged on the 7-day rollup alone.
Routing Rules puts the same state under each target on its cards.
Error budget burn is the 7-day failure rate as a share of the 1% the SLA allows. Above 100% is a breach.
Gateway liveness endpoint#
Separate from upstream health checks, Rolter exposes its own liveness endpoint at:
This endpoint always returns 200 OK as long as the gateway process is running and able to serve requests. Use it to integrate Rolter with container orchestrators and load balancers:
/healthz reflects only the gateway process’s own health, not the health of upstream providers. An instance can return 200 on /healthz while all upstream providers are unhealthy — use the [health] probing config (above) to monitor upstream availability, and Prometheus metrics to alert on routing failures.