> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Active Health Checks for Upstream Providers in Rolter > Enable Rolter's background health probing to automatically skip unavailable upstream providers during routing, before a real request ever fails. By default, Rolter discovers that an upstream is unhealthy only when a real request fails. Active health checks flip this around: a background task periodically issues a lightweight `GET` request to each provider, and any provider that fails several consecutive probes is marked unhealthy and pulled from rotation — before it causes a client-visible error. After enough consecutive successful probes, the provider is quietly restored. ## What counts as healthy Rolter probes each provider with a lightweight `GET`. Any response that is **not** a 5xx status is treated as healthy, including `401 Unauthorized` and `404 Not Found` — a timeout, connection failure, or 5xx counts as a probe failure. A single failed probe never flips a provider: it takes `consecutive_failure_threshold` failures in a row (default 3) to mark it unhealthy, and `recovery_success_threshold` successes in a row (default 2) to restore it, so one transient blip neither removes a provider nor bounces it in and out of rotation. By default (`path = "/"`), the probe targets each provider kind's **free, non-inference liveness endpoint** so a healthy result means the API itself is up, not merely that the host answers TCP — and it burns no tokens: | Provider kind | Probe | | ----------------------------- | ----------------------------------------------------------------------- | | `openai`, `openai_compatible` | `GET {api_base}/v1/models` | | `anthropic` | `GET {api_base}/v1/models` with the required `anthropic-version` header | | `system_one` | `GET {api_base}/v1/models` | Set an explicit `path` (e.g. `/healthz`) to override this for every provider and probe that exact route instead. The "any non-5xx is healthy" rule belongs to this free probe. The [end-to-end check](#end-to-end-llm-call-checks) below is judged more strictly, because it is aimed at the endpoint that has to work. When the provider has a key, the probe sends it the way that provider's API reads it (`x-api-key` for Anthropic, `api-key` for Azure OpenAI, `x-goog-api-key` for native Gemini, a bearer token for the rest), the same as a forwarded request. A `401` still counts as healthy, but only an authenticated model list can be read, so a provider whose probe is rejected contributes no models to `/v1/models`. **Fail-open guarantee:** when every target of a route is marked unhealthy, Rolter does not return an error. It fails open and still forwards the request, preferring a potential upstream error over a guaranteed 503 from the gateway. ## Configuration Health checks are **disabled by default** because probing adds background traffic to every configured provider. Add a `[health]` section to `rolter.toml` and set `enabled = true` to activate them. Master switch. Set to `true` to start the background probing task. Seconds between probe sweeps. Every provider is probed once per sweep. Per-probe timeout in seconds. A probe that does not receive a response within this window is treated as a failure. Request path appended to each provider's `api_base` when probing. Change this if your upstreams expose a dedicated health route. Maximum probes in flight at once during a sweep. Probing is bounded and jittered (each provider gets a stable offset within the first quarter of the interval), so a sweep never hits all upstreams simultaneously. Consecutive probe failures required before a provider is marked unhealthy. Consecutive probe successes required before an unhealthy provider returns to rotation. ```toml rolter.toml [health] enabled = true interval_secs = 15 timeout_secs = 3 path = "/health" ``` ## End-to-end (LLM-call) checks By default the probe hits a free liveness endpoint that burns no tokens. If you need a signal that real inference works — not just that the API answers — enable `also_track_via_llm_call` on a provider. Each sweep then sends a minimal `max_tokens = 1` completion instead of the free probe. Send a real `max_tokens = 1` completion instead of the free `/v1/models` probe. **Costs a few tokens on every sweep** (`interval_secs`) — leave it off unless you need inference-level health. Recorded as `source = llm_call` in `provider_health_events`; it feeds the same failure/recovery thresholds as the free probe. The upstream model id the check calls (e.g. `gpt-4o-mini`). **Required** when the flag is on; without it (or a resolvable api key) the checker logs a warning and falls back to the free probe. The check calls the same endpoint a forwarded request would reach, in that provider's own format: the Messages API for `anthropic`, `generateContent` for `gemini_native`, an embedding for a `tei` server, one yes/no question for a `system_one` decision server, and a chat completion for the rest. It follows your `api_base` exactly as routing does, so a base that already ends in `/v1` is not given a second one. ### What counts as healthy for an LLM-call check Because this check exists to prove that inference works, it does not give the free probe's benefit of the doubt: | Answer | Counts as | Why | | ----------------------------------------- | ----------- | --------------------------------------------------------------------------- | | `404` or `405` | **failure** | The check is pointed at the wrong place: check `api_base` and the model id. | | `401` or `403` | **failure** | The configured key cannot call the model. | | `429` | healthy | The model answered; probing backs off as described below. | | any other status below 500 (`200`, `400`) | healthy | The model answered, or declined this prompt for its own reasons. | | `5xx`, timeout, connection failure | **failure** | The provider is not serving. | A failure of the first two kinds is logged as a warning naming the provider, the status and what to check, and is recorded in `provider_health_events` with `error_kind` set to `wrong_endpoint` or `key_refused`. It counts toward `consecutive_failure_threshold` like any other failed probe. The free probe keeps its own rule and is unaffected. ```toml rolter.toml [[providers]] name = "openai" kind = "openai" api_base = "https://api.openai.com" api_key_env = "OPENAI_API_KEY" also_track_via_llm_call = true llm_probe_model = "gpt-4o-mini" ``` The tokens spent scale with your sweep frequency: one completion per provider per `interval_secs`. On a 10s interval that is ~8,640 tiny calls per provider per day. Prefer a longer `interval_secs` for LLM-call checks. ## Status-page signal (secondary) Set a provider's `status_page_url` to a statuspage.io-style `status.json` and rolter polls it on a slow cadence (`health.status_page_interval_secs`, default 60s). A non-operational indicator is recorded as a `status_page` event in `provider_health_events` and increments `rolter_status_page_degraded_total` — so a provider-wide incident surfaces on your dashboard _before_ rolter's own probes start failing. The URL is checked against the [egress policy](/configuration/egress) when the configuration loads and before every poll; a denied URL is skipped with a warning. Rolter does not follow redirects from it. This is a **secondary** signal only. It never marks a provider unhealthy or changes routing on its own — it exists to give early visibility. Transport or JSON-parse failures are logged and skipped, never counted as a provider error. The poller runs whenever any provider sets `status_page_url`, even with `health.enabled = false`. ```toml rolter.toml [[providers]] name = "anthropic" kind = "anthropic" api_base = "https://api.anthropic.com" api_key_env = "ANTHROPIC_API_KEY" status_page_url = "https://status.anthropic.com/api/v2/status.json" [health] status_page_interval_secs = 60 ``` ## Activation walkthrough Add the `[health]` section to your `rolter.toml` with `enabled = true`. The background task starts when the gateway process starts. Choose `interval_secs` based on how quickly you need to detect an outage versus how much extra background traffic you are willing to generate. For self-hosted pools, 10–30 seconds is a reasonable starting point. If your upstreams expose a dedicated health endpoint (e.g. `/health`, `/healthz`, `/ping`), set `path` to that route so the probe gets a meaningful response. If not, leave `path = "/"` — a 404 still counts as healthy. Set `timeout_secs` to a value lower than your `interval_secs`. A `timeout_secs = 2` with `interval_secs = 15` means a slow-responding provider is flagged promptly and the next sweep happens well before the previous probe would have hung the worker. If a probe itself receives `429 Too Many Requests`, the provider still counts as healthy (the API is up), but Rolter pauses probing that provider with exponential backoff — 1, 2, 4, then 8 sweeps — so the prober never contributes to tripping your provider's rate limits. Active health checks are most valuable for **self-hosted vLLM, TGI, or SGLang pools** where instances can crash or restart independently. For managed APIs like OpenAI and Anthropic, cooldowns and retries are usually sufficient — those providers are highly available and probing them generates billable traffic. ## Provider health in the dashboard **Models → Provider Health** reads `provider_health_events` over the last 7 days (probes, status-page polls and real traffic) and measures each provider against a 99% SLA. One card per provider, with the targets that carried its traffic listed inside it. The screen does not show circuit-breaker state. The gateway keeps its breakers in memory and does not report them to the control plane; trips are counted as `rolter_breaker_trip_total` in [`/metrics`](/observability/metrics). See [Circuit breakers](/reliability/circuit-breakers). Each card, and each target inside it, is in one of three states: | State | What it means | | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | below SLA | More than 1% of the last 7 days' events failed. The state holds until those failures leave the window, even if the provider has since recovered. | | SLA at risk | The 7 days still meet the SLA, but more than 1% of events in the last two hours failed, so the error budget is being spent faster than the SLA allows. | | within SLA | The 7 days meet the SLA, and so do the last two hours. | The last two hours are the two newest hourly buckets in the timeline. A probed provider is judged on its own probes and status-page polls. A provider with no probes has no events of its own, so its card is rolled up from its targets and shows the worst of their states. **Models** reads the same three states for the targets of each route: the catalog row counts the targets below SLA, or at risk when none are below, and the expanded list says "SLA at risk" in words beside the uptime. Where the timeline cannot be read, a target is judged on the 7-day rollup alone. **Routing Rules** puts the same state under each target on its cards. **Error budget burn** is the 7-day failure rate as a share of the 1% the SLA allows. Above 100% is a breach. ## Gateway liveness endpoint Separate from upstream health checks, Rolter exposes its own liveness endpoint at: ```text GET /healthz ``` This endpoint always returns `200 OK` as long as the gateway process is running and able to serve requests. Use it to integrate Rolter with container orchestrators and load balancers: ```yaml Kubernetes liveness probe livenessProbe: httpGet: path: /healthz port: 4000 initialDelaySeconds: 5 periodSeconds: 10 ``` ```dockerfile Docker HEALTHCHECK HEALTHCHECK --interval=10s --timeout=3s --start-period=5s \ CMD curl -f http://localhost:4000/healthz || exit 1 ``` `/healthz` reflects only the gateway process's own health, not the health of upstream providers. An instance can return `200` on `/healthz` while all upstream providers are unhealthy — use the `[health]` probing config (above) to monitor upstream availability, and Prometheus metrics to alert on routing failures.