Cooldowns park a target after each individual failure, but they have a blind spot: a target that is persistently down still re-enters the pool on every cooldown expiry and absorbs a probe request — wasting latency and potentially surfacing errors to real clients. Circuit breakers solve this by tracking consecutive failures and keeping a broken target out of rotation for a meaningful window, with a single controlled probe to detect recovery.
How circuit breakers work#
Each target has an independent circuit breaker that moves through three states:
Normal operation. Requests flow to the target. Consecutive failures are counted.
The target tripped after failure_threshold consecutive failures. Rolter skips it entirely for open_secs. No real traffic reaches it during this window.
The open window expired. The next request is used as a probe, and it is the only one: other requests skip the target until the probe’s result arrives. A successful response closes the breaker; another failure immediately re-opens it for another open_secs. A probe that never answers (the caller disconnected) gives up its slot after the longest of open_secs, 30 seconds and the upstream request timeout ([timeouts]), and the next request probes instead; a probe that never reached the target, such as one a full provider queue turned away, gives it up at once. One exception: when every target of a route is skipped (a route with a single target, for example), Rolter still fails open and sends requests to the probing target rather than refusing them, so the one-probe rule protects a target only while it has siblings to take the traffic. Whether that should change is tracked in #3389.
The transitions look like this:
A “transient failure” for counting purposes is the same set of conditions that trigger retries and cooldowns: HTTP 429, any 5xx response, or a connection error.
Fail-open guarantee: when every target in a route has a tripped breaker, Rolter does not reject the request. It fails open and still forwards, giving you a live error from the upstream rather than a guaranteed gateway error.
Configuration#
Circuit breakers are disabled by default. Add a [breaker] section to rolter.toml and set enabled = true to activate them.
Master switch. Set to true to activate circuit breakers on all targets. A failure_threshold of 0 also disables the breaker even when enabled = true.
Default: "false"
Number of consecutive transient failures (HTTP 429/5xx or connection error) required to trip a target open. Set to 0 to effectively disable the breaker.
Default: "5"
How long a tripped target stays open — and invisible to the balancer — before the first half-open probe is allowed through, in seconds.
Default: "30"
A low failure_threshold (e.g. 1 or 2) combined with a high open_secs can trip a breaker on transient noise and keep a healthy target out of rotation for a long time. Start with conservative values (failure_threshold = 3–5, open_secs = 30–60) and adjust based on observed behaviour.
Watching breakers across the fleet#
Every gateway replica keeps its own breakers, so one replica can have a target open while another still
sends it traffic. When the gateways run against a control plane, each one reports its breaker state
every 10 seconds, and the dashboard shows the fleet’s picture on Provider Health and on the target
rows of Models and Routing Rules: a target is marked when it is open or probing on at least one
gateway, with how many. The same data is served by GET /api/v1/breaker-state.
A target the view does not list is closed everywhere.
Choosing between cooldowns and circuit breakers#
Both mechanisms protect your pool from a degraded target, but they operate on different time scales and failure patterns:
Best for individual wobbles — a single 429, a momentary spike, or a brief connectivity hiccup. The target is back in rotation after a short, fixed window without any probe logic.
Best for sustained outages — a target that fails consistently and repeatedly. The breaker keeps it fully out of rotation and uses a single controlled probe to detect recovery, minimising wasted capacity.
Circuit breakers complement cooldowns — they are not mutually exclusive. When both are configured, a single failure triggers a cooldown park; if failures accumulate past failure_threshold across those cooldown cycles, the breaker trips open and takes over with the longer open_secs window. You get short-term protection and long-term load shedding from a single configuration.