Fallback Targets That Take Over Specific Failures

Every target of a route is normally a member of its pool: the balancing strategy spreads traffic across all of them, and a retry moves to whichever untried target comes next. Sometimes a target is not meant to share the load at all. It is the remedy for one kind of failure:

• an expensive, high-trust model that should answer only the requests a cheaper one refused on content policy;
• a burst-capacity provider that should be reached only when the main one is throttling, never on a 500 (which there would mean the same bad request);
• a provider with a different content policy, which is the right answer to exactly one failure and the wrong one to every other.

takes_over on a route target says which failures it is for.

How it works#

A target with takes_over is restricted:

• It is never dialled first, and the balancing strategy never picks it for ordinary traffic. A route of one ordinary target and one restricted target sends every request to the ordinary one, whatever its strategy or weights say.
• It is tried only as the successor of a failed attempt whose failure matches one of its classes, and only while the request has a retry left ([retry] max_retries, or the route’s own limits.retries). Failing over to it counts as a retry like any other.
• A target without takes_over behaves exactly as it always has. Adding the field to one target changes nothing about the others.

The classes are the failures the gateway already tells apart when it decides whether to retry:

Content refusals#

A refusal is not a transient failure: retrying it on an ordinary target sends the same request to the same kind of model. Rolter therefore never retries a refusal onto an ordinary target; only a target whose takes_over covers it is tried. A route with no such target forwards a refusal to the caller exactly as before, without reading it.

Rolter counts an upstream answer as a refusal when it is a 451, or a 400, 403 or 422 whose error says so: an error.code or error.type naming a content policy, content filter, safety or moderation failure (OpenAI’s content_policy_violation, Azure OpenAI’s content_filter), or the content_filter_result Azure OpenAI attaches. Any other client error, such as a malformed request, stays final and is forwarded unchanged, body included.

The categories a refusal carries are the provider’s error code and, for Azure OpenAI, each filter category it marked as filtered (hate, sexual, violence, self_harm, …). refusal:hate takes over only a refusal that named hate; refusal takes over every refusal.

Configuration#

# the cheap model serves all traffic; the premium one answers only what the# cheap one refused on content policy[[routes]]model = "assistant"[[routes.targets]]provider = "cheap"model = "small-model"[[routes.targets]]provider = "premium"model = "large-model"takes_over = ["refusal"]# a burst provider reached only when the main one is throttling[[routes]]model = "batch"[[routes.targets]]provider = "main"[[routes.targets]]provider = "burst"takes_over = ["throttled", "timeout"]

The same field is takes_over on a target in the routes API, on POST /api/v1/routes/{route_id}/targets and in the target list of PUT /api/v1/routes/{id}. The routes API returns it on every target, as an empty list for an ordinary one.

A route always needs an ordinary target#

A route whose every target sets takes_over could never dial anything. Rolter refuses one rather than discovering it at request time: rolter check and gateway startup reject the file, the seed import refuses it before writing anything, and the API refuses a request that would leave a route that way, whether it adds the last target, replaces the list or deletes the last ordinary target. Each refusal names all_targets_restricted.

Seeing why a request reached a fallback#

Every request-log row records the failure that sent the request on to the target that answered it, in fallback_reason: refusal:content_policy_violation, throttled and so on, or an empty string when the first dial answered. It is returned on each row of GET /api/v1/analytics/invocations, so a request that reached the expensive target because of a content refusal can be told from one that reached it because of load. The column comes from clickhouse/023_fallback_reason.sql; see Upgrading.

Limits#

• Weighted variants honour takes_over on their targets the same way, across the whole fallback chain.
• Realtime (WebSocket) sessions never use a restricted target: their candidates are ordered before any attempt fails.
• A content refusal returned with a 200 (a finish_reason of content_filter in a completed response) is an answer, not a failure, and is not taken over.