Skip to main content

A route on the adaptive strategy does not use its static target weights. It scores each target from live signals and sends traffic to the best one, holding a slice back to keep the scores honest. One deployment-wide policy governs every such route.

Monitor live route scores under Adaptive Routing → Telemetry, then tune the global policy under Adaptive Routing → Settings.

The dashboard shows the current report from every gateway balancing an adaptive route: whether the blend is active, how decisions split between the blend, exploration, and fallback modes, and the latency, cost, and in-flight signals behind each target score. Reports are current state rather than a time series. The decision bar has a key with each mode’s swatch and count, and on a gateway that is blending, the target with the highest score (higher is preferred) carries a Preferred badge. A gateway reports every 15 seconds, and the control plane hides samples older than 60 seconds so a stopped node cannot look healthy.

The blend#

Three signals are scored in [0, 1] and combined as a weighted sum:

SignalWhat it measures
latency_weightobserved latency — how strongly it pulls traffic
cost_weightper-token cost — how strongly it pulls traffic
load_weightin-flight load — how strongly it pushes traffic away

Only the ratio between the weights matters. Doubling all three changes nothing. Each is bounded to 0..=100, not because the blend needs the range, but so a typo cannot make one signal numerically dominant. The dashboard shows each weight’s share of the total alongside the raw number for exactly that reason.

An all-zero blend is rejected. It would not stop adaptive routing — it would turn the strategy into a random balancer, which is a far less obvious thing to read off a dashboard than a switch in the off position. At least one weight must be positive.

Exploration#

exploration_ratio is the share of traffic sent off the current best target so its rivals’ scores stay fresh. Without it, a target that had one bad minute is never measured again. It is capped at 0.5 — the gateway clamps above that, so the API refuses the value instead of quietly reinterpreting it. The settings screen takes it as a percent (5 for 0.05, capped at 50) and sends the fraction.

min_samples is the warm-up: how many requests a route must have served, across all its targets, before the blend engages. While latency_weight is above zero, the blend also needs latency samples for at least two targets. Until then the route is served by the pipeline fallback stack, which keeps learning meanwhile.

The kill switch#

enabled = false is the intended way to stop adaptive routing. Every route on the adaptive strategy falls back to serving from its static target weights.

The read response carries affected_routes — the public model names of every enabled route on the strategy — so the blast radius is visible before the switch is flipped. The dashboard renders that list next to the switch.

A disabled policy still has to be well-formed. Switching the kill switch off does not license nonsense weights that would take effect the moment it is switched back on, so validation applies either way.

Seeing what it is doing#

The scores live in the gateway, so each gateway pushes a sample to the control plane every 15 seconds over the same internal channel it polls its config on. Nothing is written per request.

GET /api/v1/adaptive-routing-telemetry returns the current state, grouped by route with one entry per reporting gateway node:

Notes worth knowing before you build on it:

  • It is current state, not history. Each report overwrites the last; nothing accumulates. A sample older than fresh_window_secs is left out rather than served as if it were live.
  • policy is what that node is running. It can lag the stored policy for a poll interval after a change, which is how you tell a node that has not converged from one that has.
  • A route missing from the response is a route no node is currently balancing adaptively.
  • score is the weighted sum of the components below it; a signal with zero weight scores 0, while its raw value (latency_ms, cost_per_mtok, in_flight) is still reported so you can judge whether to give it weight.
  • Prometheus users get the same blended score as rolter_adaptive_routing_target_score{model,target} on the gateway’s /metrics.

When the dashboard is empty#

An empty Adaptive Routing screen has two very different causes, and the screen says which one it is:

  • The kill switch is off — the shipped default. The screen names the switch, lists the routes it is holding back, and links to the settings rather than sending you to debug a gateway that is behaving correctly. Nothing is wrong with the gateway.
  • The switch is on and nothing is reporting. Then the gateway is the place to look: a report older than fresh_window_secs is hidden, and a gateway that cannot name itself is refused by the control plane before it ever reaches this screen. That refusal is not silent — the gateway logs adaptive-routing telemetry report failed, and it warns at boot when no node identity could be resolved. Set ROLTER_NODE_ID to fix it.

API#

All three endpoints are superadmin only, policy updates are audited, and a successful update bumps config_version so the change reaches the data plane through the snapshot without a restart.