Every route in Rolter picks exactly one load-balancing strategy. The strategy runs on every incoming request and returns the target that should handle it. Choosing the right strategy for your workload can meaningfully reduce latency and cost.
Strategy overview#
| Strategy | Description | Best for |
|---|---|---|
round_robin | Sequential rotation through targets | Homogeneous pools, stateless workloads |
random | Uniform random selection | Simple pools where all targets are equivalent |
power_of_two | Pick the less loaded of two randomly chosen targets | Pools with variable request durations |
consistent_hash | Hash ring keyed by session ID or prompt | Multi-turn chat, sticky sessions, KV cache reuse |
cache_aware | Approximate prefix-affinity routing to warm targets | Shared system prompts, few-shot examples, RAG |
weighted | Smooth weighted round-robin by target weight | A/B traffic splits, capacity-based routing |
pipeline | Composable filter → scored → argmax selection | Blending weight, load, and cache signals together |
cheapest | Lowest catalog price for the target’s upstream model; load breaks ties | Cost-sensitive routing across mixed-price providers |
fastest | Lowest observed request latency (EWMA); load breaks ties and carries cold routes | Heterogeneous pools where speed matters most |
precise_cache_aware | Exact resident-prefix score from vLLM KV events; load fallback | vLLM fleets with KV-event publishing |
lmcache_aware | Cache availability and free-capacity score from controller occupancy | LMCache-backed fleets |
adaptive | Weighted blend of latency, cost and load, gated by a global kill switch | Mixed price and latency pools, tuned by the gateway |
lora_aware | Prefer a target that already holds the requested LoRA adapter resident | Fleets serving many adapters over shared base weights |
predicted_latency | Rank by what this request is modelled to cost, from queue depth and prompt size | Heterogeneous pools with variable prompt sizes and deep queues |
Strategies in detail#
round_robin — sequential rotation
Rolter cycles through the target list in order, sending the first request to target 0, the second to target 1, and so on. Zero state, perfectly predictable distribution.
random — uniform random selection
Each request independently picks a target at random with equal probability. Useful when you want simple distribution without sequential ordering.
power_of_two — pick the less loaded of two
Rolter samples two targets at random and forwards to whichever has fewer in-flight requests. This approximates optimal load distribution with O(1) overhead and works well when request durations vary widely.
Enable [metrics_scrape] so Rolter folds each engine’s scheduler queue depth into the load view — this makes power_of_two aware of queued requests, not just in-flight ones.
consistent_hash — sticky sessions via hash ring
Rolter places each target on a hash ring at 100 points and hashes the incoming x-session-id header (falling back to the whole conversation, so requests that share a long system prompt but ask different questions, or attach different images or tool results, still spread) to select a target. The same session or prompt always lands on the same target, giving the upstream engine the best chance of serving it from a warm KV cache. If a target is removed, only the keys that mapped to it are reshuffled.
Send a session identifier in the request header to activate session affinity:
cache_aware — approximate prefix affinity
Rolter maintains a per-target trie of prompts it has served. For each incoming request, it scores targets by the fraction of leading prompt bytes already present in their trie. If the best match meets or exceeds the threshold (default 0.5), Rolter pins the request to that target. Otherwise it spreads to the least-warmed target.
This is the recommended strategy for self-hosted vLLM fleets with shared system prompts. See Cache-Aware Routing for a full explanation.
weighted — smooth weighted round-robin
Targets receive traffic proportional to their weight value. Rolter uses smooth weighted round-robin, so the distribution is both proportional and evenly interleaved — a 3:1 split doesn’t batch three requests together, it intersperses them.
The above sends 75 % of requests to vllm-1 and 25 % to vllm-2.
pipeline — composable multi-signal selection
The pipeline strategy runs three stages: filter (drop ineligible targets), score (combine static weight + in-flight load + prefix-cache affinity into a weighted sum per target), and argmax (pick the highest scorer, breaking ties randomly). Session affinity via x-session-id pins repeat requests to their last-served target within a TTL window.
This is the most powerful strategy and the extension point for future cost, latency, and KV-cache scorers.
Session affinity with x-session-id#
Several strategies (consistent_hash, pipeline, and cache_aware) read the x-session-id request header to pin a session to a specific target. Send any stable string that identifies the logical session — a user ID, conversation ID, or a composite:
When the header is absent, these strategies fall back to hashing the prompt text.
predicted_latency — model this request's cost, don't average past ones
fastest ranks on a per-target latency average. That is the right signal when every request is the same size, and the wrong one otherwise: a target whose average is high because it happened to serve the long prompts looks slow even when it is the emptiest box in the fleet, and a target with a deep queue looks fast right up until the queue is what you join.
predicted_latency learns a small model per target instead:
The three coefficients are learned online from completed requests. A short prompt arriving while every box is busy goes to the one that shrugs off its queue; a long prompt arriving into idle boxes goes to the one that prefills fastest.
A target that has not completed 8 requests yet predicts nothing, and Rolter reads that as unknown, not slow. Switching a route to predicted_latency therefore moves no traffic on its own — the route behaves like a least-load pipeline until the models have evidence. Only successful requests teach, so a fast-failing target cannot train itself into looking cheap.
Choosing a strategy#
Use this table to match your workload to a starting strategy:
| Situation | Recommended strategy |
|---|---|
| Managed API (OpenAI, Anthropic), homogeneous pool | round_robin |
| Many small, fast requests with equal-spec replicas | random |
| Mixed long and short requests, load can spike | power_of_two |
| Multi-turn chat where session stickiness matters | consistent_hash |
| vLLM fleet with a shared system prompt or RAG context | cache_aware |
| Canary deployment or capacity-weighted split | weighted |
| Self-hosted fleet where you want all signals combined | pipeline |
| Mixed-price providers, minimize spend | cheapest |
| Heterogeneous pool, minimize latency | fastest |
You can enable [metrics_scrape] to let Rolter pull each engine’s queue depth from its Prometheus metrics endpoint. Load-aware strategies (power_of_two and pipeline) automatically incorporate this data into their decisions.