Cache-Aware Routing: Maximize KV Cache Reuse in Rolter

When you run multiple vLLM replicas behind a gateway, the engine’s KV cache is one of the most valuable resources you have. vLLM automatically reuses the cached attention tensors for any shared prompt prefix — system prompts, few-shot examples, conversation history — but that only helps if the next matching request lands on the same replica. A naive round-robin scatters requests across replicas and destroys this locality, forcing each engine to recompute prefixes it has already processed. Cache-aware routing solves this.

The problem: cache locality vs. load distribution#

Consider a fleet of two vLLM replicas serving a chatbot with a 2 000-token system prompt. Under round-robin:

• Request 1 → replica A computes and caches the system prompt
• Request 2 → replica B computes and caches the same system prompt
• Request 3 → replica A, warm cache ✅
• Request 4 → replica B, warm cache ✅

In practice, sessions aren’t strictly alternating, so you end up with both replicas burning compute on the same prefix repeatedly. As the fleet scales, the problem compounds. Cache-aware routing keeps related requests together, so each replica builds a deep, reusable cache rather than a shallow, scattered one.

How cache-aware routing works#

Rolter’s cache_aware strategy maintains a per-target byte trie of the prompts each target has served. When a new request arrives:

1. Rolter scores every target by computing the fraction of leading prompt bytes already present in that target’s trie. The prompt is the conversation itself — each turn’s role and text, in order — not the JSON request, so two unrelated questions to the same model do not look alike just because both are chat requests.
2. If the best-matching target’s score meets or exceeds the threshold (default 0.5 — at least half the prompt prefix is cached), Rolter pins the request to that target — unless that target is overloaded.
3. If no target clears the threshold, or the pinned target is overloaded, Rolter routes to the least-loaded target (the least-warmed one when no load is known).
4. After forwarding, Rolter records the served prompt in the winning target’s trie, keeping the affinity map up to date.

A warm target counts as overloaded when it has more than 2 requests in flight beyond the least-loaded target and more than 1.5 times its load. Only targets that can take the request are compared: an unhealthy, cooling-down or circuit-broken replica is left out, so its empty queue does not make every other replica look busy. A short queue on the replica that holds the cache is worth waiting in; a long one is not, so a burst of requests sharing one system prompt spreads across the fleet instead of queueing on the first replica that served it, while each conversation’s turns keep landing where their prefix is cached.

The trie is capped at a node ceiling (default 1 million nodes) with LRU eviction: when the cap is reached, the least-recently-inserted prompt is pruned, but shared prefix nodes that are still referenced by other prompts survive.

Configuration#

Set strategy = "cache_aware" on the route. No additional parameters are required.

[[providers]]name = "vllm-1"kind = "openai_compatible"api_base = "http://localhost:8001"[[providers]]name = "vllm-2"kind = "openai_compatible"api_base = "http://localhost:8002"[[routes]]model = "llama"strategy = "cache_aware"[[routes.targets]]provider = "vllm-1"model = "meta-llama/Llama-3.1-8B-Instruct"[[routes.targets]]provider = "vllm-2"model = "meta-llama/Llama-3.1-8B-Instruct"

Your application requests the model as "llama" and Rolter handles all the routing decisions transparently.

The pipeline strategy: adding load and weight signals#

cache_aware optimizes purely for prefix affinity. If you also need to account for in-flight load and capacity-weighted targets, use the pipeline strategy instead. It runs a three-stage selection:

1. Filter — drop targets that are ineligible (unhealthy or on cooldown)
2. Score — combine static weight + in-flight load + prefix-cache affinity into a weighted sum for each target
3. Argmax — pick the highest-scoring target, with random tiebreaking

Session affinity via x-session-id pins repeat requests to their last-served target within a TTL window, layered on top of the cache-affinity scoring.

[[routes]]model = "llama-pipeline"strategy = "pipeline"[[routes.targets]]provider = "vllm-1"model = "meta-llama/Llama-3.1-8B-Instruct"weight = 2[[routes.targets]]provider = "vllm-2"model = "meta-llama/Llama-3.1-8B-Instruct"weight = 1

The weight = 2 on vllm-1 tells the pipeline scorer to bias toward that target — useful when one replica has more GPU memory or headroom.

When to use cache-aware routing#

Cache-aware routing delivers the biggest benefit when your prompts share a long common prefix:

Precise vLLM and LMCache modes#

precise_cache_aware subscribes to each target’s vLLM V1 ZMQ KV-event stream, maintains a bounded exact residency index, and scores the resident leading block fraction. Requests must send tokenizer-consistent comma-separated ids in x-rolter-vllm-token-ids. Missing ids and stale, malformed, or sequence-gapped streams fall back safely to least-load routing.

[providers.kv_events]endpoint = "tcp://vllm-1:5557"topic = "kv-events"[[routes]]model = "llama"strategy = "precise_cache_aware"

lmcache_aware polls [providers.lmcache] HTTP endpoints for { "occupancy": 0..1, "cache_available": bool } and prefers available targets with more free cache capacity. Failed and stale controllers are neutral, preserving normal routing.

Both endpoints go through the egress policy: a denied lmcache.endpoint or kv_events.endpoint makes the config fail to load, and the lmcache poll is checked again on every refresh.