> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Cache-Aware Routing: Maximize KV Cache Reuse in Rolter > Learn how Rolter's cache-aware strategy maximizes KV cache reuse across vLLM replicas, reducing time-to-first-token on prefix-heavy workloads. When you run multiple vLLM replicas behind a gateway, the engine's KV cache is one of the most valuable resources you have. vLLM automatically reuses the cached attention tensors for any shared prompt prefix — system prompts, few-shot examples, conversation history — but that only helps if the next matching request lands on the **same** replica. A naive round-robin scatters requests across replicas and destroys this locality, forcing each engine to recompute prefixes it has already processed. Cache-aware routing solves this. ## The problem: cache locality vs. load distribution Consider a fleet of two vLLM replicas serving a chatbot with a 2 000-token system prompt. Under round-robin: - Request 1 → replica A computes and caches the system prompt - Request 2 → replica B computes and caches the same system prompt - Request 3 → replica A, warm cache ✅ - Request 4 → replica B, warm cache ✅ In practice, sessions aren't strictly alternating, so you end up with both replicas burning compute on the same prefix repeatedly. As the fleet scales, the problem compounds. Cache-aware routing keeps related requests together, so each replica builds a deep, reusable cache rather than a shallow, scattered one. --- ## How cache-aware routing works Rolter's `cache_aware` strategy maintains a **per-target byte trie** of the prompts each target has served. When a new request arrives: 1. Rolter scores every target by computing the **fraction of leading prompt bytes** already present in that target's trie. The prompt is the conversation itself — each turn's role and text, in order — not the JSON request, so two unrelated questions to the same model do not look alike just because both are chat requests. 2. If the best-matching target's score meets or exceeds the threshold (default `0.5` — at least half the prompt prefix is cached), Rolter **pins the request** to that target — unless that target is overloaded. 3. If no target clears the threshold, or the pinned target is overloaded, Rolter routes to the **least-loaded target** (the least-warmed one when no load is known). 4. After forwarding, Rolter records the served prompt in the winning target's trie, keeping the affinity map up to date. A warm target counts as overloaded when it has more than 2 requests in flight beyond the least-loaded target **and** more than 1.5 times its load. Only targets that can take the request are compared: an unhealthy, cooling-down or circuit-broken replica is left out, so its empty queue does not make every other replica look busy. A short queue on the replica that holds the cache is worth waiting in; a long one is not, so a burst of requests sharing one system prompt spreads across the fleet instead of queueing on the first replica that served it, while each conversation's turns keep landing where their prefix is cached. The trie is capped at a node ceiling (default 1 million nodes) with LRU eviction: when the cap is reached, the least-recently-inserted prompt is pruned, but shared prefix nodes that are still referenced by other prompts survive. `cache_aware` is approximate — Rolter tracks prompt byte prefixes in its own trie, not vLLM's internal block state. For a vLLM fleet that publishes KV events, use `precise_cache_aware` below. --- ## Configuration Set `strategy = "cache_aware"` on the route. No additional parameters are required. ```toml rolter.toml [[providers]] name = "vllm-1" kind = "openai_compatible" api_base = "http://localhost:8001" [[providers]] name = "vllm-2" kind = "openai_compatible" api_base = "http://localhost:8002" [[routes]] model = "llama" strategy = "cache_aware" [[routes.targets]] provider = "vllm-1" model = "meta-llama/Llama-3.1-8B-Instruct" [[routes.targets]] provider = "vllm-2" model = "meta-llama/Llama-3.1-8B-Instruct" ``` Your application requests the model as `"llama"` and Rolter handles all the routing decisions transparently. --- ## The `pipeline` strategy: adding load and weight signals `cache_aware` optimizes purely for prefix affinity. If you also need to account for in-flight load and capacity-weighted targets, use the `pipeline` strategy instead. It runs a three-stage selection: 1. **Filter** — drop targets that are ineligible (unhealthy or on cooldown) 2. **Score** — combine static weight \+ in-flight load \+ prefix-cache affinity into a weighted sum for each target 3. **Argmax** — pick the highest-scoring target, with random tiebreaking Session affinity via `x-session-id` pins repeat requests to their last-served target within a TTL window, layered on top of the cache-affinity scoring. ```toml rolter.toml [[routes]] model = "llama-pipeline" strategy = "pipeline" [[routes.targets]] provider = "vllm-1" model = "meta-llama/Llama-3.1-8B-Instruct" weight = 2 [[routes.targets]] provider = "vllm-2" model = "meta-llama/Llama-3.1-8B-Instruct" weight = 1 ``` The `weight = 2` on `vllm-1` tells the pipeline scorer to bias toward that target — useful when one replica has more GPU memory or headroom. Enable `[metrics_scrape]` to pull each engine's scheduler queue depth into Rolter's load view. The `pipeline` strategy uses this data alongside in-flight counts, giving you the most accurate load signal available without any changes to the engine. --- ## When to use cache-aware routing Cache-aware routing delivers the biggest benefit when your prompts share a long common prefix: Every request starts with the same multi-paragraph system prompt. Cache-aware routing keeps those requests on the same replica, so the system prompt is computed once and reused for all subsequent calls. Prompts that embed the same set of examples before the user query benefit from prefix affinity in the same way as system prompts. Retrieval-augmented prompts that inject the same documents or context chunks gain cache reuse when the retrieved context lands on the same replica repeatedly. Each turn appends to the previous conversation history. Cache-aware routing (or `consistent_hash`) keeps the conversation on one replica so the growing context is served from cache rather than recomputed. --- ## Precise vLLM and LMCache modes `precise_cache_aware` subscribes to each target's vLLM V1 ZMQ KV-event stream, maintains a bounded exact residency index, and scores the resident leading block fraction. Requests must send tokenizer-consistent comma-separated ids in `x-rolter-vllm-token-ids`. Missing ids and stale, malformed, or sequence-gapped streams fall back safely to least-load routing. ```toml rolter.toml [providers.kv_events] endpoint = "tcp://vllm-1:5557" topic = "kv-events" [[routes]] model = "llama" strategy = "precise_cache_aware" ``` `lmcache_aware` polls `[providers.lmcache]` HTTP endpoints for `{ "occupancy": 0..1, "cache_available": bool }` and prefers available targets with more free cache capacity. Failed and stale controllers are neutral, preserving normal routing. Both endpoints go through the [egress policy](/configuration/egress): a denied `lmcache.endpoint` or `kv_events.endpoint` makes the config fail to load, and the lmcache poll is checked again on every refresh. The telemetry source is set per provider: in the provider sheet of the dashboard (the **KV events source** and **LMCache source** sections, each behind its own switch), as `[providers.kv_events]` or `[providers.lmcache]` in `rolter.toml`, or as the `kv_events` and `lmcache` fields of a provider created or updated through the control-plane API (`POST` / `PUT` on `/api/v1/orgs/{org_id}/providers`, see [Providers](/api/providers)). Leave a number blank in the sheet to use its default; turning a section's switch off and saving removes the source. A provider with no source makes the two strategies route to the least-loaded target when every target is such a provider. The strategy pickers show this caveat under the two strategies and link back to this page when your deployment has a [documentation host](/deployment/documentation-links) configured.