Response Cache: Exact and Semantic Reuse

Rolter can answer a request from a stored response instead of calling the upstream model. A cached answer costs no tokens and returns in milliseconds. Every response carries x-rolter-cache: HIT or MISS so a client can tell which one it got.

There are two layers, both opt-in per route and both stored in Redis, so every gateway replica shares them:

Exact: a request that is byte-for-byte identical to an earlier one (after Rolter applies its route parameters) gets the earlier response. Works for JSON and for streamed responses, which are buffered once and replayed as a stream.
Semantic: after an exact miss, Rolter embeds the conversation and looks for a recent response to a similar request on the same route. A reply is reused when the cosine similarity reaches the route’s threshold.

Without Redis, or when Redis or the embedding provider is unreachable, every request is a miss and is served normally.

Configuration#

[cache]enabled = true            # global switch; a route must also opt in[[routes]]model = "chat"# ...targets...[routes.cache]enabled = truettl_secs = 600            # optional; defaults to [cache].default_ttl_secsper_key = false           # true keeps each virtual key's entries private[routes.cache.semantic]provider = "openai"                 # a configured provider that serves /v1/embeddingsmodel = "text-embedding-3-small"threshold = 0.92                    # cosine similarity required for a hitmax_candidates = 256                # recent entries compared per request

What the semantic cache will answer#

A semantic hit returns a reply that was produced for a different request, so Rolter only compares requests that agree on everything except the wording of the conversation. Only the user and assistant turns are embedded and compared by similarity. Every other field of the request must match exactly before a stored reply is even considered:

Response mode: a streamed (stream: true) request is only answered from a streamed reply, and a JSON request only from a JSON reply. stream_options must match too.
Instructions: OpenAI system and developer messages, and the Anthropic top-level system. "Answer in German" and "Answer in Japanese" never share a reply.
Tools and output format: tools, tool_choice and response_format, including the full JSON schema.
Generation parameters: temperature, top_p, max_tokens, n, stop, seed and any other field in the request, including ones Rolter does not recognise.
The model and the API dialect: an OpenAI request never gets an Anthropic reply, or the reverse.

Only user and metadata are ignored, since they identify the caller and do not change the answer. Use per_key = true to keep entries private to a virtual key.

Supported request shapes#

Endpoint	Semantic lookup when…
`/v1/chat/completions`	every message is `system`, `developer`, `user` or `assistant`, with text-only content
`/v1/messages`	every message is `user` or `assistant` with text-only content, and `system` is text
`/v1/completions`	`prompt` is a single string

Every other shape skips the semantic lookup entirely, and no embedding call is made for it. Instead of guessing, Rolter sends the request upstream. The exact cache still applies to these requests:

images, audio, files, documents or any other non-text content;
a conversation that already contains tool calls or tool results (tool_calls, tool messages, tool_use / tool_result blocks). Offering tools is fine; replaying a tool exchange is not;
a batch or token-id prompt;
/v1/embeddings, /v1/audio/*, /v1/images/* and /v1/rerank, where the output has to match the exact input;
/v1/responses, which is not response-cached.

Note

Upgrading from a release without these rules retires every semantic entry the older gateway stored. Those entries were written without the compatibility rules above, so Rolter no longer reads them, and they expire on their TTL. Exact entries are unaffected.

Policies on cached responses#

A cached response goes through exactly the same post-response policy as a live one: output guardrails, post_response plugins, then the PII sanitizer’s response leg. That holds for the first response on a cached route and for every exact and semantic hit.

The cache stores the provider’s response before any of those run. So:

a guardrail or plugin that masks or rewrites a completion does so on every delivery, and the stored entry keeps the original;
a rule or plugin added by a hot reload applies to entries cached before it existed, and a blocking one refuses them;
a fail-closed plugin that cannot be reached refuses a cache hit just as it refuses a live response;
a PII restore uses the restoration token of the request being answered, so a cached entry never carries one caller’s values to another.

Turning the cache on never weakens a configured policy. It only saves the upstream call.

Choosing a threshold#

Similarity is a statement about wording, not about the correct answer. In domains where a small change in wording changes the answer (numbers, names, dates, negations), keep the threshold high or use the exact cache only. The semantic layer is bounded by max_candidates, so each lookup costs one embedding call and a scan of at most that many recent entries.