POST /v1/chat/completions: OpenAI Chat Completions API

The /v1/chat/completions endpoint is fully compatible with the OpenAI Chat Completions API. Any client or SDK that already targets OpenAI works with Rolter without code changes — set base_url to your gateway address and supply a Rolter virtual key as the API key. Rolter authenticates the request, selects an upstream target based on your route configuration, optionally injects admin-configured param defaults, and forwards the request transparently.

Request#

POST /v1/chat/completions

Headers#

Authorizationstringrequired#

Bearer <virtual-key> — your Rolter virtual key.

Content-Typestringrequired#

Must be application/json.

x-session-idstring#

Optional session identifier for affinity-based routing. Pass a stable value (e.g. a user or conversation ID) to pin multi-turn requests to the same upstream replica and improve KV-cache hit rates.

Body#

modelstringrequired#

The route name defined in your rolter.toml, for example gpt-4o. Rolter looks up this route and rewrites model to the upstream model ID before forwarding. Returns 404 if no matching route exists.

messagesarrayrequired#

Array of message objects forming the conversation history.

Show Message object fields

rolestringrequired#

The speaker role: system, user, or assistant.

contentstringrequired#

The text content of the message.

streamboolean#

Set to true to receive a streaming response over Server-Sent Events (SSE). Each event carries a data: line with a JSON delta chunk; the stream terminates with data: [DONE].

Default: "false"

temperaturenumber#

Sampling temperature between 0 and 2. Higher values produce more random output. Passed through to the upstream; may be overridden or pinned by the route’s param_policy in your config.

max_tokensinteger#

Maximum number of tokens to generate. Passed through to the upstream; may be capped by the route’s params defaults.

stopstring | array#

One or more sequences at which the model stops generating further tokens.

top_pnumber#

Nucleus sampling probability mass. An alternative to temperature; most providers recommend adjusting one but not both.

Note

Any additional fields you include in the body — frequency_penalty, presence_penalty, logit_bias, tools, response_format, and others — are forwarded to the upstream as-is. Rolter only reads model and stream for its own logic.

Streaming#

When "stream": true, Rolter proxies the upstream’s Server-Sent Events stream directly to your client. Each event looks like:

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"index":0}]}data: [DONE]

Use curl’s -N flag (--no-buffer) to disable output buffering when testing streaming from the command line.

Examples#

curl http://localhost:4000/v1/chat/completions \  -H "Authorization: Bearer sk-rolter-dev" \  -H "Content-Type: application/json" \  -d '{    "model": "gpt-4o",    "messages": [      {"role": "system", "content": "You are a helpful assistant."},      {"role": "user", "content": "Hello!"}    ]  }'

curl -N http://localhost:4000/v1/chat/completions \  -H "Authorization: Bearer sk-rolter-dev" \  -H "Content-Type: application/json" \  -d '{    "model": "gpt-4o",    "stream": true,    "messages": [      {"role": "user", "content": "Hello!"}    ]  }'

from openai import OpenAIclient = OpenAI(    base_url="http://localhost:4000/v1",    api_key="sk-rolter-dev",)response = client.chat.completions.create(    model="gpt-4o",    messages=[{"role": "user", "content": "Hello!"}],)print(response.choices[0].message.content)

from openai import OpenAIclient = OpenAI(    base_url="http://localhost:4000/v1",    api_key="sk-rolter-dev",)stream = client.chat.completions.create(    model="gpt-4o",    stream=True,    messages=[{"role": "user", "content": "Hello!"}],)for chunk in stream:    delta = chunk.choices[0].delta.content    if delta:        print(delta, end="", flush=True)

Session affinity for multi-turn conversations#

Pass x-session-id to pin a conversation thread to the same upstream replica. This is especially useful with self-hosted vLLM or SGLang pools where KV-cache reuse across turns can significantly reduce latency and cost.

curl (session affinity)

curl http://localhost:4000/v1/chat/completions \  -H "Authorization: Bearer sk-rolter-dev" \  -H "Content-Type: application/json" \  -H "x-session-id: user-abc123" \  -d '{    "model": "llama",    "messages": [      {"role": "user", "content": "What is the capital of France?"}    ]  }'

OpenAI SDK (session affinity)

from openai import OpenAIclient = OpenAI(    base_url="http://localhost:4000/v1",    api_key="sk-rolter-dev",    default_headers={"x-session-id": "user-abc123"},)response = client.chat.completions.create(    model="llama",    messages=[{"role": "user", "content": "What is the capital of France?"}],)print(response.choices[0].message.content)

Param policy#

Your rolter.toml can attach default inference parameters to a route under [routes.params] (e.g. temperature = 0.0, max_tokens = 1024). The route’s param_policy then controls whether callers can override those defaults:

mode = "allow" (default) — callers may override any default except params listed in deny.
mode = "deny" — the admin default wins for every param except those listed in allow.

When an override is denied, the admin default silently replaces the caller-supplied value before the request reaches the upstream.