Realtime

/v1/realtime is a WebSocket relay compatible with the OpenAI Realtime API. Rolter authenticates the request and selects an upstream before accepting the client’s WebSocket upgrade, then pins that upstream and its provider key for the life of the session.

Connect#

The gateway’s GET /openapi.json describes this upgrade as GET /v1/realtime, including its required model query parameter, the 101 response and the HTTP admission failures returned before a session is accepted.

Send the usual gateway virtual key and the public route model as a query parameter:

wss://gateway.example.com/v1/realtime?model=gpt-realtime

Authenticate with Authorization: Bearer <virtual-key> the same way you would for /v1/chat/completions. The model value selects a route the same way it does everywhere else in the gateway.

What is relayed#

Text, binary audio and WebSocket control frames are relayed in both directions without application-level buffering, including session.update, input_audio_buffer.*, response.* and function-call events.

If the upstream connection drops, the client must reconnect. Rolter does not fail a live session over to another target, because replaying audio or tool events mid-session is not safe to do automatically.

Budgets, rate limits and usage#

A realtime session meets the same budgets and rate limits as an HTTP request, on the same key, project, team and org scope.

Opening a session is checked before any upstream is dialled. A refused upgrade gets the HTTP status and error body a chat request would:

Opening a session counts as one request against rpm, and a 429 carries Retry-After.

While the session runs, rolter meters it per response. Each response.done event reports the tokens that response used, and rolter turns it into one row in the request log, priced and attributed like any other request, so realtime spend appears in cost tracking and analytics. Its cost is added to your budgets and its tokens to your tpm limits every usage_flush_secs seconds (15 by default), not only when the session closes. A session that stays open all day is accounted as it goes.

Audio and text tokens are priced apart. A response.done names the audio share of its input and output tokens, and when the model’s price row sets audio_input_per_mtok, audio_output_per_mtok or cached_audio_input_per_mtok those tokens are charged at them; a row without them prices every token at the text rates. See Audio and text tokens.

Input-audio transcription is metered as its own model. With input_audio_transcription on, the upstream runs a second model (whisper-1, gpt-4o-transcribe) over the caller’s audio and reports its usage on conversation.item.input_audio_transcription.completed. Each such event becomes one more row in the request log, and its cost is added to your budgets and its tokens to your tpm limits with the session’s turns. The row’s model is the transcription model the session named in session.update (session.audio.input.transcription.model, or session.input_audio_transcription.model in the beta shape) and it is priced from that model’s own price row:

• A model billed by tokens (gpt-4o-transcribe) is priced from the event’s token usage, with its audio input at audio_input_per_mtok when set.
• A model billed by the minute (whisper-1) is priced from the event’s duration at audio_input_per_minute.
• A model with no price row, a duration with no per-minute rate, or a session that never named a transcription model is logged unpriced = 1 with no cost.
• An event that carries no usage is logged usage_unknown = 1, unknown rather than free.

A full tpm window does not close a live session; it refuses the key’s next session or request, as it would for HTTP.

When a budget runs out mid-session, rolter closes the session. This happens at the next flush after the budget is spent, whether this session or other traffic spent it. The client receives a Realtime error event first:

{  "type": "error",  "event_id": "event_rolter_…",  "error": {    "type": "insufficient_quota",    "code": "insufficient_quota",    "message": "budget exceeded for Org 'acme' (limit $500.00)",    "param": null  }}

It then receives a close frame with code 1008 (policy violation). A session can overspend by at most one flush window, plus the response in flight when the budget ran out; lower usage_flush_secs to tighten that.

When the key is revoked mid-session, rolter closes the session too. Every flush tick looks the session’s virtual key up again in the current configuration. A key that was disabled, has expired or was deleted, a models list that no longer names the session’s model, a route the key can no longer reach, and a providers list that no longer includes the provider the session is connected to all end the session, so revoking access is not left waiting on max_session_secs. The client receives a Realtime error event with code invalid_api_key (key gone), model_not_allowed, route_not_allowed or provider_not_allowed, then a close frame with code 1008. The bound is one flush interval, usage_flush_secs (one second when that is 0), plus the time for the change to reach the gateway.

When the key is moved or its limits change mid-session, the same tick applies the change to the open session rather than leaving it on what it was opened with:

• Budgets, rate limits and prices edited while a session is open apply from its next tick. A budget lowered below what its scope already spent closes the session with insufficient_quota.
• A key moved to another team or project in the same organization keeps its session. Later turns are logged and charged under the new team and project, and a spent budget there closes the session.
• A key moved to another organization, or to a project whose pre_upstream plugins differ from the old one’s, closes the session with an error event of code key_scope_changed and a 1008 close. The client should open a new session, which is admitted under the key’s new scope.
• A change to any other key leaves the session alone.

A response cut short (the client leaves, the upstream drops, or the session is closed mid-answer) never reports its usage, although the provider bills what it generated. Its row is written with usage_unknown = 1 so the gap is visible rather than counted as free. See Withheld responses for how to query it.

Gateway restarts#

When a gateway receives SIGTERM (a rolling deploy, a scale-down) or Ctrl-C, it closes every live session with WebSocket close code 1001 (going away) and refuses new ones with 503. Reconnect, and the load balancer in front of the fleet sends you to a gateway that is still running.

Before the process exits, each session’s usage so far is charged to budgets and tpm, including turns still waiting for the next usage_flush_secs, and its request-log rows are queued. A response in flight at that moment is logged with status 503 and usage_unknown = 1. The gateway waits up to 10 seconds for this, then up to 5 more seconds for the request-log writer, the health-event writer and the budget and tpm recording queue to flush what they hold, so give the process a termination grace period of at least 15 seconds on top of your longest HTTP request. If ClickHouse or Redis is unreachable at that moment, the rows and charges still queued are lost after those 5 seconds.

Current limits#

• WebRTC/browser ephemeral-token handoff is not exposed. Only the WebSocket-first flow is supported today.
• No failover. A dropped upstream ends the session; there is no session-affinity retry the way HTTP routes get.
• Content policy is narrower than on chat. Built-in guardrails, the guardrail webhook and pre_upstream plugins check client text events (conversation.item.create, session.update instructions, response.create) and, for guardrails, the model’s text and transcript output. A blocked event is not forwarded and the client receives an error event with code guardrail_blocked (or plugin_blocked); the session stays open, and a blocked response is cancelled. Output is checked as it streams, so the part of a match before the delta that completes it can already have been delivered. The PII sanitizer and pre_route/post_response plugins do not run on realtime events (a fail_closed sanitizer refuses the session with sanitizer_unsupported_on_realtime), audio itself is not inspected, and a binary frame is refused while a policy applies. See Content policy for the detail.
• Audio rates need the split. An audio rate applies only to the tokens a response.done names as audio. An upstream that reports totals alone is priced at the text rates.
• A transcription model is named by the session. A session that turns transcription on in its defaults, without a session.update or a session.updated naming the model, has its transcription logged unpriced.

[realtime] configuration#

The gateway’s [realtime] block holds per-process resource limits, which bound what one gateway will hold rather than what a tenant may spend, and the metering cadence:

• max_connections (default 1000) — concurrent sessions this gateway process admits
• max_session_secs (default 3600) — hard session-duration limit
• idle_timeout_secs (default 300) — closes a session when neither side sends a frame
• usage_flush_secs (default 15) — how often finished responses are logged, charged to budgets and tpm, and followed by a budget check. 0 does this after every response instead of on a timer, and re-checks the budgets every second while the session is quiet

Set any of the first three to 0 to disable that limit. Metering itself cannot be switched off. See Configuration for where [realtime] lives in rolter.toml.