Rolter emits distributed traces via OpenTelemetry OTLP (gRPC or HTTP), which means you can send spans to any OTel-compatible backend without changing any Rolter code — just set environment variables. Rolter also propagates trace context both inbound (from your clients) and outbound (to upstream engines), so a single trace can span your entire LLM request path from client to model.
Tracing and request logging are independent features. You can use one, both, or neither depending on your observability setup.
Supported backends#
All of these accept standard OTLP, so the configuration pattern is the same regardless of which backend you use.
Configuration via environment variables#
Rolter reads the standard OpenTelemetry SDK environment variables. You don’t add anything to rolter.toml for tracing — set these in your process environment, systemd unit, Docker Compose file, or Kubernetes deployment:
| Variable | Purpose |
|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | OTLP receiver URL, e.g. http://otel-collector:4317 for gRPC. Setting this turns trace export on; unset keeps tracing stdout-only. |
OTEL_EXPORTER_OTLP_PROTOCOL | grpc (default, port 4317) or http/protobuf (port 4318) — match your collector’s receiver |
OTEL_EXPORTER_OTLP_HEADERS | Authentication headers for managed backends (comma-separated key=value) |
OTEL_SERVICE_NAME | Service name that appears on spans in your backend (default rolter) — use something like rolter-gateway |
Rolter exports traces over OTLP. Metrics are exposed on the Prometheus /metrics endpoint; point your collector’s Prometheus receiver at it (see the topology below) to bring metrics into the same pipeline.
Quick-start examples#
Trace context propagation#
Rolter participates in distributed tracing on both sides of the gateway:
Inbound context continuation
When a client sends a request with W3C traceparent / tracestate headers (or b3 headers), Rolter reads them and continues the existing trace. The gateway span becomes a child of the caller’s span — ideal for applications that already instrument their HTTP clients.
Request ID correlation
Rolter takes the caller’s request id from x-request-id (or the header set as the request ID header) and generates a unique one when none is sent. The id is echoed in that response header and stored on the request’s log row, which also carries the trace id. It is also recorded on the gateway.request span as the rolter.request.id attribute, so a request id a client quotes leads straight to its trace, even without request logging.
Outbound context injection
Rolter injects the active trace context into every upstream request it makes. If your self-hosted engine (vLLM, SGLang, TGI) exports its own spans, they join the same distributed trace automatically — no glue code required.
Browser tracing for the dashboard#
The dashboard can start its own trace in the browser, so a slow screen and the request behind it are one trace rather than two unrelated stories. It is off by default.
The dashboard is built ahead of time and served as static assets by the control
plane, so its collector endpoint cannot be baked in at build time. The control
plane injects it into the served HTML instead, as a window.__ROLTER_CONFIG__
block. Point it at an OTLP/HTTP endpoint:
This is not OTEL_EXPORTER_OTLP_ENDPOINT. That variable configures the
gateway’s own exporter, which speaks gRPC from inside your network. A browser
needs an OTLP/HTTP endpoint reachable from the user’s machine, so the two
are almost never the same URL.
With no endpoint set, the dashboard loads no tracing SDK at all — nothing is
downloaded, nothing is exported, and no request headers change. Turning it on
makes the dashboard emit document-load, user-interaction and fetch spans, and
send traceparent on its API calls so those spans join the gateway’s.
If you serve the dashboard from a different origin than the control plane, the
browser will not send traceparent cross-origin until that header is allowed
by CORS, and the dashboard’s spans will not join the gateway’s. The default
deployment serves both from the control plane, where this does not arise.
What’s inside a gateway span#
Rolter emits one span per pipeline stage, so a slow request is attributable to a named stage rather than merely slow. A trace of a single chat completion looks like this:
queue.wait measures only the time the request spent waiting for a provider
worker — not the wait plus the upstream call — so queue pressure is
distinguishable from a slow provider.
GenAI semantic conventions#
Inference spans follow the OpenTelemetry GenAI semantic conventions, so a conformant backend’s built-in GenAI views work without extra configuration. Each upstream call carries:
| Attribute | Value |
|---|---|
gen_ai.operation.name | chat, text_completion, embeddings, generate_content |
gen_ai.provider.name | the provider serving the request |
gen_ai.request.model | the model asked for |
gen_ai.response.model | the model the provider reports it served |
gen_ai.response.id | the provider’s own id for the call — the join key to their record of it |
gen_ai.response.finish_reasons | why generation stopped — stop, length, end_turn, max_tokens … |
gen_ai.output.type | text, image or speech, where the modality is expressible |
gen_ai.usage.input_tokens / output_tokens | token counts, recorded once the response completes |
gen_ai.request.encoding_formats | embeddings only — how the caller asked for the vectors (float, base64) |
gen_ai.embeddings.dimension.count | embeddings only — the width of the vectors that came back |
error.type | on failure, the status class — never the provider’s message |
The span is named {operation} {model} (for example chat gpt-4o), and span
kind is set: server for the inbound request, client for the upstream call.
gen_ai.system is still emitted alongside gen_ai.provider.name, since the
rename is recent and backends are mid-transition.
Rolter also emits the GenAI server metric family —
gen_ai.server.request.duration, gen_ai.server.time_to_first_token and
gen_ai.server.time_per_output_token — which almost no SDK instrumentation
produces, because little else sits where a gateway sits. Per-provider
time-to-first-token is exactly the signal a routing product should expose.
Three routes have no answer in the spec and deviate deliberately:
/v1/rerankhas no matching operation value, so it reports a Rolter-localrerankrather than borrowing an unrelated one and landing reranks in an embeddings dashboard./v1/audio/transcriptionsand/v1/audio/translationsreportgenerate_contentwithgen_ai.output.type=text; the input modality (audio) is not expressible today./v1/realtimegets no GenAI operation at all — a websocket session does not fit the request/response span shape.gen_ai.response.finish_reasonsis typed as an array by the spec, but the tracing layer Rolter builds on has no array field type, so a response with several choices records them comma-joined. Values stay provider-native (end_turnandstop_sequenceare not folded intostop), because collapsing them would lose the distinction they exist to make.gen_ai.request.encoding_formatsis likewise an array, and is joined the same way. Its value set isfloatandbase64, so it stays low cardinality.gen_ai.embeddings.dimension.countis read from the vector that came back, not from a requesteddimensionsthe provider is free to ignore. Abase64response has no array to measure without decoding it, so the attribute is absent there rather than guessed.
MCP calls emit no spans yet: the gateway has no MCP request path (tracked separately), so there is nothing to instrument. MCP tool calls are recorded in ClickHouse.
The opt-in content attributes (gen_ai.input.messages /
gen_ai.output.messages) stay off. Spans carry no prompt or completion
content.
Spans never carry prompt or completion content, API keys, virtual-key plaintext, or the values of headers you forward upstream. Only structural facts — which route, which provider, cache hit or miss, how long — are recorded. Redaction stays governed by your existing log settings rather than a second policy.
Metrics over OTLP#
The same counters and gauges Rolter serves on /metrics are also exported over
OTLP when an endpoint is configured. The Prometheus endpoint is unchanged — this
is a second exporter over the same numbers, so both always agree.
Metrics are collected on the SDK’s own interval
(OTEL_METRIC_EXPORT_INTERVAL, default 60s) by reading counters the gateway has
already incremented, so exporting them adds nothing to the request path.
Trying it locally#
Rolter ships two opt-in Docker Compose overlays so you can see traces without signing up for anything:
Both wire OTEL_EXPORTER_OTLP_ENDPOINT into the gateway and control plane for
you, so bringing one up is the only step. They are mutually exclusive — each
publishes OTLP on 4317/4318.
With the SigNoz overlay, the first startup waits for the schema migrator before the collector and SigNoz query service start. This can take a few minutes on a fresh volume; the collector does not need to crash and restart to wait for its databases and tables.
This local overlay does not run SigNoz’s signozspanmapper or signozllmpricing
processors. Rolter sends canonical gen_ai.* attributes, and its request logs
remain the source of costs computed from your configured prices. SigNoz views
that need collector-generated signoz.gen_ai.usage.*.cost attributes will not
have them. The overlay also omits the histogramQuantile executable function:
the bundled dogfood boards use SQL quantiles, and startup downloads no function
binary. Other SigNoz views that require that function are unsupported here.
With no OTEL_* environment set, tracing is off and costs nothing: no
exporter is built, no trace context is parsed beyond what the request log
already needed, and the request path is unchanged.
Recommended topology#
For production setups, deploy an OpenTelemetry Collector as an intermediary between Rolter and your backends. This decouples your gateway from vendor specifics and lets you fan out to multiple backends or change destinations without restarting Rolter.
The Collector can also scrape Rolter’s /metrics endpoint and your upstream engines’ /metrics endpoints in the same deployment, keeping all telemetry paths consistent.
End-to-end traces with vLLM#
If you run vLLM with --otlp-traces-endpoint pointing at the same OpenTelemetry Collector as Rolter, engine-level prefill and decode spans will appear as children of Rolter’s gateway span in the same distributed trace. You get a complete picture — from the client request, through gateway routing, down to individual inference steps — in a single trace view.
Point OTEL_EXPORTER_OTLP_ENDPOINT for Rolter at the same collector and the spans join automatically via the injected traceparent header.