> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # OpenTelemetry Distributed Tracing with Rolter Gateway > Configure Rolter to export OpenTelemetry traces to Grafana, Honeycomb, Datadog, SigNoz, or Langfuse for end-to-end LLM observability. Rolter emits distributed traces via OpenTelemetry OTLP (gRPC or HTTP), which means you can send spans to any OTel-compatible backend without changing any Rolter code — just set environment variables. Rolter also propagates trace context both inbound (from your clients) and outbound (to upstream engines), so a single trace can span your entire LLM request path from client to model. Tracing and request logging are independent features. You can use one, both, or neither depending on your observability setup. ## Supported backends All of these accept standard OTLP, so the configuration pattern is the same regardless of which backend you use. ## Configuration via environment variables Rolter reads the standard OpenTelemetry SDK environment variables. You don't add anything to `rolter.toml` for tracing — set these in your process environment, systemd unit, Docker Compose file, or Kubernetes deployment: | Variable | Purpose | | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP receiver URL, e.g. `http://otel-collector:4317` for gRPC. **Setting this turns trace export on**; unset keeps tracing stdout-only. | | `OTEL_EXPORTER_OTLP_PROTOCOL` | `grpc` (default, port 4317) or `http/protobuf` (port 4318) — match your collector's receiver | | `OTEL_EXPORTER_OTLP_HEADERS` | Authentication headers for managed backends (comma-separated `key=value`) | | `OTEL_SERVICE_NAME` | Service name that appears on spans in your backend (default `rolter`) — use something like `rolter-gateway` | Rolter exports **traces** over OTLP. Metrics are exposed on the Prometheus `/metrics` endpoint; point your collector's Prometheus receiver at it (see the topology below) to bring metrics into the same pipeline. ## Quick-start examples ```bash Grafana Tempo export OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4317 export OTEL_SERVICE_NAME=rolter-gateway ``` ```bash Honeycomb export OTEL_EXPORTER_OTLP_ENDPOINT=https://api.honeycomb.io export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=YOUR_API_KEY" export OTEL_SERVICE_NAME=rolter-gateway ``` ```bash Datadog (via OTLP intake) export OTEL_EXPORTER_OTLP_ENDPOINT=http://datadog-agent:4317 export OTEL_SERVICE_NAME=rolter-gateway ``` ```bash SigNoz export OTEL_EXPORTER_OTLP_ENDPOINT=http://signoz-otel-collector:4317 export OTEL_SERVICE_NAME=rolter-gateway ``` ```bash Langfuse export OTEL_EXPORTER_OTLP_ENDPOINT=https://cloud.langfuse.com/api/public/otel export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic " export OTEL_SERVICE_NAME=rolter-gateway ``` ## Trace context propagation Rolter participates in distributed tracing on both sides of the gateway: When a client sends a request with W3C `traceparent` / `tracestate` headers (or `b3` headers), Rolter reads them and continues the existing trace. The gateway span becomes a child of the caller's span — ideal for applications that already instrument their HTTP clients. Rolter takes the caller's request id from `x-request-id` (or the header set as the [request ID header](/configuration/client-settings#request-id-header)) and generates a unique one when none is sent. The id is echoed in that response header and stored on the request's log row, which also carries the trace id. It is also recorded on the `gateway.request` span as the `rolter.request.id` attribute, so a request id a client quotes leads straight to its trace, even without request logging. Rolter injects the active trace context into every upstream request it makes. If your self-hosted engine (vLLM, SGLang, TGI) exports its own spans, they join the same distributed trace automatically — no glue code required. ## Browser tracing for the dashboard The dashboard can start its own trace in the browser, so a slow screen and the request behind it are one trace rather than two unrelated stories. It is off by default. The dashboard is built ahead of time and served as static assets by the control plane, so its collector endpoint cannot be baked in at build time. The control plane injects it into the served HTML instead, as a `window.__ROLTER_CONFIG__` block. Point it at an **OTLP/HTTP** endpoint: ```bash ROLTER_UI_OTEL_ENDPOINT=http://localhost:4318/v1/traces \ ROLTER_UI_OTEL_SERVICE_NAME=rolter-ui \ rolter control ``` This is not `OTEL_EXPORTER_OTLP_ENDPOINT`. That variable configures the gateway's own exporter, which speaks gRPC from inside your network. A browser needs an OTLP/HTTP endpoint reachable from the **user's** machine, so the two are almost never the same URL. With no endpoint set, the dashboard loads no tracing SDK at all — nothing is downloaded, nothing is exported, and no request headers change. Turning it on makes the dashboard emit document-load, user-interaction and fetch spans, and send `traceparent` on its API calls so those spans join the gateway's. If you serve the dashboard from a different origin than the control plane, the browser will not send `traceparent` cross-origin until that header is allowed by CORS, and the dashboard's spans will not join the gateway's. The default deployment serves both from the control plane, where this does not arise. ## What's inside a gateway span Rolter emits one span per pipeline stage, so a slow request is attributable to a named stage rather than merely slow. A trace of a single chat completion looks like this: ```text POST /v1/chat/completions ├─ auth ├─ route.select strategy, candidates, route ├─ guardrails.pre redacted, webhook ├─ cache.lookup hit, kind (exact | semantic) └─ upstream.request attempt, gen_ai.system, gen_ai.request.model, status ├─ queue.wait provider └─ translate.request ``` `queue.wait` measures only the time the request spent waiting for a provider worker — not the wait plus the upstream call — so queue pressure is distinguishable from a slow provider. ### GenAI semantic conventions Inference spans follow the OpenTelemetry [GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai), so a conformant backend's built-in GenAI views work without extra configuration. Each upstream call carries: | Attribute | Value | | --------------------------------------------- | -------------------------------------------------------------------------- | | `gen_ai.operation.name` | `chat`, `text_completion`, `embeddings`, `generate_content` | | `gen_ai.provider.name` | the provider serving the request | | `gen_ai.request.model` | the model asked for | | `gen_ai.response.model` | the model the provider reports it served | | `gen_ai.response.id` | the provider's own id for the call — the join key to their record of it | | `gen_ai.response.finish_reasons` | why generation stopped — `stop`, `length`, `end_turn`, `max_tokens` … | | `gen_ai.output.type` | `text`, `image` or `speech`, where the modality is expressible | | `gen_ai.usage.input_tokens` / `output_tokens` | token counts, recorded once the response completes | | `gen_ai.request.encoding_formats` | embeddings only — how the caller asked for the vectors (`float`, `base64`) | | `gen_ai.embeddings.dimension.count` | embeddings only — the width of the vectors that came back | | `error.type` | on failure, the status class — never the provider's message | The span is named `{operation} {model}` (for example `chat gpt-4o`), and span kind is set: `server` for the inbound request, `client` for the upstream call. `gen_ai.system` is still emitted alongside `gen_ai.provider.name`, since the rename is recent and backends are mid-transition. Rolter also emits the GenAI **server** metric family — `gen_ai.server.request.duration`, `gen_ai.server.time_to_first_token` and `gen_ai.server.time_per_output_token` — which almost no SDK instrumentation produces, because little else sits where a gateway sits. Per-provider time-to-first-token is exactly the signal a routing product should expose. Three routes have no answer in the spec and deviate deliberately: - **`/v1/rerank`** has no matching operation value, so it reports a Rolter-local `rerank` rather than borrowing an unrelated one and landing reranks in an embeddings dashboard. - **`/v1/audio/transcriptions`** and **`/v1/audio/translations`** report `generate_content` with `gen_ai.output.type=text`; the _input_ modality (audio) is not expressible today. - **`/v1/realtime`** gets no GenAI operation at all — a websocket session does not fit the request/response span shape. - **`gen_ai.response.finish_reasons`** is typed as an array by the spec, but the tracing layer Rolter builds on has no array field type, so a response with several choices records them comma-joined. Values stay provider-native (`end_turn` and `stop_sequence` are not folded into `stop`), because collapsing them would lose the distinction they exist to make. - **`gen_ai.request.encoding_formats`** is likewise an array, and is joined the same way. Its value set is `float` and `base64`, so it stays low cardinality. - **`gen_ai.embeddings.dimension.count`** is read from the vector that came back, not from a requested `dimensions` the provider is free to ignore. A `base64` response has no array to measure without decoding it, so the attribute is absent there rather than guessed. MCP calls emit no spans yet: the gateway has no MCP request path (tracked separately), so there is nothing to instrument. MCP tool calls are recorded in ClickHouse. The opt-in content attributes (`gen_ai.input.messages` / `gen_ai.output.messages`) stay **off**. Spans carry no prompt or completion content. Spans never carry prompt or completion content, API keys, virtual-key plaintext, or the values of headers you forward upstream. Only structural facts — which route, which provider, cache hit or miss, how long — are recorded. Redaction stays governed by your existing log settings rather than a second policy. ## Metrics over OTLP The same counters and gauges Rolter serves on `/metrics` are also exported over OTLP when an endpoint is configured. The Prometheus endpoint is unchanged — this is a second exporter over the same numbers, so both always agree. Metrics are collected on the SDK's own interval (`OTEL_METRIC_EXPORT_INTERVAL`, default 60s) by reading counters the gateway has already incremented, so exporting them adds nothing to the request path. ## Trying it locally Rolter ships two opt-in Docker Compose overlays so you can see traces without signing up for anything: ```bash Jaeger (light, traces only) docker compose -f docker/docker-compose.yml \ -f docker/docker-compose.observability.yml up # traces at http://localhost:16686 ``` ```bash SigNoz (traces, metrics and logs) docker compose -f docker/docker-compose.yml \ -f docker/docker-compose.signoz.yml up # SigNoz at http://localhost:8080 ``` Both wire `OTEL_EXPORTER_OTLP_ENDPOINT` into the gateway and control plane for you, so bringing one up is the only step. They are mutually exclusive — each publishes OTLP on 4317/4318. With the SigNoz overlay, the first startup waits for the schema migrator before the collector and SigNoz query service start. This can take a few minutes on a fresh volume; the collector does not need to crash and restart to wait for its databases and tables. This local overlay does not run SigNoz's `signozspanmapper` or `signozllmpricing` processors. Rolter sends canonical `gen_ai.*` attributes, and its request logs remain the source of costs computed from your configured prices. SigNoz views that need collector-generated `signoz.gen_ai.usage.*.cost` attributes will not have them. The overlay also omits the `histogramQuantile` executable function: the bundled dogfood boards use SQL quantiles, and startup downloads no function binary. Other SigNoz views that require that function are unsupported here. With no `OTEL_*` environment set, tracing is off and costs nothing: no exporter is built, no trace context is parsed beyond what the request log already needed, and the request path is unchanged. ## Recommended topology For production setups, deploy an OpenTelemetry Collector as an intermediary between Rolter and your backends. This decouples your gateway from vendor specifics and lets you fan out to multiple backends or change destinations without restarting Rolter. ```text Rolter gateway │ │ OTLP (gRPC/HTTP) ▼ OpenTelemetry Collector │ │ │ ▼ ▼ ▼ Grafana Honeycomb Datadog Tempo ``` The Collector can also scrape Rolter's `/metrics` endpoint and your upstream engines' `/metrics` endpoints in the same deployment, keeping all telemetry paths consistent. ## End-to-end traces with vLLM If you run vLLM with `--otlp-traces-endpoint` pointing at the same OpenTelemetry Collector as Rolter, engine-level prefill and decode spans will appear as children of Rolter's gateway span in the same distributed trace. You get a complete picture — from the client request, through gateway routing, down to individual inference steps — in a single trace view. ```bash vLLM launch with tracing python -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-3.1-8B-Instruct \ --otlp-traces-endpoint http://otel-collector:4317 ``` Point `OTEL_EXPORTER_OTLP_ENDPOINT` for Rolter at the same collector and the spans join automatically via the injected `traceparent` header.