> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Prometheus Metrics and Histograms in Rolter Gateway > Scrape Rolter's Prometheus endpoint to monitor request rates, latency histograms, upstream errors, and auth failures across models and providers. Rolter exposes a Prometheus-compatible metrics endpoint so you can plug it straight into your existing monitoring stack — no agents or sidecars required. Metrics are updated on the request path and rendered on demand when Prometheus scrapes the endpoint. ## Scraping the endpoint By default, metrics are available at `GET /metrics` on the same port as the gateway (`:4000`). Add a Rolter job to your Prometheus configuration and you're done: ```yaml prometheus.yml scrape_configs: - job_name: rolter static_configs: - targets: ['localhost:4000'] ``` ## Changing the metrics path If another service behind the same reverse proxy already owns `/metrics`, you can move Rolter's endpoint to any path you choose: ```toml rolter.toml [server] metrics_path = "/internal/metrics" ``` The `metrics_path` value must start with `/` and must not collide with Rolter's built-in routes: `/healthz`, `/v1/models`, `/v1/chat/completions`, `/v1/completions`, or `/v1/messages`. ## Available metrics ### Request traffic **Type:** Counter Total number of requests proxied by the gateway since startup. Use this as your primary throughput signal — `rate(rolter_requests_total[1m])` gives you requests per second. **Type:** Counter Requests whose caller went away before the response finished — a closed browser tab, a client-side timeout, a pod rollout. These are logged with status `499` (nginx's *client closed request*) and are deliberately never retried onto another provider, so they cost you one upstream attempt, not two. A rising rate almost always means client timeouts are tighter than your upstream latency rather than anything being broken in the gateway. Compare `rate(rolter_client_disconnects_total[5m])` against the p95 of `rolter_request_latency_ms`. **A client disconnect is never charged to a target.** Whether the caller left while its request was queued in rolter, while the upstream was working, or part-way through a streamed answer, the request is not counted in `rolter_target_requests_total` (neither as `outcome="error"` nor as `ok`) and emits no passive health event, so it does not move the target's uptime or the `provider_health_flaps` alert. This counter is where disconnects show up. An upstream that itself answers with status `499` is not a disconnect and still counts as that target's error. **Type:** Counter Responses the provider generated and billed, but that an output guardrail or a `post_response` plugin refused to deliver. The caller got a `403`. The spend was still recorded against budgets, so this is how much you are paying for answers nobody received. Blocked response-cache hits are counted too, at no cost. See [Withheld responses](/observability/logs-and-cost#withheld-responses). **Type:** Counter Streamed requests refused with `plugin_streaming_unsupported` because a fail-closed `post_response` plugin applies to them. That plugin must approve every response and cannot see a stream. Clients that hit this should retry without `"stream": true`, or the plugin should be switched to fail open. See [Streamed responses](/configuration/plugins#streamed-responses). **Type:** Gauge Requests currently in flight to upstream providers, summed across targets and sampled when you scrape. On an idle gateway this reads `0`. A floor that never comes back to zero means requests are exiting without releasing their slot, which quietly biases every load-aware routing decision — alert on `min_over_time(rolter_inflight_requests[15m]) > 0` for a gateway that does have quiet periods. **Type:** Counter Total upstream errors, broken down by error type. Incremented when an upstream returns a 4xx/5xx or a connection-level failure occurs (after all retries are exhausted). **Type:** Counter, labelled `reason` Requests refused before the gateway knew who was asking, split by why: `missing_key` (no key presented, on a deployment that requires one), `invalid_key` (a key nobody issued, or one that is disabled or expired) and `required_header_missing`. These requests leave no row in the request log, so this counter is the only place to see how many there were and why. `rolter_auth_failures_total` is the sum of the series. **Type:** Counter Total authentication failures — requests that were rejected because the presented virtual key was missing, invalid, disabled, or expired. **Type:** Counter Connections established to upstream providers. Every increment is the connection pool failing to find a connection it could reuse. Watch the ratio, not the raw number: `rate(rolter_upstream_connections_total[5m]) / rate(rolter_requests_total[5m])`. Near zero means the pool is doing its job. Approaching one means every request is paying a fresh TCP and TLS handshake, which on a cross-region provider is most of your latency — check `pool_idle_timeout` and whether the provider is closing connections between requests. **Type:** Counter Attempts that failed before a connection existed — DNS, TCP, TLS, or a rejection by the egress policy. Distinct from `rolter_upstream_errors_total`, which counts providers that answered with an error. A provider you cannot reach and a provider returning 500 need different fixes. ### Runtime saturation **Type:** Gauge (OTLP only) Tasks that are runnable and waiting for a worker thread. A sustained non-zero value means requests are queuing inside rolter itself, before any provider is called — which no upstream latency metric will ever show you. This is the metric that separates "the provider is slow" from "we are saturated". They look identical in end-to-end latency and have opposite fixes. **Type:** Counter, seconds (OTLP only) Cumulative time runtime workers spent doing work, summed across workers. Exported as time rather than a ratio so it re-aggregates correctly: derive utilisation with `rate(tokio.runtime.worker.busy.time) / tokio.runtime.workers`. **Type:** Gauge (OTLP only) Worker threads in the runtime, and tasks currently alive. `workers` is the denominator for the utilisation calculation above; a steadily climbing `tasks.alive` with flat throughput points at tasks that are never completing. The blocking-thread-pool metrics are deliberately absent. They are the part of tokio's metrics surface still gated behind `--cfg tokio_unstable`, which would mean opting every build into an API that can change on a patch release. The metrics above need no such flag. ### Provider queue These size `[queue]` (**Settings → Performance Tuning**): `workers` is how many queue workers each provider has for sending requests upstream, and `capacity` is how many more requests can wait for one. Each series carries a `provider` label, one per configured provider, and all three are exported over OTLP as well. `workers` bounds how many requests per provider are waiting for the upstream's response headers: with `workers = 8`, eight such calls can be outstanding and a ninth waits. A worker is released as soon as the headers arrive, not when the answer is complete. A non-streamed request to an upstream that sends its headers with the finished answer holds its worker for the whole generation. A streamed one (`stream: true`) usually gets its headers almost at once and then keeps generating upstream after its worker has moved on, so `workers` does not cap how many streams, or how much GPU load, a provider carries. Cap concurrent generations at the engine itself. Before 1.0 a provider's workers took turns and sent one request at a time, so if you sized `workers` against latency measured on an earlier build, measure again. **Type:** Gauge, per `provider` Requests waiting for one of the provider's queue workers, sampled when you scrape (or at the OTLP export interval). A burst shorter than the sampling interval can fall between two samples; `rolter_provider_queue_wait_ms` records every wait. A depth that stays above zero means `workers` is the bottleneck for that provider — and latency climbs with it long before `rolter_provider_queue_rejections_total` moves. Reads `0` when queueing is off. **Type:** Gauge, per `provider` Requests sent to the provider and still waiting for its response headers: at most `workers` while queueing is on, except briefly after a `[queue]` change under load, when the replaced queue's workers are still draining its buffered requests: for that window the provider can see the old `workers` plus the new `workers` at once. A request that started before the change never swaps the old settings back in, so this happens once per change. A streamed body is not counted once its headers arrive — the same point at which a queue worker is released. A floor that never returns to `0` on a quiet gateway means requests are leaving without releasing their slot. **Type:** Histogram, milliseconds, per `provider` How long each request waited for a queue worker, from arriving at the queue to a worker taking it. The same buckets as the latency histograms (1ms → 10s), so `histogram_quantile(0.95, sum by (le, provider) (rate(rolter_provider_queue_wait_ms_bucket[5m])))` sits beside the request latency p95 and shows how much of it is queueing. **Type:** Counter Requests refused because a provider's queue was full (`backpressure = "error"` or `"drop"`), and requests that gave up waiting for room with `backpressure = "block"`. Both are the end state the gauges above give you warning of. ### Redis Budgets, rate limits and the response cache each keep their own connection to Redis. When Redis cannot be reached, budgets and rate limits **fail open**: requests go through unchecked rather than the gateway going down with the counter store. These series show when that is happening and what it let through. They appear only when the gateway runs with a Redis URL. **Type:** Gauge, per `consumer` (`budgets`, `rate_limits`, `response_cache`) `1` while the consumer holds a live connection, `0` once its last connection attempt failed or its connection was lost. Each consumer connects at startup. After that the state is what the consumer's last use found: an idle consumer does not probe, so a gateway with no budgeted traffic can keep reading `0` after Redis is back until the next request uses it. For alerting, pair it with `rolter_fail_open_total`. **Type:** Counter, per `control` (`budget`, `rate_limit`) Requests an admission check let through without consulting Redis because Redis was unavailable. This is the cost exposure of an outage: `increase(rolter_fail_open_total{control="budget"}[1h])` is how many requests spent money no budget checked. Only requests that a budget or rate limit applies to are counted. **Type:** Counter, per `consumer` Connections re-established after one was lost, and connection attempts that failed. Attempts back off exponentially up to 5 seconds during an outage, so the failure count grows by one per window per consumer, not one per request. ### Upstream engine metrics When metrics scraping is enabled, Rolter periodically pulls each provider's own Prometheus endpoint and re-exports what it finds through `/metrics`, labelled `{provider="..."}`. One scrape point then covers both Rolter and the engines behind it, and — because the label is the provider Rolter routes to — you can put engine saturation next to the routing decision that caused it. These series are scraped, not measured by Rolter. They appear only for providers that expose them: engines differ in what they publish, and a gauge an engine does not report is omitted rather than exported as zero, so `absent()` means "not reported", not "idle". **Type:** Gauge · Labels: `provider` Requests waiting to be scheduled on the upstream engine. This is the same number that feeds load-aware balancing, so it explains why Rolter steered traffic away from a target. Read from `vllm:num_requests_waiting`, `sglang:num_queue_reqs`, `num_requests_waiting` or `tgi_queue_size`, whichever appears first. **Type:** Gauge · Labels: `provider` Requests the engine is currently decoding. Paired with queue depth this separates "busy" from "backed up": high running with an empty queue is a fully-used engine, while a growing queue at flat running count is one that cannot keep up. **Type:** Gauge, 0–1 · Labels: `provider` Fraction of the engine's KV cache in use. Normalised to a fraction regardless of whether the engine reports 0–1 or a 0–100 percentage. Sustained high utilisation is the usual precursor to preemption and throughput collapse. **Type:** Gauge · Labels: `provider` Generation throughput as reported by the engine itself. Useful as a sanity check against Rolter's own latency histograms — if engine throughput is steady while Rolter's p99 climbs, the added latency is not in the engine. **Type:** Gauge, 0 or 1 · Labels: `provider` `1` when the most recent sweep reached the provider, `0` otherwise. Check this before trusting the gauges above: a failed scrape reports zeros for the counts, which without this flag is indistinguishable from a genuinely idle engine. A slow or unreachable engine cannot affect request handling. The sweep runs on its own background task under a timeout, and a failed scrape simply records the provider as down until the next sweep. ### Latency histograms Both histograms are labelled `{model="..."}` so you can track latency per model independently. Each histogram exposes the standard `_bucket`, `_sum`, and `_count` series. **Type:** Histogram · Labels: `model` End-to-end request latency in milliseconds — from the moment Rolter receives the first request byte to when the last response byte is sent to the client (including streaming). Use `histogram_quantile(0.99, rate(rolter_request_latency_ms_bucket[5m]))` for p99 latency per model. **Type:** Histogram · Labels: `model` Time-to-first-token in milliseconds for streaming requests. This measures how long the client waits before receiving the initial token — a key quality-of-service signal for interactive applications. Use the `_bucket` series for percentile calculations in Grafana or Prometheus recording rules. ### Configuration & health counters **Type:** Gauge The current configuration snapshot version. This increments each time Rolter successfully reloads its configuration. Alert on this staying flat if you expect regular updates, or spiking unexpectedly. **Type:** Gauge The snapshot format of the configuration snapshot the gateway applied last, or `0` while it holds only its bootstrap file. The control plane and the gateway agree on a format on every poll (see [Upgrading](/deployment/upgrading#upgrade-order)), so in a healthy fleet every gateway reads the same value, and the value changes only in a release whose notes say the format moved. **Type:** Gauge `1` while the control plane serves no snapshot format this gateway reads, and `0` once it answers in one the gateway does read. While it is `1`, the gateway keeps serving the config it last applied, and stops receiving changes, revoked keys included. A gateway that never applied a snapshot also reports not ready on `/readyz`. Alert on any non-zero value. It means a gateway was upgraded ahead of its control plane, or the control plane has retired the gateway's format. The gateway's log says which, and what to do. **Type:** Counter Budget-spend and rate-limit token records dropped because the post-response recording queue was full. Each drop means one request's spend is missing from its budget counters and its tokens are missing from its `tpm` windows. The queue is bounded on purpose: recording writes to Redis, and an unbounded backlog would grow worst exactly when the gateway is under stress. A non-zero rate here says the counter store is not keeping up — check Redis latency first, then raise `[usage_recording] workers` or `queue_capacity`. Sustained drops mean budget enforcement is under-counting and will let spend through. [Realtime](/api/realtime) sessions count here too. Each session meters its finished responses through a queue of its own, fixed at 1024 responses, and a response dropped from it loses its request-log row as well as its budget charge and `tpm` tokens. The `[usage_recording]` settings do not size that queue, so if drops continue after raising them, look at Redis latency during realtime sessions. **Type:** Gauge Usage records waiting to be written to the counter store, sampled at scrape time. This is the early-warning signal for the counter above: a queue that sits consistently deep is one traffic spike away from dropping. **Type:** Gauge Live per-`(model, target)` circuit-breaker entries the gateway is holding, sampled at the last configuration reload. The gateway hot-reloads configuration without restarting, so a long-lived process serves many configuration generations. Each reload retires entries for targets nothing routes to any more; targets that are currently open or half-open are kept, because forgetting one would silently re-admit traffic to an upstream that is still down. In a healthy fleet this gauge sits at or near zero — a recovered target releases its entry — and rises only while targets are actually failing. A value that climbs steadily across reloads and never falls is the signal that something is retaining state it should not. **Type:** Counter (all) Rolter also exposes a set of fine-grained counters for internal events. These are useful for debugging and alerting on edge conditions: | Metric | What it counts | | --- | --- | | `rolter_config_reloads_total` | Successful configuration hot-reloads applied | | `rolter_config_reload_failures_total` | Failed configuration snapshot fetches | | `rolter_budget_blocks_total` | Requests rejected because a spend budget was exhausted | | `rolter_rate_limit_blocks_total` | Requests rejected because a rate limit was exhausted | | `rolter_retries_total` | Upstream attempts made again after a transient failure | | `rolter_cooldowns_tripped_total` | Targets parked on a cooldown after a transient failure | | `rolter_key_cooldowns_tripped_total` | API keys parked on a cooldown after a key-level failure | | `rolter_health_down_total` | Providers marked unhealthy by an active health probe | | `rolter_health_recovered_total` | Providers restored to healthy by an active health probe | | `rolter_breaker_opened_total` | Targets tripped open by the circuit breaker after sustained failures | | `rolter_breaker_closed_total` | Targets closed by the circuit breaker after a successful half-open probe | | `rolter_metrics_scrapes_total` | Upstream `/metrics` scrape sweeps completed | ## Grafana dashboard tips Use `rolter_request_latency_ms_bucket` with `histogram_quantile()` to plot p50/p95/p99 latency panels, split by the `model` label for per-model breakdown. Use `rolter_request_ttft_ms_bucket` the same way to build a TTFT panel — critical for streaming LLM apps where perceived responsiveness matters more than total latency. Combine `rate(rolter_upstream_errors_total[5m])` and `rate(rolter_requests_total[5m])` to derive an upstream error rate percentage. Alert on a rising `rate(rolter_auth_failures_total[5m])` to detect misconfigured clients or credential rotation issues before they affect users. ## Exporting the same metrics over OTLP Everything above is also exported to an OpenTelemetry collector when one is configured, so a deployment already sending traces to SigNoz gets the metrics there too without standing up Prometheus. This is a **second exporter over the same numbers**, not a second metrics system. The Prometheus endpoint is untouched, and both exporters read one list — they cannot drift apart and report different values for the same counter. ```bash OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 rolter gateway ``` With no OTLP endpoint set, nothing is installed and nothing is exported. That is the default. ### What is exported | Kind | Metrics | How | | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------ | | Counters and gauges | every `rolter_*_total`, `rolter_config_version`, `rolter_snapshot_format`, `rolter_snapshot_format_refused` | observable | | Process and runtime | `process.*`, `tokio.runtime.*` | observable, OTLP only | | Labelled counters | `rolter_target_requests_total`, `rolter_variant_requests_total`, `rolter_complexity_route_requests_total`, `rolter_route_limit_blocks_total`, `rolter_redis_reconnects_total`, `rolter_redis_connect_failures_total`, `rolter_fail_open_total` | observable | | Labelled gauges | `rolter_provider_queue_depth`, `rolter_provider_inflight`, `rolter_redis_connected` | observable | | Histograms | `rolter_request_latency_ms`, `rolter_request_ttft_ms`, `rolter_provider_queue_wait_ms` | recorded per measurement | Counters and gauges are **observable instruments**: nothing is pushed on the request path, and the SDK reads the same atomics on its own schedule (`OTEL_METRIC_EXPORT_INTERVAL`, default 60s). The request path keeps doing nothing but an atomic add. Histograms are the one exception. OpenTelemetry builds a histogram from individual measurements, so rolter's pre-bucketed counters cannot be handed over after the fact — each value is recorded when it is measured: a request's latency when it completes, a queue wait when a worker takes the job. That work happens only where an OTLP endpoint is configured; without one the recorder is inert and allocates nothing. The OTLP histograms use the **same bucket boundaries** as the Prometheus ones (1ms → 10s). Two exporters of one measurement disagreeing about where the buckets fall would be worse than either alone, because the numbers look comparable and would not be. ### Labels become attributes Prometheus labels are exported as OTLP attributes with the same names: `provider`, `target`, `outcome` on target counters; `model` and `variant` on variant counters; `model`, `tier`, `route`, `outcome` on complexity counters; and `model` on both histograms. Values are carried raw. The backslash-escaping you see in the Prometheus text format exists for that format only — OTLP carries attribute values as typed strings, and escaping them twice would show `gpt\"4o` in your backend. ### Choosing a protocol The metrics exporter reads the same environment as the trace exporter: - `OTEL_EXPORTER_OTLP_ENDPOINT` — or `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` to send metrics somewhere other than traces - `OTEL_EXPORTER_OTLP_PROTOCOL` — `grpc` (default) or `http/protobuf` If the exporter cannot start, rolter logs the failure and carries on with local metrics only. A broken collector must never take the gateway down.