Rolter exposes a Prometheus-compatible metrics endpoint so you can plug it straight into your existing monitoring stack — no agents or sidecars required. Metrics are updated on the request path and rendered on demand when Prometheus scrapes the endpoint.
Scraping the endpoint#
By default, metrics are available at GET /metrics on the same port as the gateway (:4000). Add a Rolter job to your Prometheus configuration and you’re done:
Changing the metrics path#
If another service behind the same reverse proxy already owns /metrics, you can move Rolter’s endpoint to any path you choose:
The metrics_path value must start with / and must not collide with Rolter’s built-in routes: /healthz, /v1/models, /v1/chat/completions, /v1/completions, or /v1/messages.
Available metrics#
Request traffic#
rolter_requests_total
Type: Counter
Total number of requests proxied by the gateway since startup. Use this as your primary throughput signal — rate(rolter_requests_total[1m]) gives you requests per second.
rolter_client_disconnects_total
Type: Counter
Requests whose caller went away before the response finished — a closed browser tab, a client-side timeout, a pod rollout. These are logged with status 499 (nginx’s client closed request) and are deliberately never retried onto another provider, so they cost you one upstream attempt, not two.
A rising rate almost always means client timeouts are tighter than your upstream latency rather than anything being broken in the gateway. Compare rate(rolter_client_disconnects_total[5m]) against the p95 of rolter_request_latency_ms.
A client disconnect is never charged to a target. Whether the caller left while its request was queued in rolter, while the upstream was working, or part-way through a streamed answer, the request is not counted in rolter_target_requests_total (neither as outcome="error" nor as ok) and emits no passive health event, so it does not move the target’s uptime or the provider_health_flaps alert. This counter is where disconnects show up. An upstream that itself answers with status 499 is not a disconnect and still counts as that target’s error.
rolter_withheld_responses_total
Type: Counter
Responses the provider generated and billed, but that an output guardrail or a post_response plugin refused to deliver. The caller got a 403. The spend was still recorded against budgets, so this is how much you are paying for answers nobody received. Blocked response-cache hits are counted too, at no cost. See Withheld responses.
rolter_plugin_stream_rejections_total
Type: Counter
Streamed requests refused with plugin_streaming_unsupported because a fail-closed post_response plugin applies to them. That plugin must approve every response and cannot see a stream. Clients that hit this should retry without "stream": true, or the plugin should be switched to fail open. See Streamed responses.
rolter_inflight_requests
Type: Gauge
Requests currently in flight to upstream providers, summed across targets and sampled when you scrape.
On an idle gateway this reads 0. A floor that never comes back to zero means requests are exiting without releasing their slot, which quietly biases every load-aware routing decision — alert on min_over_time(rolter_inflight_requests[15m]) > 0 for a gateway that does have quiet periods.
rolter_upstream_errors_total
Type: Counter
Total upstream errors, broken down by error type. Incremented when an upstream returns a 4xx/5xx or a connection-level failure occurs (after all retries are exhausted).
rolter_rejected_requests_total
Type: Counter, labelled reason
Requests refused before the gateway knew who was asking, split by why: missing_key (no key presented, on a deployment that requires one), invalid_key (a key nobody issued, or one that is disabled or expired) and required_header_missing. These requests leave no row in the request log, so this counter is the only place to see how many there were and why. rolter_auth_failures_total is the sum of the series.
rolter_auth_failures_total
Type: Counter
Total authentication failures — requests that were rejected because the presented virtual key was missing, invalid, disabled, or expired.
rolter_upstream_connections_total
Type: Counter
Connections established to upstream providers. Every increment is the connection pool failing to find a connection it could reuse.
Watch the ratio, not the raw number: rate(rolter_upstream_connections_total[5m]) / rate(rolter_requests_total[5m]). Near zero means the pool is doing its job. Approaching one means every request is paying a fresh TCP and TLS handshake, which on a cross-region provider is most of your latency — check pool_idle_timeout and whether the provider is closing connections between requests.
rolter_upstream_connect_errors_total
Type: Counter
Attempts that failed before a connection existed — DNS, TCP, TLS, or a rejection by the egress policy.
Distinct from rolter_upstream_errors_total, which counts providers that answered with an error. A provider you cannot reach and a provider returning 500 need different fixes.
Runtime saturation#
tokio.runtime.queue.depth
Type: Gauge (OTLP only)
Tasks that are runnable and waiting for a worker thread. A sustained non-zero value means requests are queuing inside rolter itself, before any provider is called — which no upstream latency metric will ever show you.
This is the metric that separates “the provider is slow” from “we are saturated”. They look identical in end-to-end latency and have opposite fixes.
tokio.runtime.worker.busy.time
Type: Counter, seconds (OTLP only)
Cumulative time runtime workers spent doing work, summed across workers.
Exported as time rather than a ratio so it re-aggregates correctly: derive utilisation with rate(tokio.runtime.worker.busy.time) / tokio.runtime.workers.
tokio.runtime.workers / tokio.runtime.tasks.alive
Type: Gauge (OTLP only)
Worker threads in the runtime, and tasks currently alive. workers is the denominator for the utilisation calculation above; a steadily climbing tasks.alive with flat throughput points at tasks that are never completing.
The blocking-thread-pool metrics are deliberately absent. They are the part of
tokio’s metrics surface still gated behind --cfg tokio_unstable, which would
mean opting every build into an API that can change on a patch release. The
metrics above need no such flag.
Provider queue#
These size [queue] (Settings → Performance Tuning): workers is how many queue workers each provider has for sending requests upstream, and capacity is how many more requests can wait for one. Each series carries a provider label, one per configured provider, and all three are exported over OTLP as well.
workers bounds how many requests per provider are waiting for the upstream’s response headers: with workers = 8, eight such calls can be outstanding and a ninth waits. A worker is released as soon as the headers arrive, not when the answer is complete. A non-streamed request to an upstream that sends its headers with the finished answer holds its worker for the whole generation. A streamed one (stream: true) usually gets its headers almost at once and then keeps generating upstream after its worker has moved on, so workers does not cap how many streams, or how much GPU load, a provider carries. Cap concurrent generations at the engine itself.
Before 1.0 a provider’s workers took turns and sent one request at a time, so if you sized workers against latency measured on an earlier build, measure again.
rolter_provider_queue_depth
Type: Gauge, per provider
Requests waiting for one of the provider’s queue workers, sampled when you scrape (or at the OTLP export interval). A burst shorter than the sampling interval can fall between two samples; rolter_provider_queue_wait_ms records every wait. A depth that stays above zero means workers is the bottleneck for that provider — and latency climbs with it long before rolter_provider_queue_rejections_total moves. Reads 0 when queueing is off.
rolter_provider_inflight
Type: Gauge, per provider
Requests sent to the provider and still waiting for its response headers: at most workers while queueing is on, except briefly after a [queue] change under load, when the replaced queue’s workers are still draining its buffered requests: for that window the provider can see the old workers plus the new workers at once. A request that started before the change never swaps the old settings back in, so this happens once per change. A streamed body is not counted once its headers arrive — the same point at which a queue worker is released. A floor that never returns to 0 on a quiet gateway means requests are leaving without releasing their slot.
rolter_provider_queue_wait_ms
Type: Histogram, milliseconds, per provider
How long each request waited for a queue worker, from arriving at the queue to a worker taking it. The same buckets as the latency histograms (1ms → 10s), so histogram_quantile(0.95, sum by (le, provider) (rate(rolter_provider_queue_wait_ms_bucket[5m]))) sits beside the request latency p95 and shows how much of it is queueing.
rolter_provider_queue_rejections_total / rolter_provider_queue_timeouts_total
Type: Counter
Requests refused because a provider’s queue was full (backpressure = "error" or "drop"), and requests that gave up waiting for room with backpressure = "block". Both are the end state the gauges above give you warning of.
Redis#
Budgets, rate limits and the response cache each keep their own connection to Redis. When Redis cannot be reached, budgets and rate limits fail open: requests go through unchecked rather than the gateway going down with the counter store. These series show when that is happening and what it let through. They appear only when the gateway runs with a Redis URL.
rolter_redis_connected
Type: Gauge, per consumer (budgets, rate_limits, response_cache)
1 while the consumer holds a live connection, 0 once its last connection attempt failed or its connection was lost. Each consumer connects at startup. After that the state is what the consumer’s last use found: an idle consumer does not probe, so a gateway with no budgeted traffic can keep reading 0 after Redis is back until the next request uses it. For alerting, pair it with rolter_fail_open_total.
rolter_fail_open_total
Type: Counter, per control (budget, rate_limit)
Requests an admission check let through without consulting Redis because Redis was unavailable. This is the cost exposure of an outage: increase(rolter_fail_open_total{control="budget"}[1h]) is how many requests spent money no budget checked. Only requests that a budget or rate limit applies to are counted.
rolter_redis_reconnects_total / rolter_redis_connect_failures_total
Type: Counter, per consumer
Connections re-established after one was lost, and connection attempts that failed. Attempts back off exponentially up to 5 seconds during an outage, so the failure count grows by one per window per consumer, not one per request.
Upstream engine metrics#
When metrics scraping is enabled, Rolter periodically pulls each provider’s own Prometheus endpoint and re-exports what it finds through /metrics, labelled {provider="..."}. One scrape point then covers both Rolter and the engines behind it, and — because the label is the provider Rolter routes to — you can put engine saturation next to the routing decision that caused it.
These series are scraped, not measured by Rolter. They appear only for providers that expose them: engines differ in what they publish, and a gauge an engine does not report is omitted rather than exported as zero, so absent() means “not reported”, not “idle”.
rolter_upstream_queue_depth
Type: Gauge · Labels: provider
Requests waiting to be scheduled on the upstream engine. This is the same number that feeds load-aware balancing, so it explains why Rolter steered traffic away from a target.
Read from vllm:num_requests_waiting, sglang:num_queue_reqs, num_requests_waiting or tgi_queue_size, whichever appears first.
rolter_upstream_running_requests
Type: Gauge · Labels: provider
Requests the engine is currently decoding. Paired with queue depth this separates “busy” from “backed up”: high running with an empty queue is a fully-used engine, while a growing queue at flat running count is one that cannot keep up.
rolter_upstream_kv_cache_utilization
Type: Gauge, 0–1 · Labels: provider
Fraction of the engine’s KV cache in use. Normalised to a fraction regardless of whether the engine reports 0–1 or a 0–100 percentage. Sustained high utilisation is the usual precursor to preemption and throughput collapse.
rolter_upstream_generation_throughput_tokens_per_second
Type: Gauge · Labels: provider
Generation throughput as reported by the engine itself. Useful as a sanity check against Rolter’s own latency histograms — if engine throughput is steady while Rolter’s p99 climbs, the added latency is not in the engine.
rolter_upstream_scrape_up
Type: Gauge, 0 or 1 · Labels: provider
1 when the most recent sweep reached the provider, 0 otherwise. Check this before trusting the gauges above: a failed scrape reports zeros for the counts, which without this flag is indistinguishable from a genuinely idle engine.
A slow or unreachable engine cannot affect request handling. The sweep runs on its own background task under a timeout, and a failed scrape simply records the provider as down until the next sweep.
Latency histograms#
Both histograms are labelled {model="..."} so you can track latency per model independently. Each histogram exposes the standard _bucket, _sum, and _count series.
rolter_request_latency_ms
Type: Histogram · Labels: model
End-to-end request latency in milliseconds — from the moment Rolter receives the first request byte to when the last response byte is sent to the client (including streaming). Use histogram_quantile(0.99, rate(rolter_request_latency_ms_bucket[5m])) for p99 latency per model.
rolter_request_ttft_ms
Type: Histogram · Labels: model
Time-to-first-token in milliseconds for streaming requests. This measures how long the client waits before receiving the initial token — a key quality-of-service signal for interactive applications. Use the _bucket series for percentile calculations in Grafana or Prometheus recording rules.
Configuration & health counters#
rolter_config_version
Type: Gauge
The current configuration snapshot version. This increments each time Rolter successfully reloads its configuration. Alert on this staying flat if you expect regular updates, or spiking unexpectedly.
rolter_snapshot_format
Type: Gauge
The snapshot format of the configuration snapshot the gateway applied last, or 0 while it holds only its bootstrap file. The control plane and the gateway agree on a format on every poll (see Upgrading), so in a healthy fleet every gateway reads the same value, and the value changes only in a release whose notes say the format moved.
rolter_snapshot_format_refused
Type: Gauge
1 while the control plane serves no snapshot format this gateway reads, and 0 once it answers in one the gateway does read. While it is 1, the gateway keeps serving the config it last applied, and stops receiving changes, revoked keys included. A gateway that never applied a snapshot also reports not ready on /readyz. Alert on any non-zero value. It means a gateway was upgraded ahead of its control plane, or the control plane has retired the gateway’s format. The gateway’s log says which, and what to do.
rolter_usage_records_dropped_total
Type: Counter
Budget-spend and rate-limit token records dropped because the post-response recording queue was full. Each drop means one request’s spend is missing from its budget counters and its tokens are missing from its tpm windows.
The queue is bounded on purpose: recording writes to Redis, and an unbounded backlog would grow worst exactly when the gateway is under stress. A non-zero rate here says the counter store is not keeping up — check Redis latency first, then raise [usage_recording] workers or queue_capacity. Sustained drops mean budget enforcement is under-counting and will let spend through.
Realtime sessions count here too. Each session meters its finished responses through a queue of its own, fixed at 1024 responses, and a response dropped from it loses its request-log row as well as its budget charge and tpm tokens. The [usage_recording] settings do not size that queue, so if drops continue after raising them, look at Redis latency during realtime sessions.
rolter_usage_records_queued
Type: Gauge
Usage records waiting to be written to the counter store, sampled at scrape time. This is the early-warning signal for the counter above: a queue that sits consistently deep is one traffic spike away from dropping.
rolter_breaker_entries
Type: Gauge
Live per-(model, target) circuit-breaker entries the gateway is holding, sampled at the last configuration reload.
The gateway hot-reloads configuration without restarting, so a long-lived process serves many configuration generations. Each reload retires entries for targets nothing routes to any more; targets that are currently open or half-open are kept, because forgetting one would silently re-admit traffic to an upstream that is still down. In a healthy fleet this gauge sits at or near zero — a recovered target releases its entry — and rises only while targets are actually failing. A value that climbs steadily across reloads and never falls is the signal that something is retaining state it should not.
Additional event counters
Type: Counter (all)
Rolter also exposes a set of fine-grained counters for internal events. These are useful for debugging and alerting on edge conditions:
| Metric | What it counts |
|---|---|
rolter_config_reloads_total | Successful configuration hot-reloads applied |
rolter_config_reload_failures_total | Failed configuration snapshot fetches |
rolter_budget_blocks_total | Requests rejected because a spend budget was exhausted |
rolter_rate_limit_blocks_total | Requests rejected because a rate limit was exhausted |
rolter_retries_total | Upstream attempts made again after a transient failure |
rolter_cooldowns_tripped_total | Targets parked on a cooldown after a transient failure |
rolter_key_cooldowns_tripped_total | API keys parked on a cooldown after a key-level failure |
rolter_health_down_total | Providers marked unhealthy by an active health probe |
rolter_health_recovered_total | Providers restored to healthy by an active health probe |
rolter_breaker_opened_total | Targets tripped open by the circuit breaker after sustained failures |
rolter_breaker_closed_total | Targets closed by the circuit breaker after a successful half-open probe |
rolter_metrics_scrapes_total | Upstream /metrics scrape sweeps completed |
Grafana dashboard tips#
Use rolter_request_latency_ms_bucket with histogram_quantile() to plot p50/p95/p99 latency panels, split by the model label for per-model breakdown.
Use rolter_request_ttft_ms_bucket the same way to build a TTFT panel — critical for streaming LLM apps where perceived responsiveness matters more than total latency.
Combine rate(rolter_upstream_errors_total[5m]) and rate(rolter_requests_total[5m]) to derive an upstream error rate percentage.
Alert on a rising rate(rolter_auth_failures_total[5m]) to detect misconfigured clients or credential rotation issues before they affect users.
Exporting the same metrics over OTLP#
Everything above is also exported to an OpenTelemetry collector when one is configured, so a deployment already sending traces to SigNoz gets the metrics there too without standing up Prometheus.
This is a second exporter over the same numbers, not a second metrics system. The Prometheus endpoint is untouched, and both exporters read one list — they cannot drift apart and report different values for the same counter.
With no OTLP endpoint set, nothing is installed and nothing is exported. That is the default.
What is exported#
| Kind | Metrics | How |
|---|---|---|
| Counters and gauges | every rolter_*_total, rolter_config_version, rolter_snapshot_format, rolter_snapshot_format_refused | observable |
| Process and runtime | process.*, tokio.runtime.* | observable, OTLP only |
| Labelled counters | rolter_target_requests_total, rolter_variant_requests_total, rolter_complexity_route_requests_total, rolter_route_limit_blocks_total, rolter_redis_reconnects_total, rolter_redis_connect_failures_total, rolter_fail_open_total | observable |
| Labelled gauges | rolter_provider_queue_depth, rolter_provider_inflight, rolter_redis_connected | observable |
| Histograms | rolter_request_latency_ms, rolter_request_ttft_ms, rolter_provider_queue_wait_ms | recorded per measurement |
Counters and gauges are observable instruments: nothing is pushed on the
request path, and the SDK reads the same atomics on its own schedule
(OTEL_METRIC_EXPORT_INTERVAL, default 60s). The request path keeps doing
nothing but an atomic add.
Histograms are the one exception. OpenTelemetry builds a histogram from individual measurements, so rolter’s pre-bucketed counters cannot be handed over after the fact — each value is recorded when it is measured: a request’s latency when it completes, a queue wait when a worker takes the job. That work happens only where an OTLP endpoint is configured; without one the recorder is inert and allocates nothing.
The OTLP histograms use the same bucket boundaries as the Prometheus ones (1ms → 10s). Two exporters of one measurement disagreeing about where the buckets fall would be worse than either alone, because the numbers look comparable and would not be.
Labels become attributes#
Prometheus labels are exported as OTLP attributes with the same names:
provider, target, outcome on target counters; model and variant on
variant counters; model, tier, route, outcome on complexity counters;
and model on both histograms.
Values are carried raw. The backslash-escaping you see in the Prometheus text
format exists for that format only — OTLP carries attribute values as typed
strings, and escaping them twice would show gpt\"4o in your backend.
Choosing a protocol#
The metrics exporter reads the same environment as the trace exporter:
OTEL_EXPORTER_OTLP_ENDPOINT— orOTEL_EXPORTER_OTLP_METRICS_ENDPOINTto send metrics somewhere other than tracesOTEL_EXPORTER_OTLP_PROTOCOL—grpc(default) orhttp/protobuf
If the exporter cannot start, rolter logs the failure and carries on with local metrics only. A broken collector must never take the gateway down.