Per-Request Logs and Cost Tracking with ClickHouse

Rolter can write a structured row to ClickHouse for every request it proxies. Each row captures everything you need to audit usage, track spend, and investigate issues — token counts, cost in USD, latency, error details, and the A/B variant that served the request. Writes are asynchronous and batched off the request path, so enabling logging adds zero latency for your users.

Enabling logging#

Set clickhouse_url in the [logging] section of your config file to point at your ClickHouse HTTP interface:

[logging]clickhouse_url = "http://localhost:8123"

Logging is disabled when clickhouse_url is unset — there’s no placeholder or no-op mode.

The dashboard reads the rows back through the control plane, which needs its own CLICKHOUSE_URL (gateways with no address of their own inherit it). A control plane without one shows LLM Logs as a note that request logs need ClickHouse, MCP Logs as a note that tool-call logs need it, and the Dashboard as a note that traffic and spend need it, each naming CLICKHOUSE_URL. Cost Attribution shows the same kind of note in place of its spend figures, and Account & keys shows one above your keys (each key reads usage: unavailable), while the rest of both screens keeps working. It is information about the deployment rather than an error, so it offers no retry: nothing changes until the setting does.

What gets logged#

Every request from a caller the gateway has identified writes one row to the request_logs table with the following fields, whatever happened to it: a success, a failure after the upstream errored, or a refusal before any upstream was involved (see Failed and refused requests). With a sample_rate below 1 only the sampled requests do, each stamped with a sample_weight; a raw count() or sum() over the table shrinks with the rate, and sum(sample_weight) or sum(cost_usd * sample_weight) restores it.

The built-in fake-llm model is logged like any provider. It has no upstream, so its rows carry provider = builtin and target = fake-llm, and the token counts are the ones the fake response reports (a streamed chat completion included, though its stream carries no usage object). Its cost_usd is exactly 0, and its rows are never marked unpriced or usage_unknown: a model that costs nothing is not an accounting gap. They show up on LLM Logs and in the analytics screens with no special casing, which makes fake-llm a quick way to check that logging works end to end. Requests it refuses, for example a 400 for a missing input, are logged with that status. Requests to fake-llm are not charged against token-per-minute limits or budgets, and its rows carry no captured payload.

Failed and refused requests#

status is what the caller was told. When the gateway answers with an error of its own, it does not say which upstream caused it, so two more columns do: upstream_status is the HTTP status of the last upstream attempt that answered, and attempts is how many upstream attempts the request made. A provider that answers 429 on a route with one target reaches the caller as a 429 with code upstream_rate_limited and a Retry-After when the provider sent one, not as an overload, and its row reads status = 429, upstream_status = 429, attempts = 1. When every target of a route failed with another status, the caller gets a 503 with code upstream_unavailable that names the last upstream status, and the row carries the same.

error keeps the upstream’s reason, so LLM Logs shows a sentence such as upstream returned 429 after 1 attempt: Provider returned error: google/gemma-4-31b-it:free is temporarily rate-limited upstream. It is read from the error body the provider sent, reduced to one line of at most 300 characters, with any URL in it masked, and it is never the raw body. A connection that failed before any status line has upstream_status = 0 and the transport error in error.

A request the gateway refuses before it reaches an upstream is logged too, with an empty provider and target, attempts = 0, and the message the caller received as error: an unknown model (404), a model or route the key may not use (403), a spent budget or an unpriced model the deployment refuses (402), a rate limit (429), a request a guardrail, plugin or prompt template rejects (400), and a route with no targets (503). They are written to the log only; they add no latency or error series to /metrics, which have their own counters for these refusals.

LLM Logs shows the two columns where they explain a status. The detail drawer prints one sentence under the status badge: Upstream returned 500 after 3 attempts when an upstream answered, Upstream did not respond after 2 attempts when the connection failed before any status line, and Refused before reaching an upstream for a request the gateway turned away itself. A request that succeeded on its first attempt gets no sentence, and neither does a cache hit or a fake-llm call, which make no attempts either. In the table, a request whose status differs from the upstream’s carries the upstream’s status beside its own, such as 503 with 500 after it: a 503 the gateway made after every target of a route failed, with the sentence as the tooltip. A provider’s error handed to the caller as it came has the same number on both sides, so it gets no hint, and the drawer still says how many attempts it took. A row written before the columns existed has nothing to show and is drawn as it always was.

What is not logged is anything that happens before the gateway knows who is asking: a body that is not JSON, a request with no model, a missing, unknown or expired key, and a request refused for a missing required header or an oversized body. No tenant owns such a row, and anonymous traffic would otherwise be able to write to your log without limit. rolter_auth_failures_total counts the rejected keys, and rolter_rejected_requests_total{reason} splits them by reason (missing_key, invalid_key, required_header_missing), so “how many 401s, and why” is answerable even though none of them has a row. Everything after the key is resolved is logged, including a budget block, a rate limit and a guardrail refusal: the key bounds the write volume, since an attacker cannot mint valid keys.

Rows written before clickhouse/015_upstream_attempts.sql default both columns to 0.

The outcome of each request#

Every row also names what happened to the request in its outcome column, so a refusal is selected by its reason rather than inferred from an empty provider and a 4xx:

The refused_* rows are the requests refused after the key resolved, and read as described above: no provider or target and attempts = 0. The gateway picks the reason from the error code it answered with (insufficient_quota, rate_limit_exceeded, guardrail_blocked, plugin_blocked, model_not_found, …), and from the status where the answer has no code. New outcomes may be added in a later release; treat a name you do not know as “see status and error”.

The column is added by clickhouse/030_request_outcome.sql. Every row older than the file reads unknown, and a gateway upgraded before you apply it keeps logging and loses only the outcome. Filter by it in LLM Logs with Outcome, or with outcome= on GET /api/v1/analytics/invocations.

Calls on a stored response are logged too: GET, DELETE, cancel and input_items on /v1/responses/{id} write one row each, like any other request from a known caller. The row’s model is the model the response was created for, and its provider and target are the ones that hold it. lifecycle_operation says which call it was. The row has no tokens and no cost, and it is not marked unpriced or usage_unknown: the call generates nothing, so there is nothing to price. These calls are not billable, so they do not count against a budget, a token limit or a request limit, and they are left out of the per-model latency series on /metrics.

Failures read as they do for a model request. A provider that answered with an error has its status in upstream_status and attempts = 1; one that could not be reached is a 502 with upstream_status = 0. A call the gateway turned away itself has an empty provider and target and attempts = 0: an id that does not exist, has expired or belongs to another tenant (404, with no model since there is no stored response to take one from), access that was revoked since the response was created (403), or an operation the response’s provider has no API for (501).

LLM Logs shows which call it was. In the table a small outline label with the operation sits beside the model name: Retrieve, Delete, Cancel, Input items, Compact or Count input tokens. Hover it for the sentence, such as Call on a stored response: Retrieve. The detail drawer starts its Routing group with the same words under Stored response call. A call on an id that does not exist has no model, so the label stands alone in the model column and names the row in the drawer’s title. A request that ran a model, and a row written before the column, show neither the label nor the drawer row.

Rows written before clickhouse/016_lifecycle_operation.sql have an empty lifecycle_operation, as does every row of a request that ran a model. GET /api/v1/analytics/invocations returns the column on every row, as an empty string in those two cases, so an older row needs no special case. On a ClickHouse that has not had 016 applied the column reads as an empty string as well, so the file is not a condition of upgrading the control plane (see Upgrading). The API has no filter on the operation: look a call up by its request_id, or read the column in ClickHouse.

A realtime session writes one row per model response rather than one per session: request_id is the upgrade request’s id with the response’s ordinal appended (<id>:1, <id>:2, …), all rows of a session share its trace_id, and latency_ms and ttft_ms describe that one response. Rows arrive every [realtime] usage_flush_secs while the session is open. A response the session ended before it finished is logged with usage_unknown = 1.

Configuration reference#

All [logging] fields except clickhouse_url have sensible defaults and are optional:

[logging]clickhouse_url = "http://localhost:8123"  # required to enable logging# flush when batch reaches this many rows (default: 1000)batch_max = 1000# also flush partial batches at least this often, in milliseconds (default: 1000)flush_ms = 1000# bounded queue capacity; rows are dropped and counted when full (default: 10000)queue_capacity = 10000

Cost tracking#

The cost_usd field on each log row is computed from the [[model_prices]] entries in your config. You define the input and output price per million tokens for each model, and Rolter calculates the cost automatically:

[[model_prices]]model = "gpt-4o"input_per_mtok = 2.5output_per_mtok = 10.0cached_input_per_mtok = 1.25   # optional; defaults to input_per_mtok when absentcache_write_per_mtok = 3.125   # optional; defaults to input_per_mtok when absent

Cached prompts and thinking tokens#

Rolter reads the prompt-cache hit from the field the provider reports it in, and records it as cache_read_tokens. Every spelling below counts the hit inside the prompt total except Anthropic’s, which Rolter adds back in, so prompt_tokens is always the whole prompt and the cached share is priced at cached_input_per_mtok:

A provider that caches but is not in the table logs cache_read_tokens = 0 and is priced as fresh input until its field is added, so report the provider if you run into one. The table lists the providers whose documentation, or a response someone captured, names the field, not every provider that caches. Google documents no usage block for its OpenAI-compatible endpoints; Gemini’s is in the table from a captured response, but no response with a cache hit was found for Vertex AI’s, and Cohere’s, Upstage’s, OVHcloud’s, Scaleway’s and Ernie’s compatible endpoints are not documented either, so a cache hit from any of them is read only if it uses one of the spellings above. Doubao’s, Hunyuan’s and Gemini’s compatible endpoints are listed on the strength of the shape they copy, not of a primary source or a captured hit, so check cache_read_tokens on a request you know repeats a long prompt before relying on it.

Tokens a provider wrote to its cache are recorded as cache_write_tokens, and are part of prompt_tokens. Rolter reads Anthropic’s usage.cache_creation_input_tokens and the OpenAI-shaped spellings usage.prompt_tokens_details.cache_write_tokens (OpenRouter, Perplexity), usage.input_tokens_details.cache_write_tokens (Bedrock’s OpenAI models), usage.prompt_tokens_details.cache_creation_input_tokens (Qwen’s explicit cache) and usage.prompt_tokens_details.created_cache_tokens (vLLM). They are priced at cache_write_per_mtok, and the share Anthropic reports as written to its 1 hour cache at cache_write_1h_per_mtok (see Cache writes).

Gemini bills its thinking tokens as output, and reports them beside the answer’s count, not inside it. Rolter adds thoughtsTokenCount (total_thought_tokens on the Interactions API) to completion_tokens, so the request log, the budget and the usage block your client reads all count what the provider charged. The tokens a built-in tool such as URL context fed back to the model (toolUsePromptTokenCount, total_tool_use_tokens) are charged as input and are added to prompt_tokens. The usage Rolter shows a client always adds up: total_tokens is prompt_tokens plus completion_tokens.

The thinking share is part of completion_tokens, never on top of it, and a chat or Responses client served by a thinking model is shown how much of the completion it was: usage.completion_tokens_details.reasoning_tokens on Chat Completions, usage.output_tokens_details.reasoning_tokens on the Responses API. This is OpenAI’s own convention (its completion_tokens includes the reasoning tokens, reasoning_tokens is the breakdown, and total_tokens is the prompt plus the completion), and it holds for every provider: Together, Fireworks, Doubao, Perplexity, Upstage, vLLM and OpenAI state it that way themselves, and for the three that count their thinking beside the completion (xAI, Vertex AI and Gemini, below) Rolter moves it inside before the client sees it. A Messages client has no such field and reads the completion alone, and an answer whose provider reported no thinking states none. xAI reports its reasoning tokens beside completion_tokens, with only total_tokens adding them up (its API reference prints 32 + 9 + 110 = 151); Rolter sees that from the total and adds them to the completion, which is how xAI bills them.

Google does not document the usage of its OpenAI-compatible endpoints, but responses captured from them agree: completion_tokens leaves the thinking out. Gemini’s endpoint (provider kind = "gemini", generativelanguage.googleapis.com/v1beta/openai) has no field for it, so a thinking model’s total_tokens is more than prompt_tokens plus completion_tokens by exactly the thinking tokens; Vertex AI’s endpoint names them in usage.completion_tokens_details.reasoning_tokens beside the same short completion_tokens, and its total adds the three up. Both are billed as output, so Rolter adds them to the completion in the request log, the cost and the budget spend: for Gemini what the total has beyond the prompt and the completion, for Vertex AI and xAI the reasoning_tokens they name. Only the gemini kind is read from the total alone, because no other provider’s total is known to mean that; every other kind has only a reasoning count it names itself added.

What a client is shown. One rule for every provider and every dialect: the thinking is inside the completion and named, and total_tokens is unchanged. A Chat Completions client of the gemini, vertex or xai kind gets completion_tokens with the thinking in it, completion_tokens_details.reasoning_tokens naming it, and the provider’s own total_tokens, so prompt_tokens + completion_tokens == total_tokens and what the client adds up matches what Rolter bills. A Messages client gets output_tokens with the thinking in it, and a Responses client output_tokens plus output_tokens_details.reasoning_tokens. The rest of a Chat Completions response, streamed or not, is the provider’s own; only an answer or stream event whose usage has thinking beside the completion is rewritten, and an event with no usage is forwarded as it came. Nothing is added when the total is no more than the prompt and the completion (a model that did not think, or an endpoint that later counts thinking in completion_tokens itself), so an answer that already follows the rule is never changed and never counted twice.

Cache writes#

Writing a prompt into a provider’s cache can cost more than reading it as ordinary input: Anthropic charges a cache write at a multiple of the input rate (1.25 times for its 5 minute cache), and a higher one for its 1 hour cache (2 times). Rolter records the count as cache_write_tokens and prices it at cache_write_per_mtok, with a separate cache_write_1h_per_mtok for the 1 hour cache:

[[model_prices]]model = "claude"input_per_mtok = 3.0output_per_mtok = 15.0cached_input_per_mtok = 0.3cache_write_per_mtok = 3.75cache_write_1h_per_mtok = 6.0

The prompt total holds every token class once, so each token is charged at one rate: the cache reads at cached_input_per_mtok, the cache writes at cache_write_per_mtok (or cache_write_1h_per_mtok for those written to the 1 hour cache), and the rest of the prompt at input_per_mtok. A prompt of 2000 tokens that read 1500 from the cache and wrote 100 to it costs the 400 remaining tokens at the input rate, the 1500 at the cached rate and the 100 at the write rate. If a provider reports more cache tokens than the prompt holds, the reads are charged first and the writes take what is left, so a token is never charged twice.

• Absent means the input rate. A price without cache_write_per_mtok charges written tokens as ordinary input, exactly as every price did before the field existed, so upgrading changes no existing cost.
• A separate rate for the 1 hour cache. Anthropic reports a write to each cache apart, in usage.cache_creation (ephemeral_5m_input_tokens and ephemeral_1h_input_tokens), beside the combined cache_creation_input_tokens (Anthropic’s Messages API reference, prompt caching). Rolter keeps cache_write_tokens as the total and records the 1 hour part as cache_write_1h_tokens, so the 5 minute part is the difference. The 1 hour part is charged at cache_write_1h_per_mtok, which falls back to cache_write_per_mtok and from there to the input rate. A price that sets only cache_write_per_mtok charges both caches at it, exactly as before the field existed. A 100-token write that put 60 tokens in the 5 minute cache and 40 in the 1 hour cache costs 60 at cache_write_per_mtok and 40 at cache_write_1h_per_mtok, and the two parts are never charged on top of the total.
• Which providers report the split. Only Anthropic’s Messages API (a provider of kind anthropic, including one behind your own api_base that passes Anthropic’s usage through). It sends the split with the first event of a stream, so a streamed answer is split too. No other provider listed under Cached prompts and thinking tokens documents one: their writes are all priced at cache_write_per_mtok, whatever cache_write_1h_per_mtok says, and their rows log cache_write_1h_tokens = 0. A Chat Completions or Responses client served by an Anthropic provider is shown the split as cache_write_1h_tokens beside cache_write_tokens in prompt_tokens_details / input_tokens_details, and a Messages client served by a provider that stated one is shown Anthropic’s own cache_creation breakdown.
• Apply the ClickHouse migration to keep the share. cache_write_1h_tokens is the column clickhouse/017_cache_write_1h.sql adds to request_logs. A gateway that is upgraded before you apply it keeps logging and prices the request correctly, but cannot store the share: ClickHouse drops the field it has no column for, so only the requests that wrote to the 1 hour cache lose it, and the row’s cost_usd is unaffected. Rows older than the column read 0. GET /api/v1/analytics/invocations returns the share as cache_write_1h_tokens on every row, a number beside cache_write_tokens (see Optional columns in the invocation list); the 5 minute share is the difference, cache_write_tokens - cache_write_1h_tokens. The row detail in LLM Logs shows the split as Cache write by lifetime, the 5 minute and the 1 hour share side by side, beside the cache read and write figures. It appears only for a request whose cache_write_1h_tokens is above 0, so a request that wrote to the 5 minute cache only, one from a provider that reports no split, and a row older than the file show just the total. Applying the file is not a condition of upgrading the control plane (see Upgrading).
• Only reported writes are priced. OpenAI and Gemini report no write count (their caching is not billed per write), so their rows log cache_write_tokens = 0 and the field has no effect on them. A write count reaches Rolter from Anthropic’s cache_creation_input_tokens and from the OpenAI-shaped spellings listed under Cached prompts and thinking tokens: prompt_tokens_details.cache_write_tokens is OpenRouter’s and the one Rolter uses when it translates an Anthropic answer for a Chat Completions or Responses client, and Qwen and vLLM spell theirs differently.
• Setting it in the dashboard. The Pricing screen has a 5 minute cache-write price per Mtok (optional) field and a 1 hour cache-write price per Mtok (optional) field beside the cached-input one, and a price that sets either shows a 5 minute cache write or 1 hour cache write badge on its card. The first field is cache_write_per_mtok: Anthropic’s 5 minute cache, and the writes of any other provider that reports a write count. The second is cache_write_1h_per_mtok, which only an Anthropic provider uses. The 5 minute cache-write and 1 hour cache-write inputs in a model’s Pricing override section edit the same two rates; they are saved with the price, so they need the input or output rate too. Leave the 5 minute field empty to price writes at the input rate, and the 1 hour field empty to price those writes at the 5 minute rate. Emptying a rate that was set clears it, and saving a price without touching a field keeps the rate the price holds, so a rate set over the API or by an import survives an edit of something else. A rate the control plane refuses (one above 999999.999999) is shown at its own field. Earlier releases showed the model sheet’s input but stored its value on the route, where nothing read it: a rate entered there never changed a cost, and the field now reads and writes the price instead, so enter it again if you relied on it.
• Setting it over the API. The rates are stored with the rest of the price row as cache_write_per_mtok and cache_write_1h_per_mtok. A PUT /api/v1/model-prices that leaves either field out keeps the stored rate, so a client that has never heard of it cannot reset one. null clears cache_write_per_mtok back to the input rate and cache_write_1h_per_mtok back to cache_write_per_mtok. This differs from the other rates, which a PUT replaces. Re-importing a rolter.toml with rolter-seed --import follows the file instead: a price without the key has no write rate afterwards.

Images, audio and speech#

Some models are billed by something other than tokens: an image by the image, a transcription by the minute of audio, a text-to-speech request by the character of input. A price row carries a rate for each, in the row’s currency, beside the token rates:

[[model_prices]]model = "dall-e-3"input_per_mtok = 0output_per_mtok = 0image_per_unit = 0.04[[model_prices]]model = "whisper-1"input_per_mtok = 0output_per_mtok = 0audio_input_per_minute = 0.006[[model_prices]]model = "tts-1"input_per_mtok = 0output_per_mtok = 0speech_per_mchar = 15.0

The rates above are what OpenAI’s pricing page lists for those three models. They are examples: Rolter ships no price list, so check your provider’s before relying on one.

Which rate a request uses follows its endpoint, and the count comes from the request or the response:

How it combines and what it records:

• Added to the token cost. A model that reports tokens as well (gpt-image-1 does) is charged both: the tokens at the token rates and the images at image_per_unit. Set only the rate your provider actually bills; leave the other unset. A unit rate never replaces a token rate and a token rate never replaces a unit rate.
• Token rates are still required. A price row must name input_per_mtok and output_per_mtok, so a model billed only by the unit takes 0 for both. Zero is a rate, not a missing one.
• Unset means unpriced. An image, transcription or speech request that reports no tokens on a price with no rate for its unit is logged unpriced = 1 with cost_usd = 0, the same as a model with no price row. Before this, such a request on a row that priced only tokens was logged as priced and free.
• A duration that is not stated is unknown, not free. A transcription returned as json, text, srt or vtt carries no duration. With audio_input_per_minute set, the row is logged usage_unknown = 1 and cost_usd = 0. Ask for verbose_json, or use a model that reports its audio duration in usage, to have it priced.
• Failed requests cost nothing. A request the upstream refused records no unit cost. A response a guardrail or plugin withheld after the provider produced it is still billed: images by n, speech by its input, a transcription not at all (its duration was in the body that was withheld).
• Budgets count it. cost_usd is the figure budgets add, so unit-priced requests advance an org, team, project, key, business-unit or customer budget exactly as token-priced ones do, and a rate in another currency is converted to the base first.
• Setting it. The Pricing screen has an image, an audio minute and a speech character field, and a price that sets one shows a badge on its card. Over the API the rates are image_per_unit, audio_input_per_minute and speech_per_mchar on PUT /api/v1/model-prices, with the same rule as the cache-write rates: a body that leaves one out keeps the stored rate, null clears it. The Pricing override section of an image model’s sheet edits the image rate, and that of an audio model’s sheet the audio minute and speech character rates. Saving the sheet without touching one keeps the stored rate, and emptying one clears it.

Audio and text tokens#

A realtime or audio model bills the audio tokens it is sent and speaks at a different rate from its text tokens, and its usage says how many of each. A price row carries up to three audio rates beside the text ones, per million tokens:

[[model_prices]]model = "gpt-realtime"input_per_mtok = 4.0output_per_mtok = 16.0cached_input_per_mtok = 0.4audio_input_per_mtok = 32.0audio_output_per_mtok = 64.0cached_audio_input_per_mtok = 0.4

These figures are illustrative. Rolter ships no price list, so take the rates from your provider’s pricing page.

Where the split comes from, and what it applies to:

Each count is a share of the matching total (input_tokens, output_tokens), so a token is priced once:

• Unset falls back to the text rate. A row with no audio rate prices every token at input_per_mtok or output_per_mtok, exactly as before the audio rates existed. A response that names no audio share has nothing to apply them to, whatever the row sets.
• Cached audio has its own fallback. A cached audio token is charged at cached_audio_input_per_mtok, then at cached_input_per_mtok, then at the audio input rate. The cache hits are split into audio and text only when the provider names the audio share; an unsplit hit is read as text.
• Inconsistent counts are clamped. An audio count larger than the total it belongs to is capped at that total, so a response is never charged for more tokens than it reported.
• One cost. The figure on the request-log row is the one budgets and tpm windows see; there is no second audio-only cost.
• Setting it. The Pricing screen has an audio input, an audio output and a cached audio input field, and so does the Pricing override section of a model whose type is audio, under the same names; a field left empty there says which text rate it falls back to, and saving the model without touching one keeps the rate the price holds. Over the API they are audio_input_per_mtok, audio_output_per_mtok and cached_audio_input_per_mtok on PUT /api/v1/model-prices, and they keep the rule the cache-write rates keep: a body that leaves one out keeps the stored rate, null clears it, and a number sets it.

For how a realtime session is metered, including its input-audio transcription, see Realtime.

Prices are yours, not ours#

Rolter fetches no pricing data at runtime, ever. Every rate comes from your [[model_prices]] entries, from the dashboard’s Pricing screen or the Pricing override section of a model’s sheet (which write the same rows), or from an explicit catalogue import. That is deliberate: published list prices go stale, and most deployments have a negotiated rate, a committed-use discount or a self-hosted model whose real cost is a GPU bill. Rolter multiplies the tokens it counted by the number it holds for the model.

The dashboard’s pricing section links here for that reason — it is the answer to “where does this number come from”: from whoever typed it, or from a catalogue snapshot you imported on purpose.

Importing catalogue prices#

To save typing the list price of a model, you can import it from a catalogue. The control plane carries a vendored snapshot of LiteLLM’s model_prices_and_context_window.json (MIT, notice shipped beside it), cut down to the per-token input, output, cache read and cache write costs of the chat, completion, responses and embedding models. It is compiled into the binary, so importing works with no network, and the snapshot is refreshed by a maintainer running just refresh-prices, never by your deployment.

• GET /api/v1/model-prices/catalogue?q=gpt-4o searches it and states its source, licence, currency and snapshot date.
• POST /api/v1/model-prices/import with {"models": ["gpt-4o", …]} (at most 500 ids; superadmin, like any price write) creates the prices. Each result is created, updated, kept_override or not_in_catalogue.

In the dashboard, Pricing → Import from catalogue (superadmin) opens a searchable list of the snapshot’s models with their rates, the snapshot date and the source and licence. Pick the models and import them: the result states, per model, whether it was created, refreshed from the catalogue, kept as your own price or missing from the catalogue. The Source column of the Pricing screen marks each price as a catalogue value or as your override.

Three rules hold for every import:

• Your price wins. A price you set by hand (the Pricing screen, the API, [[model_prices]]) is an override: an import reports it as kept_override and does not touch it. Editing an imported price in the dashboard turns it into an override. To go back to the catalogue value, delete your price and import the model again.
• Catalogue prices are marked. An imported row carries its source and the date of the snapshot it came from (source_imported_at), and the dashboard shows both on the Pricing screen and in the model sheet.
• They are list prices, subject to change. The file is a third party’s best effort and drifts from the provider’s real price list, and the choice of LiteLLM as the source is itself provisional. Check an imported rate against your invoice before you rely on it for budgets.

The file states no currency; the importer states USD explicitly, so an import needs USD to be convertible in your deployment (it is the default base currency).

Cache hits and overall p95#

GET /api/v1/analytics/summary also reports cache_hit_requests, the requests answered from rolter’s response cache (a provider’s prompt cache is not counted here; it shows in each row’s cache_read_tokens), cache_read_tokens, the prompt tokens the provider served from its own prompt cache, beside prompt_tokens (a read share is the first over the second, and both are scaled by the sampling weight), and p95_latency_ms, the 95th percentile of latency over the whole window. The hit count is scaled by the log’s sampling weight like every other count, so the response-cache hit rate is cache_hit_requests / requests; the percentile is read from the sampled rows as they are, since a uniform sample already estimates it. The dashboard home shows both.

Withheld and unknown-usage traffic#

Two more flags keep a total honest. A withheld response is one the provider answered and billed but an output guardrail or a post_response plugin refused to deliver: the caller got a 403, and the cost is still in cost_usd. Unknown usage is the other way round: the upstream answered successfully but reported no token counts, so a zero there means “not reported”, not “free”.

GET /api/v1/analytics/summary and GET /api/v1/analytics/by-model report withheld_requests, withheld_cost (the part of cost_usd that bought an answer nobody saw) and usage_unknown_requests, scaled by the log’s sampling weight like every other count. Every row from GET /api/v1/analytics/invocations carries withheld and usage_unknown as 0 or 1.

The Dashboard says so whenever either count is non-zero in the window: a notice gives the withheld requests and the part of the spend they account for, and another gives the successful requests with no reported usage. Each links to LLM Logs over the same window and filters, narrowed to those rows with withheld=true or usage_unknown=true on GET /api/v1/analytics/invocations. The server applies both flags before the page is cut, inside the caller’s row scope, so a page of 50 is 50 matching rows.

LLM Logs marks a withheld row with its 403, the billed cost and the policy named in error, and shows a muted dash with an explanation instead of 0 for a row whose usage is unknown, the way it does for an unpriced one.

Unpriced traffic#

A zero cost_usd is ambiguous on its own: it means either “this request cost nothing” or “we have no price for this model”. The unpriced column tells them apart, so a spend total can say when it is incomplete rather than presenting a partial figure as final.

-- how much of a window's spend is missing, and which models are responsibleselect countIf(unpriced = 1)          as unpriced_requests,       uniqIf(model, unpriced = 1)    as unpriced_models,       round(sum(cost_usd), 6)        as known_spendfrom request_logswhere ts >= now() - interval 24 hour;

The dashboard reports the same two figures on GET /api/v1/analytics/summary (unpriced_requests, unpriced_models) and GET /api/v1/analytics/by-model (unpriced_requests), and shows a notice above the spend tile whenever a window contains unpriced traffic.

The flag is exposed per invocation too, not only in those summaries: every row from GET /api/v1/analytics/invocations carries unpriced beside its cost_usd, so a single request can be read the same way the totals are.

The LLM Logs screen does the same per request: a row whose model has no price shows an unpriced label in the cost column instead of $0.0000, because a zero there claims the request was free and a dash reads as a missing value. The table explains the label on hover, and the detail drawer prints the explanation as a sentence under it. It reads the row’s own unpriced flag to decide, which is the judgement the gateway made against the price catalogue as it was when the request was served. That matters once prices change: a request served before you priced a model stays unpriced in the log even after you add the price, and one served under a price you later delete keeps the cost it was actually charged.

The notice links to Model Prices, which lists the models that carried unpriced traffic in the last 7 days, most requests first, each with its request count and an Add price button that opens the price form with the model filled in. It reads unpriced_requests from GET /api/v1/analytics/by-model, scaled by each row’s sample weight when the request log is sampled, and leaves out a model that has a price now. The list is absent when nothing is unpriced, and on a deployment without ClickHouse the screen says it cannot name them instead of showing an empty list. The price catalogue below it is a table whose price columns sort and whose model column filters by name; figures follow the dashboard language.

The Models screen names the cause. A model with no price row shows no price set in the in/out price column instead of a dash, and an Unpriced filter chip above the table counts them and narrows the list to exactly those models — so the fix for an incomplete spend total is one click from the number that reported it. If the price catalogue itself cannot be read, the column falls back to plain figures rather than reporting every model as unpriced on no evidence.

Unpriced traffic accrues no spend, so it cannot exceed a budget no matter how much is served — budgets simply do not apply to a model you have not priced. unpriced_policy decides what to do about that:

unpriced_policy = "block"   # ignore (default) | warn | block

ignore is the default so that upgrading cannot start refusing traffic for a deployment that has not priced its models yet — which, on a fresh install, is all of them. Once your price catalogue is complete, block costs nothing and closes the gap; the unpriced-request count above is how you tell when you are there.

Two Prometheus counters track this: rolter_unpriced_served_total (requests served with no price — non-zero means your spend figures are a floor) and rolter_unpriced_blocks_total (requests refused by block).

Rows written before the unpriced column existed default to 0. That under-reports the unpriced share of historical data rather than inventing one.

Withheld responses#

An output guardrail with block, or a post_response plugin that blocks, runs after the provider has generated the answer. The caller gets a 403 and never sees the answer. The provider still charged for it, so Rolter bills it anyway:

• The row’s status is 403 (what the caller received) and withheld is 1.
• prompt_tokens, completion_tokens and cost_usd are the provider’s figures, and they count toward budgets and token rate limits like any delivered request.
• error names the policy, either guardrail_blocked: <rule name> or plugin_blocked: <plugin reason>. The rejected answer itself is never stored, even with payload capture on.

Usage is read from the provider’s response before any guardrail, plugin or PII sanitizer runs. A plugin that rewrites or strips the usage object changes what the caller sees, not what you are billed.

A response-cache hit that a rule blocks is logged with cache_hit = 1, withheld = 1 and zero cost, since nothing was generated upstream for it. The original miss was billed when it happened.

To see what blocked output is costing you:

select sum(cost_usd)            as withheld_spend,       count()                  as withheld_requestsfrom request_logswhere withheld = 1 and ts > now() - interval 30 day;

rolter_withheld_responses_total counts the same refusals in Prometheus. A sustained rate usually means a prompt is steering the model into output your policy forbids, and you pay for every attempt.

usage_unknown = 1 marks a different gap: the upstream answered but reported no usage at all, so the row’s zero tokens and cost are unknown rather than free. Most often it is an OpenAI-style stream sent without stream_options: {"include_usage": true}. A streamed Responses API answer (/v1/responses) needs nothing extra: its token counts arrive on the final response.completed (or response.incomplete) event and are logged from there. It is also set on a 502 whose upstream answered and then failed to deliver the body the gateway was reading (a reset, a truncated response): the provider may have billed an answer nobody received. Rows written before these columns existed default to 0.

MCP tool calls#

When ClickHouse is configured, the gateway records each tools/call it proxies on /mcp/{server} as one row in Observability → MCP Logs. Nothing else is recorded: initialize, tools/list and notifications leave no row, and with no clickhouse_url the gateway records nothing.

Each row carries the server, the tool, the latency and a status: success, error (the tool reported isError: true or the server answered with a JSON-RPC error), auth_denied (the gateway refused the caller’s OAuth session, or the server answered 401/403), timeout, or transport_error. It is attributed to the calling virtual key’s organization, team and project, and to the user who owns the key, which is the user whose OAuth session the call used.

The call’s arguments and result are stored only when [logging.payload_capture] is enabled, redacted with redact_fields and cut to max_bytes, the same policy as LLM payloads. Recording never slows or fails the call: if the write queue is full the row is dropped and rolter_mcp_events_dropped_total goes up. For a streamed (SSE) reply the row is written when the stream ends, so its latency includes the whole stream.

Time windows#

Every usage, spend, provider-health and MCP-log read takes an optional since (inclusive) and until (exclusive). Left out, or sent empty, they cover the last 7 days up to now. Each accepts:

• RFC 3339, which is what Date.prototype.toISOString() and most SDKs write: 2026-07-01T00:00:00Z, 2026-07-01T00:00:00.123+02:00
• a date and time with no offset, read in the ClickHouse server’s time zone: 2026-07-01 00:00:00, 2026-07-01T00:00:00.000
• a bare date: 2026-07-01

Anything else is refused with 400 and an error naming the parameter:

{  "error": {    "message": "since must be an RFC 3339 timestamp such as 2026-07-01T00:00:00Z",    "type": "invalid_request_error",    "param": "since",    "code": "invalid_time_bound"  }}

That covers a date the calendar does not have (2026-02-30), a lowercase t or z, and a Unix timestamp; send those as RFC 3339 instead. The check exists because ClickHouse reads a bound it cannot parse as 1970-01-01, so without it a typo would scan the whole log rather than fail.

Filtering LLM Logs#

Filters opens a rail beside the LLM Logs table that narrows the requests of one time window, the last 24 hours unless you pick another. The window is counted back from each refresh, so a screen left open keeps showing a single day. The control plane applies each filter before it cuts the page, so a page of 50 is 50 matching requests.

• Time window is one of the reporting windows. The address carries it as window=7d.
• Status is one of All, Errors and OK. Errors is every request answered with a status of 400 or above, plus every request that got no answer at all (status 0, shown as ERR in the table and No response in the drawer), since an upstream that never answered is the failure you are looking for. OK is every request answered with a status below 400, which includes a redirect as well as a 2xx, and never a request that got no answer. All applies no filter.
• Model takes one model, because the API filters on one exact model name. Picking a second model replaces the first.
• Virtual key takes one key, picked by name from the keys of the project you are looking at. The address carries it as key=<id>.
• Cost has three checks. Unpriced only keeps the requests recorded as unpriced (see Unpriced traffic); the API takes it as unpriced=true. Withheld responses only keeps the requests a policy refused to deliver after the provider billed them, and Unknown usage only keeps the successes whose upstream reported no usage (see Withheld and unknown-usage traffic); the API takes them as withheld=true and usage_unknown=true. They combine with each other and with every other filter, but a withheld response is a 403 and a response with unknown usage was a success, so ticking both returns nothing. Like every filter here they narrow the rows your roles can read and never widen them.
• Business unit and Customer each take several values. A request matches when it was charged to any of the ones you picked.
• Outcome keeps the requests the gateway recorded with one outcome, such as Refused: budget spent for the requests a spent budget turned away. Not recorded keeps the rows written before the gateway recorded one. The API takes it as outcome=refused_budget; a value it does not know is a 400.
• Plugin appears when the address names one (plugin=blocked:<slug>), and keeps the requests that plugin acted on. Clear filters drops it.

Clear filters at the top of the rail drops every filter at once. When nothing matches, the empty table offers the same button.

The filters are part of the page’s address, so a reload keeps them and a copied link opens the same view. Whoever opens it still sees only the rows their own role allows. The parameters use the API’s names:

/logs?status=error&model=gpt-4o&business_unit=<unit-id>&unpriced=true

A link can outlive what it names. A status value the API does not know is ignored, and so is an unpriced, withheld or usage_unknown value other than true, and a model that has since been removed from the catalogue still filters the log and still shows in the picker, so you can see it and clear it. The row open in the detail drawer is not part of the address, except when it was opened by an id lookup, whose link opens the drawer again.

Saved views#

Saved views beside Filters keeps the filters you have applied under a name, so you can come back to them in one click. On LLM Logs a view holds the time window, status, model, key, business unit and customer; on the Dashboard it holds the window, model, key, business unit and customer. It never holds an id lookup, the page you were on or the Cost checks. Pick Apply to set the screen’s filters from the view (a Cost check you had ticked is cleared, since the view does not hold it), Rename to change its name, or the bin to delete it. Each account keeps up to 50 views per screen, and a name can be used once.

Views are private: they follow your account to any browser you sign in from, and nobody else sees them. They never widen what you can read. If a view names a key, business unit or customer that you can no longer read, it is applied without that filter and the screen says how many it left out, for example Applied without: 1 key, 2 customers. The ids are not shown. Saving again keeps only the filters applied at that moment.

The Dashboard has the same control and, applying a view, shows the same Applied without: … notice for anything you can no longer read. Saved views need a signed-in account, so they are unavailable while the control plane runs in open mode.

Paging the invocation log#

GET /api/v1/analytics/invocations returns one page of requests plus the cursor that resumes it:

{ "data": [ /* newest first */ ], "next_cursor": "2026-07-19 12:00:01.000|req-1" }

Ask for the next page by handing that value straight back as cursor:

curl -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" \  "https://control.example.com/api/v1/analytics/invocations?limit=50&cursor=2026-07-19%2012%3A00%3A01.000%7Creq-1"

The cursor is a position in the list’s sort key — (ts, request_id), newest first — not a row number, and this is what makes paging a live log correct. request_logs is written continuously by the gateway, so requests land above the window while you read: a row offset would count them in and either serve you a row you already saw or skip one entirely. A keyset bound cannot be moved by rows above it, so every request that existed when you started paging is returned exactly once. request_id breaks ties, so two requests logged in the same millisecond still have one order.

Treat the cursor as opaque and pass it back unchanged. next_cursor is null only for an empty page; a short page still carries one, so stop when a page comes back with no rows. limit is clamped to 1..=200 and defaults to 50. A cursor that is not timestamp|request_id, or whose timestamp is not one of the time window forms, is a 400 with param: "cursor" and code: "invalid_cursor". The MCP call log (GET /api/v1/mcp/logs) pages the same way over (ts, event_id) and checks its cursor the same way.

The LLM Logs screen pages the same way. Next page follows the cursor the current page returned and Previous page goes back to the one it came from, so a page you return to holds the requests it held before, however much traffic arrived in between. Only the first page follows live traffic while Streaming is on; later pages stay fixed on their cursor. Changing a filter starts again from the newest request, and a page after a full last page reads as the end of the log with a Back to newest button.

Optional columns in the invocation list#

These fields of a row come from ClickHouse files that an existing deployment applies by hand (see Upgrading). Every row from GET /api/v1/analytics/invocations carries all of them, whether or not the file has been applied:

guardrail_events lists what each guardrail did to the request, as <action>:<rule name> entries (blocked:, redacted: or annotated:) and provider_blocked:<provider name> / provider_transformed:<provider name> for the guardrail provider (the bare provider_blocked / provider_transformed on rows that predate the name, or for a provider with none); it is empty for a request no guardrail touched. See What each rule did.

fallback_reason is the class of the failure that sent the request on to the target that answered it: throttled, timeout, transport, server_error, refusal or refusal:<category>. It is empty when the first dial answered. See Fallback targets.

plugin_events lists what each plugin did to the request, as <outcome>:<slug> entries (allowed, blocked, transformed or annotated) and error:<slug>:<kind> for a call that got no verdict; it is empty for a request no plugin was consulted on. See What each plugin did.

The five request-shape fields say what a request asked the model to do beyond its tokens, read by the gateway off the body the caller sent, in the OpenAI (Chat Completions, Responses) or Anthropic (Messages) dialect: has_tools is 1 when it offered the model tools, has_image_input and has_audio_input when an input message carried an image or audio, has_response_schema when it asked for output that follows a JSON schema (a bare JSON mode is not one), and turns is how many user turns it carried. They are 0 for a request that is not a chat-shaped model call, such as an embedding, and the savings advisor excludes a request that has any of them set, or more than one turn.

cache_write_1h_tokens is the part of the row’s cache_write_tokens that went to the provider’s 1 hour cache, never more than the total. The 5 minute share is the difference: cache_write_tokens - cache_write_1h_tokens. It is 0 for a provider that reports no split (every provider but Anthropic), for a request that wrote only to the 5 minute cache, and for every row older than the file.

The control plane checks which of these columns the table has by reading system.columns, selects the ones that exist and fills in the default for the rest, so a request list never fails because a file is still to be applied. It remembers the answer: a table that has all of them is looked at again every ten minutes, one that lacks a column every minute, so a file you apply shows on the next page within a minute and needs no restart. A ClickHouse user that may not read system.columns is treated as having every column, which is how the list behaved before it checked. The check is made once the request is known to be well formed, and a ClickHouse that cannot be reached is an error as it always was, never a default.

Finding one request by its id#

Every gateway response carries x-request-id (or the header the deployment set as its request ID header), and the same id is the row’s request_id. When a client reports a failed call, that id takes you straight to it:

curl -si https://gateway.example.com/v1/chat/completions ... | grep -i x-request-id# x-request-id: 3f2c9a1e-...curl -H "Authorization: Bearer $ROLTER_TOKEN" \  "https://control.example.com/api/v1/analytics/invocations?request_id=3f2c9a1e-..."

The answer is that one row, with its status, error, tokens, cost, latency and — if your role may read them — the captured bodies. trace_id looks a request up by its W3C trace id the same way, for when you start from a span in your tracing backend instead. Without a since, a lookup by either id searches every row still retained rather than the default last 7 days, so an older request is found rather than answered with an empty page. The lookup obeys the same visibility as the rest of the log: an id from a project you cannot read returns nothing.

Looking a request up in the dashboard#

LLM Logs has a field under its toolbar for the same lookup. Paste the id and press Find. A 32-character lowercase hex string is a trace id, and so is a whole traceparent header value (Rolter reads the trace id out of it); anything else is taken as a request id. A request id is matched exactly.

• One row comes back: its detail drawer opens on it. This is the usual result for a request id.
• Several rows share the trace id: they are the list, and the toolbar says it is showing a request ID or trace ID lookup. Click a row to open it.
• Nothing comes back: the table says No request found. The dashboard cannot tell an id that does not exist from one in a project your role cannot read, and neither can the API, so the message names both.

A lookup searches every retained row, not the last 24 hours, and leaves out whatever the filter rail held: starting one clears those filters, so they cannot hide the request. It does not stream. Clear lookup (or the cross in the field) goes back to the live feed.

The lookup is part of the page’s address, so a link opens on the request:

/logs?request_id=3f2c9a1e-.../logs?trace_id=0af7651916cd43dd8448eb211c80319c

The command palette does the same from anywhere in the dashboard. Open it with ⌘K (Ctrl-K on Windows and Linux), paste the id, and pick Open request … in LLM Logs (or Open trace …). The offer appears only when no screen or record matches what you typed, and only for an account that can open LLM Logs. A word has to look like an id to be offered: at least 8 characters, no spaces, with a digit in it.

Reading a request in LLM Logs#

Clicking a row in LLM Logs opens it in a detail drawer, and the row stays highlighted in the table while the drawer is open. The drawer starts with the verdict: the status, when the request started, and its request id and trace id, each with a copy button. A status of No response means the gateway never got an answer from the upstream, because the connection was refused or timed out. A trace id reads Not sent by the caller when the request carried no trace header, since there is nothing to follow into a tracing backend.

When the request failed, its error comes next, before anything else. The rest of the row is grouped:

• Routing: provider → target, the A/B variant, whether Rolter answered from its response cache, whether the response was streamed, time to first token and latency.
• Usage and cost: input and output tokens, the upstream’s prompt-cache reads and writes, the 5 minute / 1 hour split of the write (Cache write by lifetime, only when part of it went to the 1 hour cache), and the cost.
• Attribution: the virtual key by its name and prefix, the business unit and the customer. A key that cannot be looked up, for example one that has since been deleted, shows its id instead.

The captured request and response bodies follow, or the reason they are absent (see log settings). A request with neither body gets one note under a single Request and response heading, with one link to the log settings. A request with one body shows it, and the missing one gets its own short note.

Reading the table#

The Time column shows the clock with milliseconds, such as 15:34:56.789, because the milliseconds are what tell requests apart. The date is in the tooltip on the time and at the top of the detail drawer. Status and Cost stay on screen at every width, with Time and Model. Latency, Tokens and Provider drop out as the table gets narrower, widest first, and all three are in the detail drawer. The table’s own width decides, not the window’s, so opening the filter rail or the drawer hides columns instead of scrolling Cost out of view. Below 1280 px the drawer opens as a sheet over the table instead of beside it.

Who reads the MCP call log#

GET /api/v1/mcp/logs, /api/v1/mcp/logs/summary and /api/v1/mcp/logs/{event_id} follow the same rule as the request log. A signed-in user reads the tool calls of the orgs, teams and projects where they hold a role, and every call made under their own MCP OAuth sessions, whatever project the gateway attributed it to. A superadmin, the admin token and open mode read everything. Someone with no role anywhere gets an empty list and a summary of zero, and a call outside their reach answers 404 exactly like one that does not exist.

The list carries no tool arguments or results. The single-event read carries them only for a caller who is a member or above at the call’s project, or a viewer on a project whose admin set payload_min_role to viewer (the same setting that shows LLM request bodies). Anyone else gets both fields empty and payload_withheld: 1. Recording an event (POST /api/v1/mcp/events) stays with the gateway and superadmins.

Calls recorded before an MCP proxy attributed them to an org, project and user carry none of the three, so they stay visible to superadmins only. Nothing needs migrating: new calls are attributed when they are written.

Built-in dashboard#

The Rolter control plane’s dashboard queries ClickHouse directly to give you usage, spend, latency percentiles, and error rates — all sliced by model, project, and virtual key. No extra ETL or data pipeline is needed: the rows Rolter writes are the source of truth for the dashboard.