> ## Documentation index > Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt > Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=. Short keywords rank best. > Use these to discover all available pages before exploring further. # Spend Budgets and Rate Limits for LLM Cost Control > Configure USD spend caps and RPM/TPM throughput limits at org, team, project, or key scope to control LLM costs and prevent runaway usage. Rolter gives you two complementary levers for controlling how much your gateway spends and how fast it moves traffic: **spend budgets** block requests once a cumulative USD ceiling is reached, and **rate limits** reject requests that exceed per-minute throughput thresholds. Both systems operate across the same scopes — the four-level tenancy hierarchy (org → team → project → key) plus the two cost-attribution dimensions a key rolls up to (business unit, customer) — and apply a most-restrictive-wins policy when multiple rules match a single request. Budgets and rate limits both require Redis (`ROLTER_REDIS_URL`). Rolter uses Redis for distributed counters so limits remain accurate across multiple gateway instances. Without Redis, both budgets and rate limits **fail open** — they are not enforced and all requests pass through. ## Step 1 — Define model prices Before budgets can track spend, Rolter needs to know the cost of each model's tokens. Add one `[[model_prices]]` block per model with rates per million tokens, denominated in the currency you name (USD by default). The public model name as it appears in your `[[routes]]` configuration. Cost in USD per million input (prompt) tokens. Cost in USD per million output (completion) tokens. Cost per million cache-hit input tokens. Defaults to `input_per_mtok` when omitted. Cost per million input tokens written to the provider's prompt cache. Defaults to `input_per_mtok` when omitted, so a price without it charges a cache write as ordinary input. See [Cache writes](/observability/logs-and-cost#cache-writes). Cost per million input tokens written to the provider's 1 hour prompt cache, for a provider that reports the split (Anthropic). Defaults to `cache_write_per_mtok` when omitted, and from there to `input_per_mtok`. See [Cache writes](/observability/logs-and-cost#cache-writes). Cost of one generated image, for a model billed per image. Added to the request's token cost, and counted against budgets like it. See [Images, audio and speech](/observability/logs-and-cost#images-audio-and-speech). Cost of one minute of input audio on a transcription or translation. Read from the duration the response states. Cost per million characters of text sent to a text-to-speech model, counted from the request's `input`. Price per million audio input tokens, for a realtime or audio model that bills the audio it is sent apart from the text. The share of the prompt that is audio is the one the response's usage names (see [Audio and text tokens](/observability/logs-and-cost#audio-and-text-tokens)). Defaults to `input_per_mtok`. Price per million audio output tokens. Defaults to `output_per_mtok`. Price per million cached audio input tokens, for a provider that says how many of its cache hits were audio (a realtime session does). Defaults to `cached_input_per_mtok`, and from there to the audio input rate. Currency the rates above are denominated in. Any code works — ISO-4217, a crypto unit, an internal credit — as long as `[currency.rates]` can convert it to the base currency. Rates are written as ordinary numbers here, but Rolter computes with them in **exact decimal arithmetic**, not binary floating point. A rate of `0.1` means a tenth rather than the nearest binary approximation of one, so per-request costs and currency conversions carry no rounding drift, and a budget's `spend >= limit` test has an exact boundary rather than one that depends on the order requests happened to be summed in. One caveat while this rolls out: the spend _counter_ itself still accumulates in Redis in floating point, so a long-running counter can still drift very slightly from the exact sum of its requests. Making the accumulation exact too is tracked in [#967](https://github.com/rolter-ai/rolter/issues/967). ```toml rolter.toml [[model_prices]] model = "gpt-4o" input_per_mtok = 2.5 output_per_mtok = 10.0 cached_input_per_mtok = 1.25 [[model_prices]] model = "claude" input_per_mtok = 3.0 output_per_mtok = 15.0 cache_write_per_mtok = 3.75 cache_write_1h_per_mtok = 6.0 ``` Models without a `[[model_prices]]` entry log a `cost_usd` of `0` and an `unpriced` flag of `1` per request. They can still be used, but their spend does not count toward any budget. Add a price entry for every model you want budgets to track, and see [Unpriced traffic](#unpriced-traffic) below for how to make that gap fail closed instead of open. ### Pricing in another currency Spend and budgets accumulate in a single **base currency**. A model priced in anything else is converted at a rate you configure, once, when the gateway loads its config — so pricing in EUR while budgeting in USD is exact rather than off by the FX rate. ```toml rolter.toml [currency] base = "USD" rates = { EUR = 1.09, GBP = 1.27 } [[model_prices]] model = "mistral-large" input_per_mtok = 2.0 output_per_mtok = 6.0 currency = "EUR" # charged into the USD budget at 1.09 ``` A rate is how many base units one unit of that currency is worth. The base itself is always `1.0` and needs no row. Codes are open strings rather than a fixed list, so adding `BTC` or an internal credit unit means adding a rate row — no new release required. A model priced in a currency with no rate is a **config error**, not a warning: the gateway refuses to start and the dashboard refuses the write. Charging face value would bill the wrong number and charging zero would bill nothing, so neither is offered as a fallback. Rates are operator-supplied and static, which is deliberate — Rolter must run air-gapped, so it never reaches out to an FX feed. ### What the dashboard offers The currency chooser on a model's pricing section is your rate table, not a built-in list: it shows the base currency followed by every code in `[currency.rates]`. Adding `rates = { RUB = 0.011 }` makes `RUB` selectable on the next control-plane restart, with no dashboard release involved. That set is read from `GET /api/v1/currency`, which answers from the bootstrap config rather than the database — the store keeps a currency per model price and no rate table, so this is the one place that knows what a deployment can convert. It is also exactly the set the CRUD API accepts, so the chooser cannot offer a code the save would reject. A price already stored in a code that is no longer in the table still displays, flagged as unconvertible on both the Pricing screen and the model sheet. Its spend is missing from base-currency totals and budgets until you add the rate back. Every amount the dashboard renders — LLM Logs, budget cards, team cards, the spend tiles — is formatted in the base currency from that same endpoint, so a deployment budgeting in EUR reads `€` rather than a hardcoded `$`, and the budget form's **Limit** field is labelled `EUR` rather than `USD`. Model prices are the exception: they are shown in the currency the price row was written in, which is what you typed. Amounts, dates and counts all follow the dashboard's language rather than the browser's, so the separators in a number never disagree with the copy beside it. ## Step 2 — Configure spend budgets A spend budget blocks incoming requests with **HTTP 402** once the cumulative cost for a scope reaches `limit_usd` within the configured period. Add one `[[budgets]]` block per scope you want to cap. The scope level this budget applies to. One of `org`, `team`, `project`, `key`, `business_unit`, `customer`, or `user`. The UUID of the entity at the chosen scope (org ID, team ID, project ID, virtual key ID, business unit ID, or customer ID). `business_unit` and `customer` are not part of the org → team → project chain: they are the [cost-attribution dimensions](/observability/logs-and-cost) a virtual key rolls its spend up to, so a budget on one caps everything attributed to it regardless of which project the traffic came through. The control plane API and the dashboard's **Budgets & Limits** screen accept all six scopes; in the dashboard, picking one of these two lists your org's business units or customers rather than asking for a UUID. Picking a virtual key lists the current project's keys the same way. While the list loads the field shows a placeholder, and if the control plane cannot return it (keys, business units or customers alike) the screen says so under the scope picker with a **Try again** button, and the field takes the UUID meanwhile. **A budget on a person.** Members mint their own keys, as many as they like, so a key cap bounds one key and a project cap is shared by everyone in the project. A `user` budget (and a `user` rate limit) counts every key that person minted, however many, which is how "each engineer may spend $50 a month" is said. It is narrowed with `within` to the keys they minted in one org (`{ org = "…" }`) or one project (`{ project = "…" }`); without it, it counts their keys everywhere, and only a superadmin may set that. It composes like every other scope: the most restrictive matching budget wins, and the `402` names the user budget (`budget exceeded for User '…'`) when it is the one that tripped. A key nobody minted (a config-file key, or one an admin created for a project) has no owner, so no user cap reaches it. In the dashboard, pick **User** as the scope type on **Budgets & Limits**, choose the person, and the new budget or rate limit asks which keys it counts: the ones minted in one organization (the default), in one project, or, for a superadmin only, every key. Rows of this kind are named after the person everywhere on the screen. The person sees their own caps and what is left of them under **Account**; deleting the person deletes their caps. The maximum cumulative spend in USD for the period before requests are blocked. The window over which spend accumulates. One of `monthly`, `daily`, or `total`. `monthly` and `daily` windows reset automatically; `total` is a lifetime cap that never resets. ```toml rolter.toml # Hard ceiling for the entire org — resets on the first of each month [[budgets]] scope = "org" id = "org-uuid" limit_usd = 500.0 period = "monthly" # Tighter daily cap for a specific project [[budgets]] scope = "project" id = "project-uuid" limit_usd = 20.0 period = "daily" # Lifetime cap on a single virtual key [[budgets]] scope = "key" id = "vk-uuid" limit_usd = 100.0 period = "total" ``` When a request's scope chain matches multiple budget rules (e.g. both an org budget and a project budget), Rolter applies **most-restrictive-wins**: the request is blocked as soon as any matching budget is exhausted. Set an org-level budget as a hard ceiling to protect against unexpected runaway costs, then add project- or key-level budgets to allocate share to individual teams or clients. ### Unpriced traffic A budget can only block spend it can see. A model with no `[[model_prices]]` entry accrues **zero**, so it can never exhaust a budget however much is served — the control fails open on exactly the models you most recently added and are least likely to have priced. `unpriced_policy` is the deliberate decision about that. It is set deployment-wide, since it governs what the gateway may serve at all rather than the state of any one counter, and an individual budget may then ask for something stricter — see [Per-budget override](#per-budget-override): ```toml unpriced_policy = "block" # ignore (default) | warn | block ``` | policy | behaviour | | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `ignore` | Serve it and accrue nothing. Budgets silently do not cover it. | | `warn` | Serve it, but log the model once per hour so an unenforceable budget names itself. | | `block` | Refuse at admission with **HTTP 402** and error code `model_unpriced`, naming the model. The check runs before the upstream call, so no tokens are spent on traffic no budget would ever have counted. | ```json { "error": { "message": "model 'llama-3.1-8b' has no price, so its cost cannot be counted against a budget. This deployment, or a budget that applies to this key, is configured to refuse traffic it cannot account for; set a price for this model to serve it.", "type": "invalid_request_error", "code": "model_unpriced", "param": "model" } } ``` `ignore` is the default so that upgrading Rolter cannot start refusing traffic for a deployment that has not priced its models yet. Once every routed model has a price, `block` never fires and turns the budget into a guarantee rather than a best effort. Unlike budgets themselves, `unpriced_policy` needs no Redis — being unpriced is a property of the config, not of a counter. It is enforced even when budgets are failing open. The built-in `fake-llm` model is answered by the gateway before any price lookup, so `block` never makes it unreachable. #### Per-budget override `unpriced_policy` on `[gateway]` is the deployment's floor. A single budget can ask for something **stricter** than that floor without imposing it on the rest of the deployment: ```toml [[budgets]] scope = "team" id = "platform" limit_usd = 500 period = "monthly" unpriced_policy = "block" # this team refuses what it cannot account for ``` In the dashboard the same field is on **Limits → Add budget**, as _Unpriced traffic_, and on the edit form of an existing budget. Leaving it at _Inherit deployment setting_ is what every budget does by default, and is how every budget created before this field existed behaves. Resolution is **most-restrictive-wins** (`ignore` < `warn` < `block`) across everything that applies to the request — the deployment setting and every budget matching the request's scope chain. Two consequences follow, and both are deliberate: - **A budget can only tighten.** A project budget set to `ignore` does not undo an org budget set to `block`, the same way it cannot raise the org's spend cap. A control a narrower scope could loosen would not be a control. - **The deployment setting is a floor, not merely a fallback.** If the deployment is set to `block`, no budget can opt back into serving unpriced traffic. A request that matches no budget at all is answered by the deployment setting, unchanged. The override lives on the budget row, so it works for budgets defined in the database through the control plane as well as for `rolter.toml` ones — a setting only `rolter.toml` could reach would silently do nothing for the deployments that use the dashboard. ### Seeing the spend so far Each budget card on **Budgets & Limits** shows how much of the cap is spent, how much is left and when the counter resets. The bar turns to the warning colour at 80% of the cap and to the danger colour once the cap is reached, and the same figures are written out beside it. The number is the counter the gateway enforces, read from Redis by the control plane through [`GET /api/v1/budgets/spend`](/api/budgets-and-rate-limits#spend-so-far), so the control plane needs the same `ROLTER_REDIS_URL` as the gateways. Without it, or while Redis is unreachable, the card says the spend is unavailable instead of showing zero. ## Step 3 — Configure rate limits A rate limit rejects requests with **HTTP 429** (plus a `Retry-After` header) when the number of requests or tokens in the trailing one-minute window exceeds the cap. Add one `[[rate_limits]]` block per scope you want to throttle. To protect one model rather than a caller, set `rpm`, `tpm` or `concurrency` on the route instead; see [Route limits](/concepts/providers-and-routes#route-limits). The scope level this limit applies to. One of `org`, `team`, `project`, `key`, `business_unit`, `customer`, or `user`. The UUID of the entity at the chosen scope. Maximum requests per minute. Omit to leave request count uncapped. Maximum tokens per minute (prompt \+ completion combined). Omit to leave token throughput uncapped. ```toml rolter.toml # Cap a single virtual key to 600 req/min and 150k tok/min [[rate_limits]] scope = "key" id = "vk-uuid" rpm = 600 tpm = 150000 # Cap an entire project to 2000 req/min (no token cap) [[rate_limits]] scope = "project" id = "project-uuid" rpm = 2000 ``` At least one of `rpm` or `tpm` must be set. A `[[rate_limits]]` block with neither field never actually caps anything — Rolter will warn about this during config validation. Like budgets, rate limits follow **most-restrictive-wins** across the scope chain. A per-key limit can throttle a client well below the project-level ceiling, and an org-level limit acts as a global backstop. ## Seeing every cap in the organization The **Budgets & Limits** screen opens on your current project. Pick **All scopes** as the scope type to list every budget and rate limit you may read across the organization, grouped under the scope each belongs to and named, not shown as an id. Pick one scope type to filter the list, and then one scope to narrow it to that scope. A cap is created on a picked scope, so the add buttons stay off until one is picked. The spend-so-far figure is read per scope and appears once a scope is picked. You see the caps your role allows, and the same rule applies as when you pick scopes one at a time: a project member sees the caps of their project, its team and the organization. The API behind the list is in [Budgets & rate limits](/api/budgets-and-rate-limits). ## Changing a budget or rate limit Budgets and rate limits stored in the database are edited in place. On the dashboard's **Budgets & Limits** screen, the pencil on a budget or rate-limit card opens the same form used to create one, filled in with the current values. Only the fields you change are sent. On a rate limit, clearing a field lifts that cap, and at least one of the two has to stay. The period is picked from **Daily**, **Monthly** and **Lifetime**, the windows the gateway has. They follow the calendar in UTC, so there is no rolling seven-day budget. A budget stored with a period the gateway does not recognise, such as `7d` from an older release, is enforced as monthly. Its card keeps the stored text in a warning badge and says so, the Providers screen lists it among the config problems, and its edit form opens with no period selected and cannot be saved until you pick one of the three windows. The API accepts a few more spellings of the same three windows; see [Periods](/api/budgets-and-rate-limits#periods). Over the API the same edit is `PATCH /api/v1/budgets/{id}` or `PATCH /api/v1/rate-limits/{id}`. See [Budgets & rate limits API](/api/budgets-and-rate-limits) for the body and its rules. Editing takes the `admin` role on the scope the cap lives on, the same as creating or deleting one. An edit is a single write, so a gateway sees the old cap or the new one and never a moment with no cap on the scope. That is the reason to prefer it over deleting a budget and creating a new one. What happens to spend already counted: - **Changing the limit** keeps it. Spend is counted per scope and period window, not per budget row, so raising a $500 monthly cap to $750 halfway through the month leaves the $300 already spent in place. - **Changing the period** moves the budget onto the new window's counter, for example today's for a switch to `daily`. The bin on a card deletes that budget or rate limit, and asks first. The dialog names the cap, the amount and period for a budget or the caps for a rate limit, since a scope can hold several, and says which scope it leaves without it. Once you confirm and the gateway picks up the change, that scope runs without the cap: uncapped, unless another budget or rate limit covers it. Budgets defined in `rolter.toml` are edited in the file, and a gateway running from that file needs a restart to apply them; see [Hot Reload](/configuration/hot-reload). A card shows a budget's period in words when it is one the gateway recognises: `daily` (stored as `daily`, `1d` or `24h`), `monthly` (`monthly`, or the dashboard's default `30d`) or `lifetime` (`total`, `lifetime` or `all`). Any other text is shown exactly as stored, in a warning badge. ## Realtime sessions A [`/v1/realtime`](/api/realtime) WebSocket session is held to the same budgets and rate limits, on the same scope chain as the key that opens it: - **Opening a session** is refused with `402` when a budget is spent (or `model_unpriced` applies), and with `429` when an `rpm` or `tpm` window is full. It counts as one request against `rpm`. - **While it runs**, each finished model response is priced from the usage the provider reports and charged to every applicable budget and `tpm` limit. Charges land every `[realtime] usage_flush_secs` (15 seconds by default), so a session that stays open for hours is accounted as it goes. - **When a budget runs out mid-session**, the session is closed at the next flush with an `error` event (code `insufficient_quota`) and a `1008` close frame, whether this session or other traffic spent the budget. A session can overspend by at most one flush window plus the response in flight. A full `tpm` window refuses the key's next session or request but does not close one already open. The process-local `[realtime]` caps (`max_connections`, `max_session_secs`, `idle_timeout_secs`) still apply on top: they protect the gateway, while budgets and rate limits are the per-tenant policy. ## How counters work Rolter keeps sliding-window counters in Redis — one counter per (scope, id, one-minute bucket). To estimate the trailing 60 seconds, a check adds the current bucket to a weighted share of the previous one. That share is weighted by how much of the previous minute still lies inside the window. This is an approximation: it treats the previous minute's requests as evenly spread, so a burst at the very end or start of a minute is weighted slightly low or high until the next bucket. Admission itself is exact. Checking every applicable limit and counting the request run as a single atomic step on the Redis server, so concurrent requests can never all take the same free slot. That holds whether they reach one gateway instance or several behind a load balancer. With `rpm = 1`, a burst of simultaneous requests admits exactly one. A request that one scope refuses is not counted against any other scope. `tpm` works differently, because token usage is only known once a response finishes. A request is refused once the trailing window has already reached the cap, and tokens are added afterwards. Requests admitted together just before the cap can overshoot it by their own usage. The `Retry-After` response header on a 429 tells the client how many seconds to wait before the trailing window slides enough to admit the next request. Spend and tokens are recorded from what the **provider billed**, not from what the caller received. If an output guardrail or a `post_response` plugin blocks an answer the provider already generated, the caller gets a `403`, but the tokens and cost still count toward every budget and `tpm` limit in scope. Usage is read before any policy runs, so a plugin that rewrites the `usage` object cannot lower the bill. A response-cache hit costs nothing, blocked or not. See [Withheld responses](/observability/logs-and-cost#withheld-responses). ### When Redis restarts or drops the connection While Redis is unreachable, budgets and rate limits **fail open**: requests pass, and spend and token counts are not recorded. The response cache also misses on every lookup. A Redis outage never takes the gateway down. When Redis is reachable again, each gateway reconnects by itself. You don't need to restart it or push a config change. Counters that Redis kept through the outage are enforced again from the next request. That covers a restart with persistence, a failover, or a connection that was only dropped (`CLIENT KILL`, or an idle timeout on a proxy in between). - **Reconnects are paced.** After a failed attempt, each gateway waits before trying again: 100 ms at first, doubling up to 5 s. In the meantime requests fail open at once, so there is no per-request reconnect storm and no added latency beyond one connection attempt (at most 1 s). - **Spend can go unrecorded while a connection drops.** A spend or token update that was in flight when the connection dropped is not retried, because retrying could count it twice. At most about one request's worth per gateway goes unrecorded at that moment. Admission checks are retried on the new connection, so the first request after a drop is still checked against the limits. - **The logs show the fail-open window.** It starts at `redis connection lost` or `redis unavailable; failing open` and ends at `redis connection re-established; enforcement resumed`. The `consumer` field names the budget enforcer, the rate limiter or the response cache. ## Complete example The following snippet brings together pricing, a budget, and a rate limit for a production setup: ```toml rolter.toml # Per-model pricing [[model_prices]] model = "gpt-4o" input_per_mtok = 2.5 output_per_mtok = 10.0 cached_input_per_mtok = 1.25 # Org-wide monthly spend cap [[budgets]] scope = "org" id = "acme-org-uuid" limit_usd = 500.0 period = "monthly" # Tighter cap for the analytics project [[budgets]] scope = "project" id = "analytics-project-uuid" limit_usd = 50.0 period = "monthly" # Per-key throughput limit for a high-volume client [[rate_limits]] scope = "key" id = "analytics-vk-uuid" rpm = 600 tpm = 150000 ```