Rolter gives you two complementary levers for controlling how much your gateway spends and how fast it moves traffic: spend budgets block requests once a cumulative USD ceiling is reached, and rate limits reject requests that exceed per-minute throughput thresholds. Both systems operate across the same scopes — the four-level tenancy hierarchy (org → team → project → key) plus the two cost-attribution dimensions a key rolls up to (business unit, customer) — and apply a most-restrictive-wins policy when multiple rules match a single request.
Budgets and rate limits both require Redis (ROLTER_REDIS_URL). Rolter uses Redis for distributed counters so limits remain accurate across multiple gateway instances. Without Redis, both budgets and rate limits fail open — they are not enforced and all requests pass through.
Step 1 — Define model prices#
Before budgets can track spend, Rolter needs to know the cost of each model’s tokens. Add one [[model_prices]] block per model with rates per million tokens, denominated in the currency you name (USD by default).
The public model name as it appears in your [[routes]] configuration.
Cost in USD per million input (prompt) tokens.
Cost in USD per million output (completion) tokens.
Cost per million cache-hit input tokens. Defaults to input_per_mtok when omitted.
Cost per million input tokens written to the provider’s prompt cache. Defaults to input_per_mtok when omitted, so a price without it charges a cache write as ordinary input. See Cache writes.
Cost per million input tokens written to the provider’s 1 hour prompt cache, for a provider that reports the split (Anthropic). Defaults to cache_write_per_mtok when omitted, and from there to input_per_mtok. See Cache writes.
Cost of one generated image, for a model billed per image. Added to the request’s token cost, and counted against budgets like it. See Images, audio and speech.
Cost of one minute of input audio on a transcription or translation. Read from the duration the response states.
Cost per million characters of text sent to a text-to-speech model, counted from the request’s input.
Price per million audio input tokens, for a realtime or audio model that bills the audio it is sent apart from the text. The share of the prompt that is audio is the one the response’s usage names (see Audio and text tokens). Defaults to input_per_mtok.
Price per million audio output tokens. Defaults to output_per_mtok.
Price per million cached audio input tokens, for a provider that says how many of its cache hits were audio (a realtime session does). Defaults to cached_input_per_mtok, and from there to the audio input rate.
Currency the rates above are denominated in. Any code works — ISO-4217, a crypto unit, an internal credit — as long as [currency.rates] can convert it to the base currency.
Default: "USD"
Rates are written as ordinary numbers here, but Rolter computes with them in
exact decimal arithmetic, not binary floating point. A rate of 0.1 means
a tenth rather than the nearest binary approximation of one, so per-request
costs and currency conversions carry no rounding drift, and a budget’s
spend >= limit test has an exact boundary rather than one that depends on
the order requests happened to be summed in.
One caveat while this rolls out: the spend counter itself still accumulates in Redis in floating point, so a long-running counter can still drift very slightly from the exact sum of its requests. Making the accumulation exact too is tracked in #967.
Models without a [[model_prices]] entry log a cost_usd of 0 and an unpriced flag of 1 per request. They can still be used, but their spend does not count toward any budget. Add a price entry for every model you want budgets to track, and see Unpriced traffic below for how to make that gap fail closed instead of open.
Pricing in another currency#
Spend and budgets accumulate in a single base currency. A model priced in anything else is converted at a rate you configure, once, when the gateway loads its config — so pricing in EUR while budgeting in USD is exact rather than off by the FX rate.
A rate is how many base units one unit of that currency is worth. The base itself is always 1.0 and needs no row. Codes are open strings rather than a fixed list, so adding BTC or an internal credit unit means adding a rate row — no new release required.
A model priced in a currency with no rate is a config error, not a warning: the gateway refuses to start and the dashboard refuses the write. Charging face value would bill the wrong number and charging zero would bill nothing, so neither is offered as a fallback.
Rates are operator-supplied and static, which is deliberate — Rolter must run air-gapped, so it never reaches out to an FX feed.
What the dashboard offers#
The currency chooser on a model’s pricing section is your rate table, not a built-in list: it shows the base currency followed by every code in [currency.rates]. Adding rates = { RUB = 0.011 } makes RUB selectable on the next control-plane restart, with no dashboard release involved.
That set is read from GET /api/v1/currency, which answers from the bootstrap config rather than the database — the store keeps a currency per model price and no rate table, so this is the one place that knows what a deployment can convert. It is also exactly the set the CRUD API accepts, so the chooser cannot offer a code the save would reject.
A price already stored in a code that is no longer in the table still displays, flagged as unconvertible on both the Pricing screen and the model sheet. Its spend is missing from base-currency totals and budgets until you add the rate back.
Every amount the dashboard renders — LLM Logs, budget cards, team cards, the spend tiles — is formatted in the base currency from that same endpoint, so a deployment budgeting in EUR reads € rather than a hardcoded $, and the budget form’s Limit field is labelled EUR rather than USD. Model prices are the exception: they are shown in the currency the price row was written in, which is what you typed. Amounts, dates and counts all follow the dashboard’s language rather than the browser’s, so the separators in a number never disagree with the copy beside it.
Step 2 — Configure spend budgets#
A spend budget blocks incoming requests with HTTP 402 once the cumulative cost for a scope reaches limit_usd within the configured period. Add one [[budgets]] block per scope you want to cap.
The scope level this budget applies to. One of org, team, project, key, business_unit, customer, or user.
The UUID of the entity at the chosen scope (org ID, team ID, project ID, virtual key ID, business unit ID, or customer ID).
business_unit and customer are not part of the org → team → project chain: they are the cost-attribution dimensions a virtual key rolls its spend up to, so a budget on one caps everything attributed to it regardless of which project the traffic came through. The control plane API and the dashboard’s Budgets & Limits screen accept all six scopes; in the dashboard, picking one of these two lists your org’s business units or customers rather than asking for a UUID. Picking a virtual key lists the current project’s keys the same way. While the list loads the field shows a placeholder, and if the control plane cannot return it (keys, business units or customers alike) the screen says so under the scope picker with a Try again button, and the field takes the UUID meanwhile.
A budget on a person. Members mint their own keys, as many as they like, so a key cap bounds one key and a project cap is shared by everyone in the project. A user budget (and a user rate limit) counts every key that person minted, however many, which is how “each engineer may spend $50 a month” is said. It is narrowed with within to the keys they minted in one org ({ org = "…" }) or one project ({ project = "…" }); without it, it counts their keys everywhere, and only a superadmin may set that. It composes like every other scope: the most restrictive matching budget wins, and the 402 names the user budget (budget exceeded for User '…') when it is the one that tripped. A key nobody minted (a config-file key, or one an admin created for a project) has no owner, so no user cap reaches it. In the dashboard, pick User as the scope type on Budgets & Limits, choose the person, and the new budget or rate limit asks which keys it counts: the ones minted in one organization (the default), in one project, or, for a superadmin only, every key. Rows of this kind are named after the person everywhere on the screen. The person sees their own caps and what is left of them under Account; deleting the person deletes their caps.
The maximum cumulative spend in USD for the period before requests are blocked.
The window over which spend accumulates. One of monthly, daily, or total. monthly and daily windows reset automatically; total is a lifetime cap that never resets.
Default: "monthly"
When a request’s scope chain matches multiple budget rules (e.g. both an org budget and a project budget), Rolter applies most-restrictive-wins: the request is blocked as soon as any matching budget is exhausted.
Set an org-level budget as a hard ceiling to protect against unexpected runaway costs, then add project- or key-level budgets to allocate share to individual teams or clients.
Unpriced traffic#
A budget can only block spend it can see. A model with no [[model_prices]] entry accrues zero, so it can never exhaust a budget however much is served — the control fails open on exactly the models you most recently added and are least likely to have priced.
unpriced_policy is the deliberate decision about that. It is set deployment-wide, since it governs what the gateway may serve at all rather than the state of any one counter, and an individual budget may then ask for something stricter — see Per-budget override:
| policy | behaviour |
|---|---|
ignore | Serve it and accrue nothing. Budgets silently do not cover it. |
warn | Serve it, but log the model once per hour so an unenforceable budget names itself. |
block | Refuse at admission with HTTP 402 and error code model_unpriced, naming the model. The check runs before the upstream call, so no tokens are spent on traffic no budget would ever have counted. |
ignore is the default so that upgrading Rolter cannot start refusing traffic for a deployment that has not priced its models yet. Once every routed model has a price, block never fires and turns the budget into a guarantee rather than a best effort.
Unlike budgets themselves, unpriced_policy needs no Redis — being unpriced is a property of the config, not of a counter. It is enforced even when budgets are failing open.
The built-in fake-llm model is answered by the gateway before any price lookup, so block never makes it unreachable.
Per-budget override#
unpriced_policy on [gateway] is the deployment’s floor. A single budget can
ask for something stricter than that floor without imposing it on the rest
of the deployment:
In the dashboard the same field is on Limits → Add budget, as Unpriced traffic, and on the edit form of an existing budget. Leaving it at Inherit deployment setting is what every budget does by default, and is how every budget created before this field existed behaves.
Resolution is most-restrictive-wins (ignore < warn < block) across
everything that applies to the request — the deployment setting and every
budget matching the request’s scope chain. Two consequences follow, and both
are deliberate:
- A budget can only tighten. A project budget set to
ignoredoes not undo an org budget set toblock, the same way it cannot raise the org’s spend cap. A control a narrower scope could loosen would not be a control. - The deployment setting is a floor, not merely a fallback. If the
deployment is set to
block, no budget can opt back into serving unpriced traffic.
A request that matches no budget at all is answered by the deployment setting, unchanged.
The override lives on the budget row, so it works for budgets defined in the
database through the control plane as well as for rolter.toml ones — a
setting only rolter.toml could reach would silently do nothing for the
deployments that use the dashboard.
Seeing the spend so far#
Each budget card on Budgets & Limits shows how much of the cap is spent, how much is left and when the counter resets. The bar turns to the warning colour at 80% of the cap and to the danger colour once the cap is reached, and the same figures are written out beside it. The number is the counter the gateway enforces, read from Redis by the control plane through GET /api/v1/budgets/spend, so the control plane needs the same ROLTER_REDIS_URL as the gateways. Without it, or while Redis is unreachable, the card says the spend is unavailable instead of showing zero.
Step 3 — Configure rate limits#
A rate limit rejects requests with HTTP 429 (plus a Retry-After header) when the number of requests or tokens in the trailing one-minute window exceeds the cap. Add one [[rate_limits]] block per scope you want to throttle. To protect one model rather than a caller, set rpm, tpm or concurrency on the route instead; see Route limits.
The scope level this limit applies to. One of org, team, project, key, business_unit, customer, or user.
The UUID of the entity at the chosen scope.
Maximum requests per minute. Omit to leave request count uncapped.
Maximum tokens per minute (prompt + completion combined). Omit to leave token throughput uncapped.
At least one of rpm or tpm must be set. A [[rate_limits]] block with neither field never actually caps anything — Rolter will warn about this during config validation.
Like budgets, rate limits follow most-restrictive-wins across the scope chain. A per-key limit can throttle a client well below the project-level ceiling, and an org-level limit acts as a global backstop.
Seeing every cap in the organization#
The Budgets & Limits screen opens on your current project. Pick All scopes as the scope type to list every budget and rate limit you may read across the organization, grouped under the scope each belongs to and named, not shown as an id. Pick one scope type to filter the list, and then one scope to narrow it to that scope. A cap is created on a picked scope, so the add buttons stay off until one is picked. The spend-so-far figure is read per scope and appears once a scope is picked.
You see the caps your role allows, and the same rule applies as when you pick scopes one at a time: a project member sees the caps of their project, its team and the organization. The API behind the list is in Budgets & rate limits.
Changing a budget or rate limit#
Budgets and rate limits stored in the database are edited in place. On the dashboard’s Budgets & Limits screen, the pencil on a budget or rate-limit card opens the same form used to create one, filled in with the current values. Only the fields you change are sent. On a rate limit, clearing a field lifts that cap, and at least one of the two has to stay.
The period is picked from Daily, Monthly and Lifetime, the windows
the gateway has. They follow the calendar in UTC, so there is no rolling
seven-day budget. A budget stored with a period the gateway does not recognise,
such as 7d from an older release, is enforced as monthly. Its card keeps the
stored text in a warning badge and says so, the Providers screen lists it among
the config problems, and its edit form opens with no period selected and cannot
be saved until you pick one of the three windows. The API accepts a few more
spellings of the same three windows; see
Periods.
Over the API the same edit is PATCH /api/v1/budgets/{id} or
PATCH /api/v1/rate-limits/{id}. See
Budgets & rate limits API for the body and its
rules. Editing takes the admin role on the scope the cap lives on, the same
as creating or deleting one.
An edit is a single write, so a gateway sees the old cap or the new one and never a moment with no cap on the scope. That is the reason to prefer it over deleting a budget and creating a new one. What happens to spend already counted:
- Changing the limit keeps it. Spend is counted per scope and period window, not per budget row, so raising a 750 halfway through the month leaves the $300 already spent in place.
- Changing the period moves the budget onto the new window’s counter, for
example today’s for a switch to
daily.
The bin on a card deletes that budget or rate limit, and asks first. The dialog names the cap, the amount and period for a budget or the caps for a rate limit, since a scope can hold several, and says which scope it leaves without it. Once you confirm and the gateway picks up the change, that scope runs without the cap: uncapped, unless another budget or rate limit covers it.
Budgets defined in rolter.toml are edited in the file, and a gateway running
from that file needs a restart to apply them; see
Hot Reload.
A card shows a budget’s period in words when it is one the gateway recognises:
daily (stored as daily, 1d or 24h), monthly (monthly, or the
dashboard’s default 30d) or lifetime (total, lifetime or all). Any
other text is shown exactly as stored, in a warning badge.
Realtime sessions#
A /v1/realtime WebSocket session is held to the same budgets and rate limits, on the same scope chain as the key that opens it:
- Opening a session is refused with
402when a budget is spent (ormodel_unpricedapplies), and with429when anrpmortpmwindow is full. It counts as one request againstrpm. - While it runs, each finished model response is priced from the usage the provider reports and charged to every applicable budget and
tpmlimit. Charges land every[realtime] usage_flush_secs(15 seconds by default), so a session that stays open for hours is accounted as it goes. - When a budget runs out mid-session, the session is closed at the next flush with an
errorevent (codeinsufficient_quota) and a1008close frame, whether this session or other traffic spent the budget. A session can overspend by at most one flush window plus the response in flight.
A full tpm window refuses the key’s next session or request but does not close one already open. The process-local [realtime] caps (max_connections, max_session_secs, idle_timeout_secs) still apply on top: they protect the gateway, while budgets and rate limits are the per-tenant policy.
How counters work#
Rolter keeps sliding-window counters in Redis — one counter per (scope, id, one-minute bucket). To estimate the trailing 60 seconds, a check adds the current bucket to a weighted share of the previous one. That share is weighted by how much of the previous minute still lies inside the window. This is an approximation: it treats the previous minute’s requests as evenly spread, so a burst at the very end or start of a minute is weighted slightly low or high until the next bucket.
Admission itself is exact. Checking every applicable limit and counting the request run as a single atomic step on the Redis server, so concurrent requests can never all take the same free slot. That holds whether they reach one gateway instance or several behind a load balancer. With rpm = 1, a burst of simultaneous requests admits exactly one. A request that one scope refuses is not counted against any other scope.
tpm works differently, because token usage is only known once a response finishes. A request is refused once the trailing window has already reached the cap, and tokens are added afterwards. Requests admitted together just before the cap can overshoot it by their own usage.
The Retry-After response header on a 429 tells the client how many seconds to wait before the trailing window slides enough to admit the next request.
Spend and tokens are recorded from what the provider billed, not from what the caller received. If an output guardrail or a post_response plugin blocks an answer the provider already generated, the caller gets a 403, but the tokens and cost still count toward every budget and tpm limit in scope. Usage is read before any policy runs, so a plugin that rewrites the usage object cannot lower the bill. A response-cache hit costs nothing, blocked or not. See Withheld responses.
When Redis restarts or drops the connection#
While Redis is unreachable, budgets and rate limits fail open: requests pass, and spend and token counts are not recorded. The response cache also misses on every lookup. A Redis outage never takes the gateway down.
When Redis is reachable again, each gateway reconnects by itself. You don’t need to restart it or push a config change. Counters that Redis kept through the outage are enforced again from the next request. That covers a restart with persistence, a failover, or a connection that was only dropped (CLIENT KILL, or an idle timeout on a proxy in between).
- Reconnects are paced. After a failed attempt, each gateway waits before trying again: 100 ms at first, doubling up to 5 s. In the meantime requests fail open at once, so there is no per-request reconnect storm and no added latency beyond one connection attempt (at most 1 s).
- Spend can go unrecorded while a connection drops. A spend or token update that was in flight when the connection dropped is not retried, because retrying could count it twice. At most about one request’s worth per gateway goes unrecorded at that moment. Admission checks are retried on the new connection, so the first request after a drop is still checked against the limits.
- The logs show the fail-open window. It starts at
redis connection lostorredis unavailable; failing openand ends atredis connection re-established; enforcement resumed. Theconsumerfield names the budget enforcer, the rate limiter or the response cache.
Complete example#
The following snippet brings together pricing, a budget, and a rate limit for a production setup: