Upgrading Rolter should be a change of image tag. Your rolter.toml is carried forward for you, and skipping releases costs nothing — you do not have to step through intermediate versions to get from an old build to a current one.
This page covers the config half of an upgrade. The database half is separate and already automatic: schema migrations are embedded in the binary and applied on start, and they are append-only, so the same rules apply there.
Preview an upgrade before you run it#
Run the new build against your existing config with --migrations:
It reads the file and reports what the new build would do with it. Nothing is written — not by this command, and not by the gateway at boot.
The second step does not retire anything. api_key, api_key_env and egress_proxy keep working in every file, forever, so a config can still be rolled back onto an older build; they are only deprecated, and Rolter logs a warning that names the plural spelling when it loads a file that still uses them.
The listing is exhaustive. Anything not named there is untouched.
A pending migration is not a fault. It is reported as a warning so that a CI job running rolter check --strict can notice a config drifting behind, but a config that is behind starts normally and serves traffic normally.
What schema_version is#
A rolter.toml may carry a version stamp at the top:
It tells Rolter which spelling of the config the file was written for. You do not have to add it. A file without one is treated as the oldest version, which is correct for every config written before the stamp existed.
Skipping releases#
Migrations form a chain, and upgrading applies every step from your file’s version up to the running build’s. Going from version 1 to version 4 runs the same three steps in the same order as three separate upgrades would, in one pass.
There is no version you have to stop at on the way.
Rolling back#
Rolling back to an older Rolter is supported, and the config will not be what stops you.
Rolter’s config types never reject a key they do not recognise. An older build reading a newer file ignores what it has not heard of and starts. The same holds for the stamp itself: a build that meets a schema_version from the future loads the file unchanged rather than refusing.
That last clause is the part to plan around. A rollback starts, but any setting that only the newer build understood stops taking effect, silently — the value is in the file and nothing reads it. Check the release notes for what you configured between the two versions.
Database migrations do not roll back. Rolter’s schema migrations are forward-only, so an older binary may meet a database schema newer than it expects. Restore from a backup taken before the upgrade if you need to go back across a migration — see Backup & restore.
Upgrading a multi-replica deployment#
During a rolling upgrade your control plane and your gateways are briefly different builds. Config migration is per-process and happens at load, so each replica carries its own file forward independently and a mixed fleet is not a problem for the config.
The control-plane-to-gateway snapshot is negotiated. Each gateway states the snapshot format it reads every time it polls, and the control plane serves that format or refuses with 406 Not Acceptable. A mismatched pair is therefore either compatible or refused out loud, never a config the gateway applies in part. Most releases do not change the format at all, and the release notes say when one does.
Upgrade order#
- Upgrade the control plane first, then the gateways. Upgrade every control-plane replica, then every gateway. A gateway may lag behind the control plane, but it may not lead it.
- With the Helm chart, one
helm upgradedoes this, even though both Deployments move together. When the format changed, a new gateway pod that starts before a new control plane is ready answers/readyzwith503 snapshot_format_unsupported. It stays out of the Service and the rollout waits, while the old pods keep serving. While the rollout waits, capacity drops by at most the Deployment’smaxUnavailable.progressDeadlineSecondsmay report the rollout as stalled even though it then completes on its own. - Do not turn on what a release introduces until every gateway runs it. An older gateway leaves out any route, key or provider that uses the new feature, and names it in its log. For a deployment-wide setting, it stops receiving config changes until it is upgraded.
- Skipping releases. A gateway can skip any number of releases. The control plane can skip releases as long as it still serves the oldest gateway’s format. The Deprecations section of the release notes lists each format that is going away, and when.
- Rolling back goes in reverse: first the gateways, then the control plane. Rolling the control plane back across a database migration needs the restore described in Rolling back.
A control plane serves an older format for at least two minor releases and 90 days after a newer one replaces it. A gateway newer than its control plane is not supported. It works when the format did not change between the two builds, and is refused when it did. See Versioning.
What a mismatch looks like#
On each refused gateway you get one line when the refusal starts, then the usual config snapshot poll failed warning on every poll:
- The
rolter_snapshot_format_refusedgauge reads1, androlter_config_reload_failures_totalkeeps climbing.rolter_snapshot_formatis the format of the snapshot the gateway holds. See Metrics. - A gateway that never received a snapshot answers
/readyzwith503 snapshot_format_unsupported. A gateway that did receive one stays ready and keeps serving its last config. It stops picking up changes, revoked keys included, until the mismatch is fixed. - The control plane logs a
warnthat names the node, its build and both format ranges, once per node. On the Cluster screen, the node shows as not converged once the config moves past the version it holds.
A gateway built before negotiation existed shows only the line it already writes, config snapshot poll failed: snapshot endpoint returned 406 Not Acceptable:, followed by the message and what to do.
A config that the older format cannot express at all is refused with snapshot_format_inexpressible instead, and the gateway behaves the same way. This happens only for a deployment-wide setting the older gateways cannot carry. A route, key or provider they cannot carry is left out instead (rule 3 above).
Captured payloads from older gateways#
Roll the gateways out before the control plane when you upgrade from v0.1.0 or earlier with payload capture on.
ClickHouse migrations under clickhouse/ add columns that newer gateways write, for example log_id on request_logs and request_payloads (012_request_log_key.sql). Gateways insert with input_format_skip_unknown_fields=1, so a gateway upgraded before the migration is applied keeps logging and simply drops the new field until it is. Apply the migration to keep the new column filled; until you do, captured bodies join their log rows the old way, by request id and timestamp.
Eight columns of request_logs are added by files an existing deployment applies by hand, because ClickHouse only runs the files under clickhouse/ when its data directory is first created: 015_upstream_attempts.sql (upstream_status and attempts), 016_lifecycle_operation.sql (lifecycle_operation) 017_cache_write_1h.sql (cache_write_1h_tokens), 018_guardrail_events.sql (guardrail_events, what each guardrail rule did to a request) 019_plugin_events.sql (plugin_events, what each plugin did) 021_request_log_seq.sql (seq, the gateway’s per-process start sequence that orders requests inside one millisecond) and 023_fallback_reason.sql (fallback_reason, the failure that sent a request on to a fallback target). For the bundled compose stack, run a file with docker compose -f docker/docker-compose.yml exec -T clickhouse clickhouse-client --multiquery < clickhouse/017_cache_write_1h.sql; every statement is idempotent, so running a file twice is harmless.
030_request_outcome.sql adds outcome the same way: what happened to each request, by name, so a refusal is selected by its reason (see The outcome of each request). Until it is applied a gateway drops the field and LLM Logs reads every row as Not recorded; rows written before it read unknown afterwards too.
031_request_shape.sql adds five request-shape columns the same way (has_tools, has_image_input, has_audio_input, has_response_schema, turns): what a request asked the model to do beyond its tokens, which the savings advisor reads to leave out requests a cheaper model may not serve. Until it is applied a gateway drops the fields and the advisor’s report says the shape is not recorded.
022_param_adjustments.sql adds param_adjustments the same way: the request parameters a translated request did not forward as sent (see Adjusted parameters). Until it is applied a gateway leaves the field out of the rows it writes and loses only those entries; the x-rolter-param-adjustments response header does not depend on it.
None of these files has to come before the control plane, for any of the fifteen columns they add (the eight above, param_adjustments, outcome and the five request-shape columns). LLM Logs and GET /api/v1/analytics/invocations look at which of them the table has and read a default for each one it lacks, so a control plane newer than the file lists requests as usual: upstream_status, attempts, cache_write_1h_tokens, seq, has_tools, has_image_input, has_audio_input, has_response_schema and turns read 0, lifecycle_operation and fallback_reason read an empty string, outcome reads unknown, and guardrail_events, plugin_events and param_adjustments read an empty list, exactly what a row written before the column reads once the file is applied. The control plane asks ClickHouse at most once a minute while a column is missing, so applying a file shows on the next page within a minute, without a restart. A control plane from before this behaviour (it arrived with #2903) names upstream_status, attempts and lifecycle_operation unconditionally and cannot list requests until 015 and 016 are applied, so apply them first if you are moving to one of those builds.
Apply the files to keep the data, not to be allowed to upgrade. A gateway that gets there first is safe: it inserts with input_format_skip_unknown_fields=1 and drops a field the table has no column for, so every request is still logged and priced. Until you apply a file the dropped fields are the ones it adds: the upstream status and attempts of a request, which call on a stored response a row records, and the 1 hour share of a cache write (only for a request that wrote to the 1 hour cache; the row’s cost is computed by the gateway and is the same either way). Until 017 is applied, LLM Logs shows no Cache write by lifetime line in a row’s detail, since the share it splits the write by is 0; once the file is applied it appears for every new request that wrote to the 1 hour cache. See Cache writes.
The Logs screen attaches a captured body to its log row by the request id and the row’s timestamp. Gateways from v0.1.0 and earlier left both timestamps to ClickHouse, which stamped the log row and the payload row separately, so they never match. Once the control plane is upgraded, bodies those gateways captured stop showing in the log drawer, and they stay hidden until payload_retention_hours removes them (7 days by default). Nothing is exposed and nothing is lost from the log rows themselves; only the payload panel goes empty for those requests. Upgrading the gateways first keeps the window of new bodies written that way as short as the rollout.
Model price rates#
The migration that adds 0084_model_price_rate_checks.sql makes the database refuse a model price whose rate is negative or NaN. Before it, PUT /api/v1/model-prices accepted both: a negative rate turned a request into a credit against its budget, and NaN was stored but could not be read back, so the model was priced at zero. The API refuses both now (400, naming the field), and so does a rolter.toml with a negative rate. It also refuses a rate of a million or more, which the numeric(12, 6) columns cannot hold and which used to be a 500.
The migration cannot fail on a database that already holds such a row. It adds each check as NOT VALID, which binds every insert and update from then on and leaves the rows already there alone, then validates the ones it can. A database with no bad row ends with all four checks validated and needs nothing from you. A check whose column does hold a bad row stays NOT VALID, and the migration prints a notice naming it.
Until you correct the row, Rolter leaves it out rather than guess a price for it. The model is unpriced, not free: its cost_usd is recorded as 0 with unpriced = 1, and unpriced_policy and any budget’s override decide what that means, exactly as for a model you never priced. The omission is listed by GET /api/v1/config/problems, and logged by the control plane as it builds the snapshot.
To see whether any check is still unvalidated, and which rows break it:
Correct each row on the Pricing screen (saving a model’s price replaces its input, output and cached-input rates), or delete the price to leave the model unpriced on purpose. Then validate whatever the first query still shows as convalidated = false:
The four names are model_prices_input_per_mtok_rate, model_prices_output_per_mtok_rate, model_prices_cached_input_per_mtok_rate and model_prices_cache_write_per_mtok_rate. Validating is optional: an unvalidated check protects new writes just the same. It only changes what convalidated reports.
Route advanced settings#
A route’s advanced block used to store, validate and display several settings that the gateway never read, and the dashboard and these docs described them as working (#2924). This release applies the ones it can and removes the rest. Both halves change something for a route that already carries them, so look before you roll out.
Settings that start to apply. A route that carries limits.timeout_secs, limits.retries or headers now uses them. Until now they changed nothing. A saved timeout that is shorter than a model’s time to first byte now ends those requests early, a retry budget replaces [retry].max_retries for that route (a value above 10 is applied as 10), and every header is sent to the upstream. To list the routes that set any of them:
A header that names a credential (authorization, x-api-key, api-key, x-goog-api-key) or the request’s framing (host, content-length) is not sent. The snapshot leaves it out and GET /api/v1/config/problems names it; the route keeps serving. The credential belongs on the provider.
Settings that are gone. base_url, insecure_tls, pricing and the context_window limit never had an effect, and requests were served exactly as if they were not set. A route’s base_url never redirected a request, and insecure_tls = true never turned off certificate verification. If you believed either, nothing changes on the wire, but the setting is gone: point a route at a second endpoint with a second provider, and trust a private CA with [tls].ca_bundles or the provider’s ca_bundles. To list the routes that still carry one:
Nothing is migrated, and a stored value that is no longer read keeps loading without complaint. The routes API drops these keys from a write instead of refusing it, so an older client or script still saves. Saving a model’s sheet in the dashboard sheds them from that route. To clear them all at once:
A rolter.toml that sets any of them under [routes.advanced] still loads. rolter check and the gateway’s startup log report each one as an unknown key.
Route limits. limits.rpm, limits.tpm and limits.concurrency are enforced again, this time with a reader (#2936): a request over a route’s cap is refused with 429 and Retry-After. Migration 0090_clear_unread_route_limits.sql removes the three keys from every stored route first, because a value typed into the old form while nothing read it would otherwise start refusing traffic on the day you deploy. A route that should be limited has to be given its cap again after the upgrade, from the model sheet or the routes API. A rolter.toml that sets them is read as written, so check a file you carried over from an older release. Enforcement needs a gateway that has the reader, so finish the upgrade, gateways included, before you set a cap (rule 3): until a gateway restarts on the new version, the cap is stored and not applied. rpm and tpm need redis_url to be enforced; concurrency does not. See Route limits.
Security and egress settings#
The migration 0086_security_egress_settings_shape.sql makes the database refuse three settings whose stored shape the API already enforced: required_headers on the security settings, egress_proxies on a provider, and injected_headers on the client settings. Before it, a value written around the API with SQL, a seed or an import could be read back as something weaker than what you meant (#2943), and now each is handled on purpose:
| Setting | If a stored value cannot be read |
|---|---|
required_headers (security settings) | The control plane publishes no snapshot. Gateways keep the config they last received, rule included, and a gateway that restarts meanwhile serves nothing until the rule is repaired. The effective-config views in the dashboard return an error, while the Security screen and /api/v1/config/problems keep working; Effective config lists those problem lines under its error. |
egress_proxies (a provider) | That provider is left out of the snapshot, with every route target and group member that named it, so its traffic never goes out around the proxy pool. A route with no other target is not served. |
injected_headers (client settings) | The entries that cannot be read are not injected, and the rest are. This setting gates nothing. |
Each case is listed by GET /api/v1/config/problems and logged by the control plane (a refused snapshot also names the setting in each gateway’s config snapshot poll failed log line), naming the setting, the provider or the header, and never a value. A header’s value is not repeated because it may be a shared secret. A required header counts as unreadable when the setting is not an object, when a value is not a string or is blank, when a name is blank, or when one header is listed twice with different values.
The migration cannot fail on a database that already holds such a row. It adds each check as NOT VALID, which binds every insert and update from then on and leaves the rows already there alone, then validates the ones it can. A database with no bad row ends with all three checks validated and needs nothing from you. A check whose table does hold a bad row stays NOT VALID, and the migration prints a notice naming it. To see whether any is still unvalidated:
Find the rows that break them, correct each one, then validate whatever the first query still shows as convalidated = false. Save the security settings again from the Security screen, or PUT /api/v1/security-settings, to repair required_headers; set a provider’s proxy pool on the Providers screen; save the client settings again for injected_headers.
Validating is optional: an unvalidated check protects new writes just the same. It only changes what convalidated reports.
Upstream TLS trust store#
Rolter now verifies upstream HTTPS against the host’s system trust store (/etc/ssl/certs, or the bundle SSL_CERT_FILE names) instead of a copy of the public roots built into the binary. The published images ship ca-certificates, so they need nothing from you. A binary you run on a host or in an image of your own with no CA store fails every upstream handshake until you install ca-certificates or point SSL_CERT_FILE at a PEM bundle. ROLTER_CA_BUNDLE and [tls].ca_bundles still add private CAs on top of that store.
MCP tool manifests and tool groups#
Two MCP rules that were stored but not enforced are enforced from this release:
- Tool groups. A server that an enabled tool group names serves only the tools those groups list. A call to any other tool is refused with
403mcp_tool_denied. Check your enabled groups before you upgrade. - Declared manifests. A server that declares tools serves only those once an organization’s saved MCP settings refuse unlisted tools. To keep upgrades from refusing calls, an organization that never saved its MCP settings allows unlisted tools, and the upgrade sets Allow unlisted tools on for every organization that had saved them. Turn it off on MCP → MCP Settings to enforce a server’s manifest.
A checklist#
- Back up the control-plane database, and confirm you have
ROLTER_KEKoutside it. - Run
rolter check --config rolter.toml --migrationswith the new binary and read the plan. - Roll out. Config migrates in memory on start; database migrations apply on the control plane’s start.
- Confirm the gateway serves traffic and the dashboard lists your providers.