> ## Documentation index
> Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt
> Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=<keywords or question>. Short keywords rank best.
> Use these to discover all available pages before exploring further.

# Upgrading Rolter

> How Rolter carries your configuration across a version boundary, how to preview an upgrade before you run it, and why skipping releases is safe.

Upgrading Rolter should be a change of image tag. Your `rolter.toml` is carried forward for you, and skipping releases costs nothing — you do not have to step through intermediate versions to get from an old build to a current one.

This page covers the config half of an upgrade. The database half is separate and already automatic: schema migrations are embedded in the binary and applied on start, and they are append-only, so the same rules apply there.

## Preview an upgrade before you run it

Run the new build against your existing config with `--migrations`:

```bash
rolter check --config rolter.toml --migrations
```

It reads the file and reports what the new build would do with it. Nothing is written — not by this command, and not by the gateway at boot.

```
config schema: version 1 -> 3, 2 migration(s) pending
  1 -> 2  rewrite the deprecated [[providers]] / [[provider_groups]] arrays as the tiered { readonly, default } tables (ADR-0022)
      providers: the deprecated [[providers]] array becomes [providers] readonly = [...]; 7 entries moved to the readonly tier
  2 -> 3  rewrite a provider's api_key / api_key_env as an api_keys entry and its egress_proxy as a one-element egress_proxies (#1720)
      providers.readonly[0].api_key_env: api_key_env becomes a single entry of api_keys; the provider resolves the same credential
```

The second step does not retire anything. `api_key`, `api_key_env` and `egress_proxy` keep working in every file, forever, so a config can still be rolled back onto an older build; they are only deprecated, and Rolter logs a warning that names the plural spelling when it loads a file that still uses them.

The listing is exhaustive. Anything not named there is untouched.

<Note>
  A pending migration is not a fault. It is reported as a warning so that a CI job running `rolter check --strict` can notice a config drifting behind, but a config that is behind starts normally and serves traffic normally.
</Note>

## `rolter check` runs the gateway's validation

From this release `rolter check --config` runs the same validation the gateway runs at startup and reports each problem it finds as an error: a route on an unknown provider, a duplicate route, a target with weight `0`, a budget whose limit is not positive, a route whose every target is a [fallback target](/reliability/fallback-targets), and the rest. Before, the gateway only logged these and served everything else, and `rolter check` let the file through.

The Helm chart's preflight init containers and the compose stack's preflight run `rolter check --strict`, so **a config with such a problem now stops the rollout** instead of starting a gateway that answers the affected routes with errors. Run the new binary's `rolter check --config rolter.toml` against your file before you upgrade, and fix what it lists. The `rolter.example.toml` the release ships, and the file `rolter init` writes from it, pass it.

## What `schema_version` is

A `rolter.toml` may carry a version stamp at the top:

```toml
schema_version = 3
```

It tells Rolter which spelling of the config the file was written for. You do not have to add it. A file without one is treated as the oldest version, which is correct for every config written before the stamp existed.

## Skipping releases

Migrations form a chain, and upgrading applies every step from your file's version up to the running build's. Going from version 1 to version 4 runs the same three steps in the same order as three separate upgrades would, in one pass.

There is no version you have to stop at on the way.

## Rolling back

Rolling back to an older Rolter is supported, and the config will not be what stops you.

Rolter's config types never reject a key they do not recognise. An older build reading a newer file ignores what it has not heard of and starts. The same holds for the stamp itself: a build that meets a `schema_version` from the future loads the file unchanged rather than refusing.

```
warn   config is newer than this build
       The file is stamped schema_version 5, and this build understands 3. It loads
       unchanged and keys this build does not know are ignored, so a rollback still
       starts — but settings written for the newer version are silently inactive.
```

That last clause is the part to plan around. A rollback starts, but any setting that only the newer build understood stops taking effect, silently — the value is in the file and nothing reads it. Check the release notes for what you configured between the two versions.

<Warning>
  Database migrations do **not** roll back. Rolter's schema migrations are forward-only, so an older binary may meet a database schema newer than it expects. Restore from a backup taken before the upgrade if you need to go back across a migration — see [Backup & restore](/deployment/backup-and-restore).
</Warning>

## Upgrading a multi-replica deployment

During a rolling upgrade your control plane and your gateways are briefly different builds. Config migration is per-process and happens at load, so each replica carries its own file forward independently and a mixed fleet is not a problem for the config.

The control-plane-to-gateway snapshot is negotiated. Each gateway states the snapshot format it reads every time it polls, and the control plane serves that format or refuses with `406 Not Acceptable`. A mismatched pair is therefore either compatible or refused out loud, never a config the gateway applies in part. Most releases do not change the format at all, and the release notes say when one does.

### Upgrade order

1. **Upgrade the control plane first, then the gateways.** Upgrade every control-plane replica, then every gateway. A gateway may lag behind the control plane, but it may not lead it.
2. **With the Helm chart, one `helm upgrade` does this**, even though both Deployments move together. When the format changed, a new gateway pod that starts before a new control plane is ready answers `/readyz` with `503 snapshot_format_unsupported`. It stays out of the Service and the rollout waits, while the old pods keep serving. While the rollout waits, capacity drops by at most the Deployment's `maxUnavailable`. `progressDeadlineSeconds` may report the rollout as stalled even though it then completes on its own.
3. **Do not turn on what a release introduces until every gateway runs it.** An older gateway leaves out any route, key or provider that uses the new feature, and names it in its log. For a deployment-wide setting, it stops receiving config changes until it is upgraded.
4. **Skipping releases.** A gateway can skip any number of releases. The control plane can skip releases as long as it still serves the oldest gateway's format. The **Deprecations** section of the release notes lists each format that is going away, and when.
5. **Rolling back goes in reverse:** first the gateways, then the control plane. Rolling the control plane back across a database migration needs the restore described in [Rolling back](#rolling-back).

A control plane serves an older format for at least two minor releases and 90 days after a newer one replaces it. A gateway newer than its control plane is not supported. It works when the format did not change between the two builds, and is refused when it did. See [Versioning](/community/versioning#control-plane-and-gateway).

### What a mismatch looks like

On each refused gateway you get one line when the refusal starts, then the usual `config snapshot poll failed` warning on every poll:

```
ERROR snapshot format refused: the control plane serves formats 2-3 and this gateway
      reads format 1 (snapshot_format_unsupported). Keeping the config applied at
      version 41. Upgrade this gateway, or roll the control plane back.
```

- The `rolter_snapshot_format_refused` gauge reads `1`, and `rolter_config_reload_failures_total` keeps climbing. `rolter_snapshot_format` is the format of the snapshot the gateway holds. See [Metrics](/observability/metrics).
- A gateway that never received a snapshot answers `/readyz` with `503 snapshot_format_unsupported`. A gateway that did receive one stays ready and keeps serving its last config. It stops picking up changes, revoked keys included, until the mismatch is fixed.
- The control plane logs a `warn` that names the node, its build and both format ranges, once per node. On the **Cluster** screen, the node shows as not converged once the config moves past the version it holds.

A gateway built before negotiation existed shows only the line it already writes, `config snapshot poll failed: snapshot endpoint returned 406 Not Acceptable:`, followed by the message and what to do.

A config that the older format cannot express at all is refused with `snapshot_format_inexpressible` instead, and the gateway behaves the same way. This happens only for a deployment-wide setting the older gateways cannot carry. A route, key or provider they cannot carry is left out instead (rule 3 above).

### Captured payloads from older gateways

Roll the gateways out before the control plane when you upgrade from v0.1.0 or earlier with [payload capture](/observability/log-settings#payload-capture) on.

ClickHouse migrations under `clickhouse/` add columns that newer gateways write, for example `log_id` on `request_logs` and `request_payloads` (`012_request_log_key.sql`). Gateways insert with `input_format_skip_unknown_fields=1`, so a gateway upgraded before the migration is applied keeps logging and simply drops the new field until it is. Apply the migration to keep the new column filled; until you do, captured bodies join their log rows the old way, by request id and timestamp.

Eight columns of `request_logs` are added by files an existing deployment applies by hand, because ClickHouse only runs the files under `clickhouse/` when its data directory is first created: `015_upstream_attempts.sql` (`upstream_status` and `attempts`), `016_lifecycle_operation.sql` (`lifecycle_operation`) `017_cache_write_1h.sql` (`cache_write_1h_tokens`), `018_guardrail_events.sql` (`guardrail_events`, what each guardrail rule did to a request) `019_plugin_events.sql` (`plugin_events`, what each plugin did) `021_request_log_seq.sql` (`seq`, the gateway's per-process start sequence that orders requests inside one millisecond) and `023_fallback_reason.sql` (`fallback_reason`, the failure that sent a request on to a [fallback target](/reliability/fallback-targets)). For the bundled compose stack, run a file with `docker compose -f docker/docker-compose.yml exec -T clickhouse clickhouse-client --multiquery < clickhouse/017_cache_write_1h.sql`; every statement is idempotent, so running a file twice is harmless.

`030_request_outcome.sql` adds `outcome` the same way: what happened to each request, by name, so a refusal is selected by its reason (see [The outcome of each request](/observability/logs-and-cost#the-outcome-of-each-request)). Until it is applied a gateway drops the field and **LLM Logs** reads every row as **Not recorded**; rows written before it read `unknown` afterwards too.

`031_request_shape.sql` adds five request-shape columns the same way (`has_tools`, `has_image_input`, `has_audio_input`, `has_response_schema`, `turns`): what a request asked the model to do beyond its tokens, which the [savings advisor](/observability/savings-advisor) reads to leave out requests a cheaper model may not serve. Until it is applied a gateway drops the fields and the advisor's report says the shape is not recorded.

`022_param_adjustments.sql` adds `param_adjustments` the same way: the request parameters a translated request did not forward as sent (see [Adjusted parameters](/api/overview#adjusted-parameters)). Until it is applied a gateway leaves the field out of the rows it writes and loses only those entries; the `x-rolter-param-adjustments` response header does not depend on it.

None of these files has to come before the control plane, for any of the fifteen columns they add (the eight above, `param_adjustments`, `outcome` and the five request-shape columns). **LLM Logs** and `GET /api/v1/analytics/invocations` look at which of them the table has and read a default for each one it lacks, so a control plane newer than the file lists requests as usual: `upstream_status`, `attempts`, `cache_write_1h_tokens`, `seq`, `has_tools`, `has_image_input`, `has_audio_input`, `has_response_schema` and `turns` read `0`, `lifecycle_operation` and `fallback_reason` read an empty string, `outcome` reads `unknown`, and `guardrail_events`, `plugin_events` and `param_adjustments` read an empty list, exactly what a row written before the column reads once the file is applied. The control plane asks ClickHouse at most once a minute while a column is missing, so applying a file shows on the next page within a minute, without a restart. A control plane from before this behaviour (it arrived with [#2903](https://github.com/rolter-ai/rolter/issues/2903)) names `upstream_status`, `attempts` and `lifecycle_operation` unconditionally and cannot list requests until `015` and `016` are applied, so apply them first if you are moving to one of those builds.

Apply the files to keep the data, not to be allowed to upgrade. A gateway that gets there first is safe: it inserts with `input_format_skip_unknown_fields=1` and drops a field the table has no column for, so every request is still logged and priced. Until you apply a file the dropped fields are the ones it adds: the upstream status and attempts of a request, which call on a stored response a row records, and the 1 hour share of a cache write (only for a request that wrote to the 1 hour cache; the row's cost is computed by the gateway and is the same either way). Until `017` is applied, **LLM Logs** shows no **Cache write by lifetime** line in a row's detail, since the share it splits the write by is `0`; once the file is applied it appears for every new request that wrote to the 1 hour cache. See [Cache writes](/observability/logs-and-cost#cache-writes).

The Logs screen attaches a captured body to its log row by the request id and the row's timestamp. Gateways from v0.1.0 and earlier left both timestamps to ClickHouse, which stamped the log row and the payload row separately, so they never match. Once the control plane is upgraded, bodies those gateways captured stop showing in the log drawer, and they stay hidden until `payload_retention_hours` removes them (7 days by default). Nothing is exposed and nothing is lost from the log rows themselves; only the payload panel goes empty for those requests. Upgrading the gateways first keeps the window of new bodies written that way as short as the rollout.

## Model price rates

The migration that adds `0084_model_price_rate_checks.sql` makes the database refuse a model price whose rate is negative or `NaN`. Before it, `PUT /api/v1/model-prices` accepted both: a negative rate turned a request into a credit against its budget, and `NaN` was stored but could not be read back, so the model was priced at zero. The API refuses both now (`400`, naming the field), and so does a `rolter.toml` with a negative rate. It also refuses a rate of a million or more, which the `numeric(12, 6)` columns cannot hold and which used to be a `500`.

The migration cannot fail on a database that already holds such a row. It adds each check as `NOT VALID`, which binds every insert and update from then on and leaves the rows already there alone, then validates the ones it can. A database with no bad row ends with all four checks validated and needs nothing from you. A check whose column does hold a bad row stays `NOT VALID`, and the migration prints a notice naming it.

Until you correct the row, Rolter leaves it out rather than guess a price for it. The model is **unpriced**, not free: its `cost_usd` is recorded as `0` with `unpriced = 1`, and `unpriced_policy` and any budget's override decide what that means, exactly as for a model you never priced. The omission is listed by `GET /api/v1/config/problems`, and logged by the control plane as it builds the snapshot.

To see whether any check is still unvalidated, and which rows break it:

```sql
select conname, convalidated
from pg_constraint
where conrelid = 'model_prices'::regclass and conname like '%\_rate';

select model, input_per_mtok, output_per_mtok, cached_input_per_mtok, cache_write_per_mtok
from model_prices
where input_per_mtok < 0 or input_per_mtok = 'NaN'
   or output_per_mtok < 0 or output_per_mtok = 'NaN'
   or cached_input_per_mtok < 0 or cached_input_per_mtok = 'NaN'
   or cache_write_per_mtok < 0 or cache_write_per_mtok = 'NaN';
```

Correct each row on the **Pricing** screen (saving a model's price replaces its input, output and cached-input rates), or delete the price to leave the model unpriced on purpose. Then validate whatever the first query still shows as `convalidated = false`:

```sql
alter table model_prices validate constraint model_prices_input_per_mtok_rate;
```

The four names are `model_prices_input_per_mtok_rate`, `model_prices_output_per_mtok_rate`, `model_prices_cached_input_per_mtok_rate` and `model_prices_cache_write_per_mtok_rate`. Validating is optional: an unvalidated check protects new writes just the same. It only changes what `convalidated` reports.

## Route advanced settings

A route's `advanced` block used to store, validate and display several settings that the gateway never read, and the dashboard and these docs described them as working (#2924). This release applies the ones it can and removes the rest. Both halves change something for a route that already carries them, so look before you roll out.

**Settings that start to apply.** A route that carries `limits.timeout_secs`, `limits.retries` or `headers` now uses them. Until now they changed nothing. A saved timeout that is shorter than a model's time to first byte now ends those requests early, a retry budget replaces `[retry].max_retries` for that route (a value above 10 is applied as 10), and every header is sent to the upstream. To list the routes that set any of them:

```sql
select model,
       advanced -> 'limits' ->> 'timeout_secs' as timeout_secs,
       advanced -> 'limits' ->> 'retries' as retries,
       advanced -> 'headers' as headers
from routes
where advanced -> 'limits' ? 'timeout_secs'
   or advanced -> 'limits' ? 'retries'
   or coalesce(advanced -> 'headers', '{}'::jsonb) <> '{}'::jsonb;
```

A header that names a credential (`authorization`, `x-api-key`, `api-key`, `x-goog-api-key`) or the request's framing (`host`, `content-length`) is not sent. The snapshot leaves it out and `GET /api/v1/config/problems` names it; the route keeps serving. The credential belongs on the provider.

**Settings that are gone.** `base_url`, `insecure_tls`, `pricing` and the `context_window` limit never had an effect, and requests were served exactly as if they were not set. A route's `base_url` never redirected a request, and `insecure_tls = true` never turned off certificate verification. If you believed either, nothing changes on the wire, but the setting is gone: point a route at a second endpoint with a second provider, and trust a private CA with `[tls].ca_bundles` or the provider's `ca_bundles`. To list the routes that still carry one:

```sql
select model, advanced ->> 'base_url' as base_url, advanced ->> 'insecure_tls' as insecure_tls
from routes
where advanced ?| array['base_url', 'insecure_tls', 'pricing']
   or advanced -> 'limits' ? 'context_window';
```

Nothing is migrated, and a stored value that is no longer read keeps loading without complaint. The routes API drops these keys from a write instead of refusing it, so an older client or script still saves. Saving a model's sheet in the dashboard sheds them from that route. To clear them all at once:

```sql
update routes
set advanced = (advanced - 'base_url' - 'insecure_tls' - 'pricing') #- '{limits,context_window}'
where advanced ?| array['base_url', 'insecure_tls', 'pricing']
   or advanced -> 'limits' ? 'context_window';
```

A `rolter.toml` that sets any of them under `[routes.advanced]` still loads. `rolter check` and the gateway's startup log report each one as an unknown key.

**Route limits.** `limits.rpm`, `limits.tpm` and `limits.concurrency` are enforced again, this time with a reader ([#2936](https://github.com/rolter-ai/rolter/issues/2936)): a request over a route's cap is refused with `429` and `Retry-After`. Migration `0090_clear_unread_route_limits.sql` removes the three keys from every stored route first, because a value typed into the old form while nothing read it would otherwise start refusing traffic on the day you deploy. A route that should be limited has to be given its cap again after the upgrade, from the model sheet or the routes API. A `rolter.toml` that sets them is read as written, so check a file you carried over from an older release. Enforcement needs a gateway that has the reader, so finish the upgrade, gateways included, before you set a cap ([rule 3](#upgrade-order)): until a gateway restarts on the new version, the cap is stored and not applied. `rpm` and `tpm` need `redis_url` to be enforced; `concurrency` does not. See [Route limits](/concepts/providers-and-routes#route-limits).

## Security and egress settings

The migration `0086_security_egress_settings_shape.sql` makes the database refuse three settings whose stored shape the API already enforced: `required_headers` on the security settings, `egress_proxies` on a provider, and `injected_headers` on the client settings. Before it, a value written around the API with SQL, a seed or an import could be read back as something weaker than what you meant (#2943), and now each is handled on purpose:

| Setting                                | If a stored value cannot be read                                                                                                                                                                                                                                                                                                                                                                 |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `required_headers` (security settings) | The control plane publishes **no snapshot**. Gateways keep the config they last received, rule included, and a gateway that restarts meanwhile serves nothing until the rule is repaired. The effective-config views in the dashboard return an error, while the **Security** screen and `/api/v1/config/problems` keep working; **Effective config** lists those problem lines under its error. |
| `egress_proxies` (a provider)          | That provider is left out of the snapshot, with every route target and group member that named it, so its traffic never goes out around the proxy pool. A route with no other target is not served.                                                                                                                                                                                              |
| `injected_headers` (client settings)   | The entries that cannot be read are not injected, and the rest are. This setting gates nothing.                                                                                                                                                                                                                                                                                                  |

Each case is listed by `GET /api/v1/config/problems` and logged by the control plane (a refused snapshot also names the setting in each gateway's `config snapshot poll failed` log line), naming the setting, the provider or the header, and never a value. A header's value is not repeated because it may be a shared secret. A required header counts as unreadable when the setting is not an object, when a value is not a string or is blank, when a name is blank, or when one header is listed twice with different values.

The migration cannot fail on a database that already holds such a row. It adds each check as `NOT VALID`, which binds every insert and update from then on and leaves the rows already there alone, then validates the ones it can. A database with no bad row ends with all three checks validated and needs nothing from you. A check whose table does hold a bad row stays `NOT VALID`, and the migration prints a notice naming it. To see whether any is still unvalidated:

```sql
select conname, convalidated
from pg_constraint
where conname in ('security_settings_required_headers_shape',
                  'providers_egress_proxies_strings',
                  'client_settings_injected_headers_shape');
```

Find the rows that break them, correct each one, then validate whatever the first query still shows as `convalidated = false`. Save the security settings again from the **Security** screen, or `PUT /api/v1/security-settings`, to repair `required_headers`; set a provider's proxy pool on the **Providers** screen; save the client settings again for `injected_headers`.

```sql
select name, egress_proxies
from providers
where case when jsonb_typeof(egress_proxies) = 'array'
           then jsonb_path_exists(egress_proxies, 'strict $[*] ? (@.type() != "string")')
           else true end;

select required_headers from security_settings
where case when jsonb_typeof(required_headers) = 'object'
           then jsonb_path_exists(required_headers, 'strict $.keyvalue() ? (@.key like_regex "^\\s*$" || @.value.type() != "string" || @.value like_regex "^\\s*$")')
           else true end;

alter table security_settings validate constraint security_settings_required_headers_shape;
```

Validating is optional: an unvalidated check protects new writes just the same. It only changes what `convalidated` reports.

## Upstream TLS trust store

Rolter now verifies upstream HTTPS against the host's system trust store (`/etc/ssl/certs`, or the bundle `SSL_CERT_FILE` names) instead of a copy of the public roots built into the binary. The published images ship `ca-certificates`, so they need nothing from you. A binary you run on a host or in an image of your own with no CA store fails every upstream handshake until you install `ca-certificates` or point `SSL_CERT_FILE` at a PEM bundle. `ROLTER_CA_BUNDLE` and `[tls].ca_bundles` still add private CAs on top of that store.

## MCP tool manifests and tool groups

Two MCP rules that were stored but not enforced are enforced from this release:

- **Tool groups.** A server that an enabled tool group names serves only the tools those groups list. A call to any other tool is refused with `403` `mcp_tool_denied`. Check your enabled groups before you upgrade.
- **Declared manifests.** A server that declares tools serves only those once an organization's saved MCP settings refuse unlisted tools. To keep upgrades from refusing calls, an organization that never saved its MCP settings allows unlisted tools, and the upgrade sets **Allow unlisted tools** on for every organization that had saved them. Turn it off on **MCP → MCP Settings** to enforce a server's manifest.

Organizations created from this release on start the other way round: until they save their MCP settings, a server's declared manifest is enforced and a call to a tool it leaves out is refused with `403` `mcp_tool_denied`. They opt out by saving **MCP → MCP Settings** with **Allow unlisted tools** on. Organizations that already exist are not touched, whether or not they ever saved settings, and creating an organization still stores no settings row, so its timeouts and retries keep following the deployment defaults.

## Log erasure and the MCP tool-call log

A [log erasure](/observability/log-erasure) now also deletes (or, for `include_logs: false`, blanks the bodies of) the subject's rows in `mcp_tool_call_logs`. A ClickHouse that was never given `clickhouse/007_mcp_tool_call_logs.sql` has no such table, so the erasure fails at its MCP statement with the usual `failed` record after the request-log statements were already submitted. Apply the migration before the first erasure after upgrading; submitting the erasure again is safe.

## A checklist

1. Back up the control-plane database, and confirm you have `ROLTER_KEK` outside it.
2. Run `rolter check --config rolter.toml --migrations` with the **new** binary and read the plan.
3. Roll out. Config migrates in memory on start; database migrations apply on the control plane's start.
4. Confirm the gateway serves traffic and the dashboard lists your providers.