Alerting: Channels, Rules and History

Alerting is a control-plane feature. A rule watches one signal from the request log, compares it with a threshold every minute, and moves between ok and firing. A channel is the webhook a rule reports to: each change of state is POSTed to it as JSON. The history records each change and whether it was delivered.

In the dashboard it is the Alerting group, with one screen for each: Channels, Alert Rules and History. The API is /api/v1/alert-channels, /api/v1/alert-rules and /api/v1/alert-notifications.

Requirements#

• A control plane with a database. The alerting API and the evaluator are only mounted when the control plane runs with ROLTER_DATABASE_URL.
• ClickHouse on the control plane (CLICKHOUSE_URL), with the gateway writing request logs to it. Every signal is a query over the request_logs or provider_health_events table; without ClickHouse, evaluation fails with alert evaluation requires CLICKHOUSE_URL.
• ROLTER_KEK, only if a channel stores a secret. The same key is needed at delivery time to unseal it.
• A route from the control plane to each channel endpoint that the egress policy permits.
• An organization, team or project admin for the scope you want to watch, or a superadmin account, an operator (a deployment role) or the deployment admin token for the whole deployment (see Scopes and ownership and Permissions).

Rules read the request log, and the log can be sampled. With a sample_rate below 1, each kept row carries the number of real requests it stands for, and request_volume and spend_velocity add those up, so a threshold set against real traffic still means real traffic. error_rate is the weighted share of failing requests, a ratio of weighted counts. p95_latency_ms is an estimate from the sample. Rows written before the sample weight was recorded count as one request each.

How a rule is evaluated#

Every 60 seconds the control plane loads the enabled rules and evaluates each one. The first pass runs a minute after the control plane starts.

An evaluation is one ClickHouse query over the trailing window, window_secs seconds back from now. The result is the rule’s last_value. With the default comparison of above, a value greater than or equal to threshold makes the rule firing, otherwise ok; with below, a value less than or equal to it does. A window with nothing to measure is handled by the rule’s no_data policy. Windows longer than a minute overlap from one pass to the next, so a 5-minute rule looks at a sliding 5 minutes each time.

A rule without a scope reads the whole deployment. To watch a slice, give it a scope. A rule still cannot be narrowed to one provider, model or route.

For alerts that need that, use the gateway’s Prometheus metrics, which carry per-model and per-provider labels.

Signals#

Each signal’s value is in its own unit, and a rule’s threshold is in the same unit. The API stores and returns the value as the query computes it; the dashboard shows it under a translated name and in that unit.

Spend is in the deployment’s settlement currency, the base of the [currency] table (USD unless configured). The cost_usd column name is historic and does not mean dollars.

An empty window reads 0 for every signal, except that error_rate and p95_latency_ms have no value at all over zero requests (a rate or a percentile of nothing is not 0): the evaluation reports no data, stored as a last_value of null, and the rule’s no_data policy decides what that means. The other three signals have a true zero to compare.

No data and below rules#

Two rule fields cover what a high-only threshold cannot:

Rules saved before these fields existed read as above and ignore, which is how they always behaved. no_data other than ignore is accepted only on error_rate and p95_latency_ms, the signals that can lack data; the API rejects it elsewhere rather than store a policy that could never apply.

A rule that fires on no data records why: its history row’s detail ends in no data in window, and the delivery carries "no_data": true, a null value and a text saying so (see Delivery). A rule that was already firing when a window has no data is not reported again.

Traffic stopped. To be paged when the gateway, or everything in front of it, goes quiet, use request_volume with comparison: below. A count of zero requests is a real reading for that signal, so threshold: 0 fires when the window holds no requests at all:

{  "name": "traffic stopped",  "signal": "request_volume",  "comparison": "below",  "threshold": 0,  "window_secs": 300,  "channel_id": "3f6c2a4e-8d1b-4c1e-9a57-2b0d6e9f1c23",  "enabled": true}

For an error-rate rule that should not stay silent when there are no requests to count is signal: error_rate, no_data: fire. A rule on the request log also stops when the gateway stops logging, so a dead log pipeline pages too; it cannot tell the two apart.

States and transitions#

A new rule starts as unknown. Each evaluation compares the reading with what the rule last reported, which is the state of its newest history row, and records a row when the two differ:

“At or over” and “under” are for an above rule; a below rule reads them the other way round, and a window with no data is one more reading, resolved by no_data into firing, resolved or no change.

So a rule created or enabled while its condition already holds reports firing on its first evaluation, and a rule whose first reading is under the threshold records nothing until it first fires. A rule that stays firing for an hour produces one row and one delivery, not sixty, once that delivery succeeds.

Disabling a rule stops the scheduled evaluation and keeps its state as it was. Editing a rule keeps its state too, so a new threshold takes effect on the next pass and reports only if the result differs from what the rule last reported.

When an evaluation fails, the rule’s state becomes error and last_error says why:

Beside last_error the rule carries last_error_code, a stable code a client can translate, and last_error_params, the numbers the text interpolates. last_error stays as the English fallback. Both are null while the last evaluation succeeded. A failed Evaluate now returns the same code as error.code.

An error never carries the ClickHouse URL, which can hold a password; the full error is in the control-plane log under alert signal query failed. last_value and last_evaluated_at keep the last successful reading, and an error writes no history row and sends nothing. The next successful evaluation clears last_error and compares with the last reported state as usual, so a rule that was firing before the error is not reported again, and one whose condition cleared during it still reports resolved.

Every control-plane process runs its own evaluator, so with more than one replica each rule is evaluated by several of them. An evaluation holds the rule’s row lock only while it decides and writes the history, and the scheduled pass skips a rule another replica is holding, so one replica delivers and records each change of state. The delivery itself runs after that, in the background, so a receiver that is slow or not answering delays only its own rule’s history row, never the evaluation of the others. If another replica records a newer reading while one replica’s query is still running, the older reading is dropped rather than applied, so a slow query cannot undo a newer result. last_evaluated_at is the time the reading was taken.

Evaluate now#

Each rule card has an Evaluate now button, backed by POST /api/v1/alert-rules/{id}/evaluate. It runs one evaluation immediately, whether or not the rule is enabled, and can record and deliver a transition exactly as the scheduled pass would. It also retries a state that was never delivered straight away, without waiting for the retry delay. If a scheduled pass is evaluating the same rule at that moment, it waits for it to finish first.

The response has three fields: rule, the updated rule; notified, which is true only when a transition was delivered; and notification, the history row the evaluation wrote, or null when it wrote none: the reading was not a transition and nothing was retried. A transition the endpoint refused looks like this, with rule left out:

{  "notified": false,  "notification": {    "id": "0d9c7a52-5f1e-4a8b-9d0e-3c2b1a0f9e8d",    "rule_id": "5b1f0c2e-7a44-4f0e-8d1c-9e3a6b2d4c10",    "channel_id": "3f6c2a4e-8d1b-4c1e-9a57-2b0d6e9f1c23",    "state": "firing",    "delivery_status": "failed",    "detail": "HTTP 401",    "detail_code": "http_error",    "detail_params": { "status": 401 },    "value": 0.07,    "threshold": 0.05,    "sent_at": "2026-09-27T10:15:02.184Z"  }}

In the dashboard, the toast after Evaluate now says what became of a transition: delivered with the endpoint’s status, failed with the reason, or recorded without delivery, followed by the reading and the threshold it crossed in the signal’s unit. A failed delivery turns the toast red.

A failed evaluation sets the rule to error and comes back as an error toast carrying the same reason as last_error, which makes this the quickest way to check that ClickHouse is reachable from the control plane.

Channels#

A channel is a named webhook endpoint. webhook is the only kind.

The endpoint is stored as the HTTP client parses it, so HTTPS://Alerts.Example.com is saved as https://alerts.example.com/.

The Add channel sheet asks for a name, the endpoint URL and the optional secret, and creates the channel enabled. Each channel card shows its kind, a secret set badge when a secret is stored, an enable switch, an edit button and a delete button.

Delivery#

When a rule’s state changes and its channel is enabled, the control plane sends one POST to the endpoint with a JSON body:

{  "id": "0d9c7a52-5f1e-4a8b-9d0e-3c2b1a0f9e8d",  "state": "firing",  "rule": { "id": "5b1f0c2e-7a44-4f0e-8d1c-9e3a6b2d4c10", "name": "high error rate" },  "signal": "error_rate",  "value": 0.07,  "no_data": false,  "comparison": "above",  "threshold": 0.05,  "unit": "fraction",  "currency": null,  "window_secs": 300,  "evaluated_at": "2026-09-27T10:15:02.113Z",  "text": "high error rate is firing: error_rate is 7%, threshold 5%, over the last 5 minutes"}

value and threshold are in the signal’s API unit, which unit names, so an error_rate of 7% arrives as 0.07. text words them for a person: a percentage for error_rate, ms for p95_latency_ms, the settlement currency per hour for spend_velocity (EUR 12.5 per hour), and a count with its noun for request_volume and provider_health_flaps. The window is a duration (over the last 5 minutes), not a second count.

The request carries Content-Type: application/json, a User-Agent of rolter/<version>, and, when the channel stores a secret, Authorization: Bearer <secret>. The body has the same shape whichever service receives it, so a destination that wants its own format, such as a paging or chat service, needs a small relay in front of it that reads this body and posts its own.

A delivery succeeds on any 2xx response. It is bounded to 10 seconds including a 5-second connect, and redirects are not followed, so a 3xx counts as a failure. The response body is never read or stored. At most 16 deliveries run at once per control-plane process.

Retries#

A failed delivery is recorded as failed and the rule still moves to its new state. While that state holds, the scheduled pass sends it again: on the next pass, then 2, 4, 8, 16 and 32 minutes after the attempt before, then hourly, each attempt a new history row with a new id. Retries stop at the first delivered row, or when the state changes, and the new state is delivered as usual. A rule that resolves before its firing got through sends resolved.

A transition recorded as skipped because the channel was disabled or the rule had none is sent on the first pass after the rule has an enabled channel. A rule without one writes no more rows while its state holds.

A receiver can see the same firing or resolved more than once, from these retries, and should treat each request’s id as the delivery, not the alert. Check the History screen, or GET /api/v1/alert-notifications, for failed rows after changing an endpoint or its secret, and use Evaluate now to retry one without waiting.

Endpoints and the egress policy#

A channel endpoint is a URL the control plane will POST to, so it goes through the deployment’s egress policy (the [egress] table of the bootstrap config) when the channel is saved and again before every delivery. The default policy denies link-local addresses, which is where cloud instance metadata lives, and can be tightened to deny loopback or private ranges too. A channel whose endpoint the policy denies is rejected with 400 when saved, and a delivery to an endpoint the policy has come to deny since is recorded as failed without a request being sent. Such a channel can still be switched off, renamed or deleted, but not switched back on or pointed at another denied address.

Only an endpoint whose host is an IP address is checked, in whatever spelling the URL uses: http://2852039166/, http://169.254.169.254./ and http://[::ffff:169.254.169.254]/ are all checked as 169.254.169.254. An endpoint with a hostname is not checked against the policy at all, neither when it is saved nor before a delivery, so a hostname that resolves to a private or loopback address is reached even with block_private or block_loopback on. Checking the address a hostname resolves to when the request connects is tracked in #1949.

The secret is write-only#

A managed_secret is encrypted with the deployment KEK before it reaches Postgres. With ROLTER_KEK unset, storing one fails with storing channel credentials requires ROLTER_KEK. It is never returned: responses carry secret_configured, a boolean. The secret is sent as an HTTP header, so a value with a control character or a line break, such as a trailing newline read from a file, is rejected with 400.

An update that leaves managed_secret out keeps the stored secret, so flipping the enable switch cannot drop it, unless the update moves the endpoint to another origin: a different scheme, host or port. That clears the stored secret, because it was given for the receiver at the old endpoint. Send the new receiver’s managed_secret in the same request to keep the channel authenticated.

The secret is unsealed only to build the Authorization header of a delivery. If it cannot be unsealed, because ROLTER_KEK is unset or differs from the key it was sealed with, the delivery is recorded as failed and nothing is sent: the control plane never sends a request without a credential the channel was configured with.

The sealed column is covered by rolter kek verify and listed in Backup and restore.

Deleting a channel#

Rules pointing at a deleted channel stay in place with their channel_id cleared, so their next transitions are recorded as skipped with no channel configured. History rows that named the channel keep their other fields. A delivery already in flight when the channel is deleted still finishes and is recorded.

Rules#

comparison and no_data may be left out of a request, and read as their defaults. The server also returns state, last_value (null after a window with no data), last_evaluated_at and last_error.

The Add rule sheet lists each signal by its dashboard name, with a line under the picker saying what it counts and over what. The threshold field’s label names the unit it is typed in, such as Threshold (%) or Threshold (EUR per hour), and the field starts from that signal’s own default: 5 (%) for error_rate, 8000 ms for p95_latency_ms, 50 per hour for spend_velocity, 1000 requests for request_volume and 10 failed health events for provider_health_flaps. Choosing another signal resets the threshold to that signal’s default, since a number typed in one unit means nothing in another.

error_rate is typed as a percentage from 0 to 100 and sent as the fraction the API stores: typing 5 sends 0.05. The sheet will not save a percentage over 100, which the fraction can never reach, a negative threshold, or a window outside 60 to 86400 seconds. The window starts at 300 seconds and the first channel is selected; pick none (record only) for a rule without one. The rule is created enabled.

Each rule card shows its state, signal, threshold, window, last value, when it was last evaluated and its channel, with an enable switch, Evaluate now, an edit button and a delete button. The threshold and last value are in the signal’s unit (5%, 840 ms, €12.50/h, 340 requests in 5m) and the window is a duration (5m). When it was last evaluated reads as how long ago (3 min. ago), with the full date and time on hover. A rule in error shows its last_error under the figures, in the same amber as its state. A rule that could not be evaluated has no reading to compare, so it does not borrow the red of firing. The state, like the delivery statuses below, is shown in the dashboard’s language rather than as the stored identifier.

Some starting points, with the threshold as the API takes it and as the dashboard shows it:

Tune request_volume to your own traffic. With the default above it catches a surge; for a drop, use comparison: below (see No data and below rules).

Changing a rule or channel#

The edit button on a rule or channel card opens the sheet that adds one, filled in from the row. Change the threshold, window, signal, channel or endpoint and choose Save. Do not delete and recreate instead: deleting a rule also deletes its history, and deleting a channel clears it from every rule that delivers through it. An edited rule keeps its state and history, and the change applies from its next evaluation.

Saving sends the enable switch back as it was, so editing a rule or channel that is switched off leaves it off. The edit button follows the same permission as the switch, alert_rule or alert_channel update.

An existing rule opens on its own threshold, in the signal’s unit: a stored 0.05 for error_rate opens as 5. Only choosing another signal resets it to that signal’s default.

A channel’s secret field opens empty, because a stored secret is never read back. Leave it blank to keep the stored secret, or type one to replace it. A secret belongs to the endpoint’s origin, so pointing the channel at another scheme, host or port without typing a new secret drops the stored one. The sheet says so beside the field before you save.

The same change through the API is a PUT, which replaces the whole row, so send every field. A body without enabled turns the rule or channel off.

curl -X PUT https://control.example.com/api/v1/alert-rules/$RULE_ID \  -H "Authorization: Bearer $ROLTER_TOKEN" \  -H "Content-Type: application/json" \  -d '{        "name": "high error rate",        "signal": "error_rate",        "threshold": 0.02,        "window_secs": 300,        "channel_id": "3f6c2a4e-8d1b-4c1e-9a57-2b0d6e9f1c23",        "enabled": true      }'

A channel update takes name, endpoint, enabled and, only to replace it, managed_secret.

History#

Alerting → History lists the rows 100 at a time, newest first, with the state (firing or resolved), delivery status, time, rule, reading and detail. The state and delivery columns come first so they stay in view on a phone. Every change of state has a row, including those of rules with no channel.

A full page may be followed by older rows, and Load older transitions under the list reads the next page from the last row shown (the before cursor below). A page that comes back short is the end, and the button goes away. The toolbar narrows the list, and each picker is sent to the API, so it reaches its own newest rows rather than searching the page already loaded:

• Rule is sent as rule_id.
• State is sent as state.
• Delivery is sent as delivery_status.

Changing a picker starts the list again from the newest matching row.

The Reading column shows the value the rule read when the row was written and, under it, the threshold it was held against, both in the signal’s unit (11% above 5%, 840 ms above 2,000 ms). A transition written for a window with no data reads No data, and a row from before readings were recorded shows —.

The Detail column and the rule card’s last_error are translated from the detail_code and last_error_code below, with their detail_params / last_error_params filling in the numbers. A code this build does not know, a row written before codes existed, and a code whose numbers did not arrive show the control plane’s English detail / last_error as stored. A long detail wraps onto further lines inside its column rather than being cut, so a diagnosis such as channel secret could not be unsealed; check ROLTER_KEK reads in full. On a phone the detail is the last column, so scroll the list sideways to reach it.

A detail never contains the endpoint’s response body, which can echo the credential it rejected. sent_at is when the transition was recorded, just before the delivery attempt started. A retried state has one row per attempt.

The history is append-only: the API has no update or delete for it. Deleting a rule deletes that rule’s rows along with it.

Each row also carries detail_code, a stable code for the outcome, and detail_params, the numbers the text interpolates. detail stays as the English fallback. A row written before codes existed has null for both.

A row written for a window with no data adds "no_data": true to its detail_params.

value and threshold record the reading the row was written for and the rule’s threshold at that moment, in the signal’s API unit (see Delivery). value is null for a no-data transition and on older rows. The Evaluate now response carries the same fields on its notification.

The API takes these optional query parameters:

The filters combine. The list is newest first, ordered by sent_at then id, so rows written in the same instant keep a stable order. To read the next page, pass the last row’s sent_at and id as before; an empty page, or one shorter than limit, is the end. A value outside the lists above, or a malformed before, answers 400 with code invalid_field.

curl -s "https://control.example.com/api/v1/alert-notifications?delivery_status=failed&limit=50&before=2026-09-27T10:15:02.184Z,0d9c7a52-5f1e-4a8b-9d0e-3c2b1a0f9e8d" \  -H "Authorization: Bearer $ROLTER_TOKEN"

Scopes and ownership#

A rule can watch one scope instead of the deployment: an organization, a team, a project, a virtual key, a business unit or a customer. It then evaluates only the request-log rows of that scope, so a team lead’s error-rate rule sees their team’s traffic and nothing else. The scope is set when the rule is created (scope_type and scope_id in the body, given together) and cannot be changed afterwards; create a new rule to watch another scope. provider_health_flaps is recorded per organization, so it can only watch an organization or the whole deployment. An organization’s rule counts the health events of the providers that organization owns; providers defined in the gateway’s own config file belong to no organization and are counted only by a deployment-wide rule. A rule whose team, project or key has since been deleted reports a scope error (scope_gone) and does not fire; delete it.

Creating, editing, deleting and evaluating a rule takes admin at its scope: an organization admin for an organization rule, a team admin for a team’s, and so on. A rule for a business unit or a customer is the organization admin’s. A rule with no scope, which reads every tenant, stays superadmin-only.

A channel belongs to one organization or to the deployment. An organization admin owns the channels of their organization (org_id in the create body); a deployment channel is created by a superadmin and can be named by any rule. A rule can notify only a channel of its own organization or a deployment channel, and a deployment-wide rule only a deployment channel. The control plane refuses anything else with a 400 on channel_id, answering an unknown channel the same way so the id cannot be probed, and it checks again at delivery. Names are unique per owner, so two organizations can each have a rule called prod errors.

The lists are narrowed to what the caller may use. Rules and notification history show only the scopes the caller administers. Channels show the caller’s organizations’ channels and the deployment’s, but a deployment channel’s address is withheld (endpoint is empty and manageable is false), as it may carry a token. A caller who administers nothing gets 403.

Permissions and audit#

Every operation is decided at the owner of the row, as described above, and the dashboard shows the Alerting group to anyone who administers a scope. Deployment rows and channels are for a superadmin or an operator (a deployment role), like the other deployment-wide settings in RBAC.

Evaluating a rule by hand counts as creating history, because it can write a row.

Every create, update and delete of a channel or rule writes an audit entry (alert.channel.create, alert.rule.delete and so on). Create and update entries carry the name, the enabled flag and, for a rule, the signal and scope. A channel update entry also records endpoint_changed, secret_replaced and secret_cleared, as booleans. Neither a channel secret nor its endpoint ever appears in one, since an endpoint’s query string can carry a token.

All of these operations are in the control plane’s OpenAPI document under the alerting tag.