> ## Documentation index
> Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt
> Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=<keywords or question>. Short keywords rank best.
> Use these to discover all available pages before exploring further.

# Alerting: Channels, Rules and History

> Threshold rules over error rate, latency, spend, traffic and provider health, evaluated every minute from ClickHouse, with an optional webhook channel per rule and a history of every state change.

Alerting is a control-plane feature. A **rule** watches one signal from the
request log, compares it with a threshold every minute, and moves between `ok`
and `firing`. A **channel** is the webhook a rule reports to: each change of
state is POSTed to it as JSON. The **history** records each change and whether
it was delivered.

In the dashboard it is the **Alerting** group, with one screen for each:
**Channels**, **Alert Rules** and **History**. The API is `/api/v1/alert-channels`,
`/api/v1/alert-rules` and `/api/v1/alert-notifications`.

## Requirements

- A control plane with a database. The alerting API and the evaluator are only
  mounted when the control plane runs with `ROLTER_DATABASE_URL`.
- ClickHouse on the control plane (`CLICKHOUSE_URL`), with the gateway
  [writing request logs](/observability/logs-and-cost#enabling-logging) to it.
  Every signal is a query over the `request_logs` or `provider_health_events`
  table; without ClickHouse, evaluation fails with
  `alert evaluation requires CLICKHOUSE_URL`.
- `ROLTER_KEK`, only if a channel stores a secret. The same key is needed at
  delivery time to unseal it.
- A route from the control plane to each channel endpoint that the
  [egress policy](#endpoints-and-the-egress-policy) permits.
- An organization, team or project admin for the scope you want to watch, or a
  superadmin account, an operator (a [deployment role](/security/rbac#deployment-roles)) or the deployment admin token for the whole deployment
  (see [Scopes and ownership](#scopes-and-ownership) and
  [Permissions](#permissions-and-audit)).

Rules read the request log, and the log can be sampled. With a
[`sample_rate`](/observability/log-settings#sampling) below `1`, each kept row
carries the number of real requests it stands for, and `request_volume` and
`spend_velocity` add those up, so a threshold set against real traffic still
means real traffic. `error_rate` is the weighted share of failing requests, a
ratio of weighted counts. `p95_latency_ms` is an estimate from the sample.
Rows written before the sample weight was recorded count as one request each.

## How a rule is evaluated

Every 60 seconds the control plane loads the enabled rules and evaluates each
one. The first pass runs a minute after the control plane starts.

An evaluation is one ClickHouse query over the trailing window, `window_secs`
seconds back from now. The result is the rule's `last_value`. With the default
`comparison` of `above`, a value greater than or equal to `threshold` makes the
rule `firing`, otherwise `ok`; with `below`, a value less than or equal to it
does. A window with nothing to measure is handled by the rule's
[`no_data` policy](#no-data-and-below-rules). Windows
longer than a minute overlap from one pass to the next, so a 5-minute rule
looks at a sliding 5 minutes each time.

A rule without a scope reads the whole deployment. To watch a slice, give it a
[scope](#scopes-and-ownership). A rule still cannot be narrowed to one
provider, model or route.

For alerts that need that, use the gateway's Prometheus
[metrics](/observability/metrics), which carry per-model and per-provider
labels.

### Signals

Each signal's value is in its own unit, and a rule's `threshold` is in the same
unit. The API stores and returns the value as the query computes it; the
dashboard shows it under a translated name and in that unit.

| Signal                  | Value                                                                                                                                                                                                                                           | Unit (API)                      | Dashboard                                                     | Table                    |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------- | ------------------------ |
| `error_rate`            | Share of logged requests in the window with an HTTP status of 500 or above. 4xx responses, `429` included, do not count.                                                                                                                        | a fraction from `0` to `1`      | **Error rate**, a percentage: `0.05` shows as `5%`            | `request_logs`           |
| `p95_latency_ms`        | 95th-percentile request latency, from ClickHouse's approximate `quantile`.                                                                                                                                                                      | milliseconds                    | **p95 latency**: `840 ms`                                     | `request_logs`           |
| `spend_velocity`        | Spend in the window scaled to an hourly rate, `sum(cost_usd) × 3600 / window_secs`, so a 5-minute and a 1-hour window give comparable figures.                                                                                                  | settlement currency per hour    | **Spend rate**: `€12.50/h`                                    | `request_logs`           |
| `request_volume`        | Number of logged requests in the window. A count, not a rate: the same traffic reads twice as high over a window twice as long.                                                                                                                 | requests per window             | **Request volume**: `340 requests in 5m`                      | `request_logs`           |
| `provider_health_flaps` | Number of provider health observations in the window whose outcome was not `ok` (an error or a timeout), across every target and every source, active probes and passive request results alike. Despite the name it counts failures, not flaps. | failed health events per window | **Provider health failures**: `10 failed health events in 5m` | `provider_health_events` |

Spend is in the deployment's settlement currency, the `base` of the
[`[currency]` table](/security/budgets-and-rate-limits#pricing-in-another-currency)
(USD unless configured). The `cost_usd` column name is historic and does not
mean dollars.

An empty window reads `0` for every signal, except that `error_rate` and
`p95_latency_ms` have no value at all over zero requests (a rate or a
percentile of nothing is not `0`): the evaluation reports **no data**, stored as
a `last_value` of `null`, and the rule's [`no_data`](#no-data-and-below-rules)
policy decides what that means. The other three signals have a true zero to
compare.

### No data and below rules

Two rule fields cover what a high-only threshold cannot:

| Field        | Values                 | Default  | Meaning                                                                                                                                             |
| ------------ | ---------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| `comparison` | `above`, `below`       | `above`  | `above` fires at `value >= threshold`, `below` at `value <= threshold`. Both are inclusive, so a value exactly at the threshold fires either way.   |
| `no_data`    | `ignore`, `fire`, `ok` | `ignore` | What a window with no data does. `ignore` keeps the rule's state. `fire` treats it as a breach. `ok` treats it as healthy, resolving a firing rule. |

Rules saved before these fields existed read as `above` and `ignore`, which is
how they always behaved. `no_data` other than `ignore` is accepted only on
`error_rate` and `p95_latency_ms`, the signals that can lack data; the API
rejects it elsewhere rather than store a policy that could never apply.

A rule that fires on no data records why: its history row's `detail` ends in
`no data in window`, and the delivery carries `"no_data": true`, a `null`
`value` and a `text` saying so (see [Delivery](#delivery)). A rule that
was already `firing` when a window has no data is not reported again.

**Traffic stopped.** To be paged when the gateway, or everything in front of
it, goes quiet, use `request_volume` with `comparison: below`. A count of zero
requests is a real reading for that signal, so `threshold: 0` fires when the
window holds no requests at all:

```json
{
  "name": "traffic stopped",
  "signal": "request_volume",
  "comparison": "below",
  "threshold": 0,
  "window_secs": 300,
  "channel_id": "3f6c2a4e-8d1b-4c1e-9a57-2b0d6e9f1c23",
  "enabled": true
}
```

For an error-rate rule that should not stay silent when there are no
requests to count is `signal: error_rate`, `no_data: fire`. A rule on the
request log also stops when the gateway stops _logging_, so a dead log pipeline
pages too; it cannot tell the two apart.

### States and transitions

A new rule starts as `unknown`. Each evaluation compares the reading with what
the rule last reported, which is the state of its newest history row, and
records a row when the two differ:

| Reading    | Last reported              | History row and delivery                      |
| ---------- | -------------------------- | --------------------------------------------- |
| at or over | nothing yet, or `resolved` | `firing`                                      |
| under      | `firing`                   | `resolved`                                    |
| at or over | `firing`                   | nothing, or a [retry](#retries) of `firing`   |
| under      | `resolved`                 | nothing, or a [retry](#retries) of `resolved` |
| under      | nothing yet                | nothing                                       |

"At or over" and "under" are for an `above` rule; a `below` rule reads them
the other way round, and a window with no data is one more reading, resolved
by `no_data` into `firing`, `resolved` or no change.

So a rule created or enabled while its condition already holds reports
`firing` on its first evaluation, and a rule whose first reading is under the
threshold records nothing until it first fires. A rule that stays `firing` for
an hour produces one row and one delivery, not sixty, once that delivery
succeeds.

Disabling a rule stops the scheduled evaluation and keeps its state as it was.
Editing a rule keeps its state too, so a new threshold takes effect on the next
pass and reports only if the result differs from what the rule last reported.

When an evaluation fails, the rule's state becomes `error` and `last_error`
says why:

| `last_error`                                      | Cause                                                                                            |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `alert evaluation requires CLICKHOUSE_URL`        | The control plane has no ClickHouse configured.                                                  |
| `could not connect to ClickHouse`                 | ClickHouse did not accept the connection.                                                        |
| `analytics query timed out after 30s`             | The query ran past the 30-second bound.                                                          |
| `clickhouse query failed (<status>): <exception>` | ClickHouse answered with an error, for example a missing `request_logs` table. Cut at 512 bytes. |
| `ClickHouse returned a response that is not JSON` | Something other than ClickHouse answered at `CLICKHOUSE_URL`.                                    |

Beside `last_error` the rule carries `last_error_code`, a stable code a client
can translate, and `last_error_params`, the numbers the text interpolates.
`last_error` stays as the English fallback. Both are `null` while the last
evaluation succeeded. A failed **Evaluate now** returns the same code as
`error.code`.

| `last_error_code`           | `last_error_params` | Cause                                                    |
| --------------------------- | ------------------- | -------------------------------------------------------- |
| `clickhouse_not_configured` |                     | The control plane has no ClickHouse configured.          |
| `clickhouse_unreachable`    |                     | ClickHouse did not accept the connection.                |
| `clickhouse_timeout`        |                     | The request to ClickHouse timed out.                     |
| `query_timeout`             | `seconds`           | The query ran past the bound.                            |
| `clickhouse_error`          |                     | ClickHouse answered with an error.                       |
| `clickhouse_bad_response`   |                     | Something other than ClickHouse answered.                |
| `query_failed`              |                     | The query failed for another transport reason.           |
| `unknown_signal`            | `signal`            | The stored rule names a signal this build does not know. |

An error never carries the ClickHouse URL, which can hold a password; the full
error is in the control-plane log under `alert signal query failed`.
`last_value` and `last_evaluated_at` keep the last successful reading, and an
error writes no history row and sends nothing. The next successful evaluation
clears `last_error` and compares with the last reported state as usual, so a
rule that was `firing` before the error is not reported again, and one whose
condition cleared during it still reports `resolved`.

Every control-plane process runs its own evaluator, so with more than one
replica each rule is evaluated by several of them. An evaluation holds the
rule's row lock only while it decides and writes the history, and the
scheduled pass skips a rule another replica is holding, so one replica delivers
and records each change of state. The delivery itself runs after that, in the
background, so a receiver that is slow or not answering delays only its own
rule's history row, never the evaluation of the others. If another replica records a newer reading
while one replica's query is still running, the older reading is dropped rather
than applied, so a slow query cannot undo a newer result. `last_evaluated_at`
is the time the reading was taken.

### Evaluate now

Each rule card has an **Evaluate now** button, backed by
`POST /api/v1/alert-rules/{id}/evaluate`. It runs one evaluation immediately,
whether or not the rule is enabled, and can record and deliver a transition
exactly as the scheduled pass would. It also retries a state that was never
delivered straight away, without waiting for the [retry](#retries) delay. If a
scheduled pass is evaluating the same rule at that moment, it waits for it to
finish first.

The response has three fields: `rule`, the updated rule; `notified`, which is
`true` only when a transition was delivered; and `notification`, the history
row the evaluation wrote, or `null` when it wrote none: the reading was not a
transition and nothing was retried. A transition the endpoint refused looks
like this, with `rule` left out:

```json
{
  "notified": false,
  "notification": {
    "id": "0d9c7a52-5f1e-4a8b-9d0e-3c2b1a0f9e8d",
    "rule_id": "5b1f0c2e-7a44-4f0e-8d1c-9e3a6b2d4c10",
    "channel_id": "3f6c2a4e-8d1b-4c1e-9a57-2b0d6e9f1c23",
    "state": "firing",
    "delivery_status": "failed",
    "detail": "HTTP 401",
    "detail_code": "http_error",
    "detail_params": { "status": 401 },
    "value": 0.07,
    "threshold": 0.05,
    "sent_at": "2026-09-27T10:15:02.184Z"
  }
}
```

In the dashboard, the toast after **Evaluate now** says what became of a
transition: delivered with the endpoint's status, failed with the reason, or
recorded without delivery, followed by the reading and the threshold it crossed in
the signal's unit. A failed delivery turns the toast red.

A failed evaluation sets the rule to `error` and comes back as an error toast
carrying the same reason as `last_error`, which makes this the quickest way to
check that ClickHouse is reachable from the control plane.

## Channels

A channel is a named webhook endpoint. `webhook` is the only kind.

| Field            | Meaning                                                                                                                  |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `name`           | Unique, 1 to 128 characters after trimming surrounding whitespace.                                                       |
| `kind`           | Always `webhook`, set by the server.                                                                                     |
| `endpoint`       | An `http` or `https` URL with a host. A URL carrying `user:password@` is rejected; put a credential in `managed_secret`. |
| `enabled`        | A disabled channel receives nothing; its rules' transitions are recorded as `skipped`. Defaults to `false` in the API.   |
| `managed_secret` | Optional bearer secret. Write-only, sealed before storage.                                                               |

The endpoint is stored as the HTTP client parses it, so
`HTTPS://Alerts.Example.com` is saved as `https://alerts.example.com/`.

The **Add channel** sheet asks for a name, the endpoint URL and the optional
secret, and creates the channel enabled. Each channel card shows its kind, a
**secret set** badge when a secret is stored, an enable switch, an edit button
and a delete button.

### Delivery

When a rule's state changes and its channel is enabled, the control plane sends
one `POST` to the endpoint with a JSON body:

```json
{
  "id": "0d9c7a52-5f1e-4a8b-9d0e-3c2b1a0f9e8d",
  "state": "firing",
  "rule": { "id": "5b1f0c2e-7a44-4f0e-8d1c-9e3a6b2d4c10", "name": "high error rate" },
  "signal": "error_rate",
  "value": 0.07,
  "no_data": false,
  "comparison": "above",
  "threshold": 0.05,
  "unit": "fraction",
  "currency": null,
  "window_secs": 300,
  "evaluated_at": "2026-09-27T10:15:02.113Z",
  "text": "high error rate is firing: error_rate is 7%, threshold 5%, over the last 5 minutes"
}
```

| Field          | Meaning                                                                                                        |
| -------------- | -------------------------------------------------------------------------------------------------------------- |
| `id`           | The history row this delivery is recorded as, so a request can be matched to the History screen.               |
| `state`        | `firing` or `resolved`.                                                                                        |
| `rule`         | The rule's `id` and `name`.                                                                                    |
| `signal`       | The [signal](#signals) the rule reads.                                                                         |
| `value`        | The reading that caused the change, in the signal's unit. `null` when `no_data` is `true`.                     |
| `no_data`      | `true` when the change came from a window with no data, under the rule's `no_data` policy.                     |
| `comparison`   | The rule's `comparison`, `above` or `below`.                                                                   |
| `threshold`    | The rule's threshold. `firing` means `value >= threshold`, or `<=` for a `below` rule.                         |
| `unit`         | The unit of `value` and `threshold`: `fraction` (0-1), `ms`, `currency_per_hour` or `count` (over the window). |
| `currency`     | The settlement currency code when `unit` is `currency_per_hour`, otherwise `null`.                             |
| `window_secs`  | The trailing window the value was read over.                                                                   |
| `evaluated_at` | When the evaluation that took the reading ran, RFC 3339 in UTC.                                                |
| `text`         | The same facts as one line, for a relay or a chat tool to display as-is.                                       |

`value` and `threshold` are in the signal's API unit, which `unit` names, so an
`error_rate` of 7% arrives as `0.07`. `text` words them for a person: a
percentage for `error_rate`, `ms` for `p95_latency_ms`, the settlement currency
per hour for `spend_velocity` (`EUR 12.5 per hour`), and a count with its noun
for `request_volume` and `provider_health_flaps`. The window is a duration
(`over the last 5 minutes`), not a second count.

The request carries `Content-Type: application/json`, a `User-Agent` of
`rolter/<version>`, and, when the channel stores a secret,
`Authorization: Bearer <secret>`. The body has the same shape whichever
service receives it, so a destination that wants its own format, such as a
paging or chat service, needs a small relay in front of it that reads this
body and posts its own.

A delivery succeeds on any `2xx` response. It is bounded to 10 seconds
including a 5-second connect, and redirects are not followed, so a `3xx`
counts as a failure. The response body is never read or stored. At most 16
deliveries run at once per control-plane process.

### Retries

A failed delivery is recorded as `failed` and the rule still moves to its new
state. While that state holds, the scheduled pass sends it again: on the next
pass, then 2, 4, 8, 16 and 32 minutes after the attempt before, then hourly,
each attempt a new history row with a new `id`. Retries stop at the first `delivered` row, or when
the state changes, and the new state is delivered as usual. A rule that
resolves before its `firing` got through sends `resolved`.

A transition recorded as `skipped` because the channel was disabled or the rule
had none is sent on the first pass after the rule has an enabled channel. A
rule without one writes no more rows while its state holds.

A receiver can see the same `firing` or `resolved` more than once, from these
retries, and should treat each request's `id` as the delivery, not the alert.
Check the **History** screen, or `GET /api/v1/alert-notifications`, for
`failed` rows after changing an endpoint or its secret, and use
**Evaluate now** to retry one without waiting.

### Endpoints and the egress policy

A channel endpoint is a URL the control plane will POST to, so it goes through
the deployment's egress policy (the `[egress]` table of the bootstrap config)
when the channel is saved and again before every delivery. The default policy
denies link-local addresses, which is where cloud instance metadata lives, and
can be tightened to deny loopback or private ranges too. A channel whose
endpoint the policy denies is rejected with `400` when saved, and a delivery to
an endpoint the policy has come to deny since is recorded as `failed` without a
request being sent. Such a channel can still be switched off, renamed or
deleted, but not switched back on or pointed at another denied address.

Only an endpoint whose host is an IP address is checked, in whatever spelling
the URL uses: `http://2852039166/`, `http://169.254.169.254./` and
`http://[::ffff:169.254.169.254]/` are all checked as `169.254.169.254`. An
endpoint with a hostname is not checked against the policy at all, neither
when it is saved nor before a delivery, so a hostname that resolves to a
private or loopback address is reached even with `block_private` or
`block_loopback` on. Checking the address a hostname resolves to when the
request connects is tracked in
[#1949](https://github.com/rolter-ai/rolter/issues/1949).

### The secret is write-only

A `managed_secret` is encrypted with the deployment KEK before it reaches
Postgres. With `ROLTER_KEK` unset, storing one fails with
`storing channel credentials requires ROLTER_KEK`. It is never returned: responses carry
`secret_configured`, a boolean. The secret is sent as an HTTP header, so a
value with a control character or a line break, such as a trailing newline
read from a file, is rejected with `400`.

An update that leaves `managed_secret` out keeps the stored secret, so flipping
the enable switch cannot drop it, unless the update moves the endpoint to
another origin: a different scheme, host or port. That clears the stored
secret, because it was given for the receiver at the old endpoint. Send the new
receiver's `managed_secret` in the same request to keep the channel
authenticated.

The secret is unsealed only to build the `Authorization` header of a delivery.
If it cannot be unsealed, because `ROLTER_KEK` is unset or differs from the key
it was sealed with, the delivery is recorded as `failed` and nothing is sent:
the control plane never sends a request without a credential the channel was
configured with.

The sealed column is covered by `rolter kek verify` and listed in
[Backup and restore](/deployment/backup-and-restore).

### Deleting a channel

Rules pointing at a deleted channel stay in place with their `channel_id`
cleared, so their next transitions are recorded as `skipped` with
`no channel configured`. History rows that named the channel keep their other
fields. A delivery already in flight when the channel is deleted still
finishes and is recorded.

## Rules

| Field         | Meaning                                                                                          |
| ------------- | ------------------------------------------------------------------------------------------------ |
| `name`        | Unique, 1 to 128 characters after trimming surrounding whitespace.                               |
| `signal`      | One of the [signals](#signals) above.                                                            |
| `threshold`   | A finite number, `0` or above, in the signal's own unit. See `comparison`.                       |
| `comparison`  | `above` (default) fires at `value >= threshold`, `below` at `value <= threshold`.                |
| `no_data`     | `ignore` (default), `fire` or `ok`: what a [window with no data](#no-data-and-below-rules) does. |
| `window_secs` | Trailing window in seconds, from `60` to `86400` (one minute to one day).                        |
| `channel_id`  | The channel to deliver to, or `null` to record transitions without delivering them.              |
| `enabled`     | Only enabled rules are evaluated on schedule. Defaults to `false` in the API.                    |

`comparison` and `no_data` may be left out of a request, and read as their
defaults. The server also returns `state`, `last_value` (`null` after a window
with no data), `last_evaluated_at` and
`last_error`.

The **Add rule** sheet lists each signal by its dashboard name, with a line
under the picker saying what it counts and over what. The threshold field's
label names the unit it is typed in, such as **Threshold (%)** or
**Threshold (EUR per hour)**, and the field starts from that signal's own
default: `5` (%) for `error_rate`, `8000` ms for `p95_latency_ms`, `50` per
hour for `spend_velocity`, `1000` requests for `request_volume` and `10` failed
health events for `provider_health_flaps`. Choosing another signal resets the
threshold to that signal's default, since a number typed in one unit means
nothing in another.

`error_rate` is typed as a percentage from 0 to 100 and sent as the fraction
the API stores: typing `5` sends `0.05`. The sheet will not save a percentage
over 100, which the fraction can never reach, a negative threshold, or a window
outside 60 to 86400 seconds. The window starts at 300 seconds and the first
channel is selected; pick **none (record only)** for a rule without one. The
rule is created enabled.

Each rule card shows its state, signal, threshold, window, last value, when it
was last evaluated and its channel, with an enable switch, **Evaluate now**, an
edit button and a delete button. The threshold and last value are in the signal's unit
(`5%`, `840 ms`, `€12.50/h`, `340 requests in 5m`) and the window is a
duration (`5m`). When it was last evaluated reads as how long ago (`3 min. ago`),
with the full date and time on hover. A rule in `error` shows its `last_error`
under the figures, in the same amber as its state. A rule that could not be
evaluated has no reading to compare, so it does not borrow the red of `firing`.
The state, like the delivery statuses below, is shown in the dashboard's
language rather than as the stored identifier.

Some starting points, with the threshold as the API takes it and as the
dashboard shows it:

| Signal                  | Threshold (API) | Dashboard                       | Window | Fires when                                                        |
| ----------------------- | --------------- | ------------------------------- | ------ | ----------------------------------------------------------------- |
| `error_rate`            | `0.05`          | `5%`                            | `300`  | 5% or more of the last five minutes' requests are 5xx             |
| `p95_latency_ms`        | `8000`          | `8,000 ms`                      | `600`  | p95 latency over ten minutes reaches 8 seconds                    |
| `spend_velocity`        | `50`            | `$50.00/h`                      | `3600` | the last hour spent 50 or more in the settlement currency         |
| `request_volume`        | `1000`          | `1,000 requests in 5m`          | `300`  | a thousand or more requests in five minutes, about three a second |
| `provider_health_flaps` | `10`            | `10 failed health events in 5m` | `300`  | ten or more failed health observations in 5 minutes               |

Tune `request_volume` to your own traffic. With the default `above` it catches a
surge; for a drop, use `comparison: below` (see
[No data and below rules](#no-data-and-below-rules)).

### Changing a rule or channel

The edit button on a rule or channel card opens the sheet that adds one, filled
in from the row. Change the threshold, window, signal, channel or endpoint and
choose **Save**. Do not delete and recreate instead: deleting a rule also deletes
its history, and deleting a channel clears it from every rule that delivers
through it. An edited rule keeps its state and history, and the change applies
from its next evaluation.

Saving sends the enable switch back as it was, so editing a rule or channel that
is switched off leaves it off. The edit button follows the same permission as
the switch, `alert_rule` or `alert_channel` update.

An existing rule opens on its own threshold, in the signal's unit: a stored
`0.05` for `error_rate` opens as `5`. Only choosing another signal resets it to
that signal's default.

A channel's secret field opens empty, because a stored secret is never read
back. Leave it blank to keep the stored secret, or type one to replace it. A
secret belongs to the endpoint's origin, so pointing the channel at another
scheme, host or port without typing a new secret drops the stored one. The
sheet says so beside the field before you save.

The same change through the API is a `PUT`, which replaces the whole row, so
send every field. A body without `enabled` turns the rule or channel off.

```bash
curl -X PUT https://control.example.com/api/v1/alert-rules/$RULE_ID \
  -H "Authorization: Bearer $ROLTER_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "high error rate",
        "signal": "error_rate",
        "threshold": 0.02,
        "window_secs": 300,
        "channel_id": "3f6c2a4e-8d1b-4c1e-9a57-2b0d6e9f1c23",
        "enabled": true
      }'
```

A channel update takes `name`, `endpoint`, `enabled` and, only to replace it,
`managed_secret`.

## History

**Alerting → History** lists the rows 100 at a time, newest first, with the
state (`firing` or `resolved`), delivery status, time, rule, reading and detail.
The state and delivery columns come first so they stay in view on a phone. Every
change of state has a row, including those of rules with no channel.

A full page may be followed by older rows, and **Load older transitions** under
the list reads the next page from the last row shown (the `before` cursor below).
A page that comes back short is the end, and the button goes away. The toolbar
narrows the list, and each picker is sent to the API, so it reaches its own
newest rows rather than searching the page already loaded:

- **Rule** is sent as `rule_id`.
- **State** is sent as `state`.
- **Delivery** is sent as `delivery_status`.

Changing a picker starts the list again from the newest matching row.

The **Reading** column shows the value the rule read when the row was written and,
under it, the threshold it was held against, both in the signal's unit (`11%`
above `5%`, `840 ms` above `2,000 ms`). A transition written for a window with no
data reads **No data**, and a row from before readings were recorded shows `—`.

The **Detail** column and the rule card's `last_error` are translated from the
`detail_code` and `last_error_code` below, with their `detail_params` /
`last_error_params` filling in the numbers. A code this build does not know, a
row written before codes existed, and a code whose numbers did not arrive show
the control plane's English `detail` / `last_error` as stored. A long detail wraps
onto further lines inside its column rather than being cut, so a diagnosis such
as `channel secret could not be unsealed; check ROLTER_KEK` reads in full. On a
phone the detail is the last column, so scroll the list sideways to reach it.

| Delivery status | Detail                                                                                                                                                                                                                                                                                                                                                    |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pending`       | `delivery in progress`. The transition is recorded and the request is being sent; the row becomes `delivered` or `failed` when it ends. A row still `pending` after 5 minutes is taken as lost (the control plane stopped before sending) and the state is sent again.                                                                                    |
| `delivered`     | The endpoint's status, for example `HTTP 200`.                                                                                                                                                                                                                                                                                                            |
| `failed`        | The endpoint's status when it answered outside `2xx` (`HTTP 401`, `HTTP 503`, `HTTP 302`), or what stopped the request: `timed out after 10s`, `could not connect to the endpoint`, `delivery failed`, `endpoint denied by the egress policy`, `channel secret could not be unsealed; check ROLTER_KEK`, or `channel secret is not a valid header value`. |
| `skipped`       | `no channel configured` or `channel disabled`.                                                                                                                                                                                                                                                                                                            |

A detail never contains the endpoint's response body, which can echo the
credential it rejected. `sent_at` is when the transition was recorded, just before the
delivery attempt started. A [retried](#retries) state has one row per attempt.

The history is append-only: the API has no update or delete for it. Deleting a
rule deletes that rule's rows along with it.

Each row also carries `detail_code`, a stable code for the outcome, and
`detail_params`, the numbers the text interpolates. `detail` stays as the English
fallback. A row written before codes existed has `null` for both.

| `detail_code`        | `detail_params` | Outcome                                                      |
| -------------------- | --------------- | ------------------------------------------------------------ |
| `delivery_pending`   |                 | `pending`: the request is being sent.                        |
| `http_ok`            | `status`        | `delivered`: the endpoint answered `2xx`.                    |
| `http_error`         | `status`        | `failed`: the endpoint answered outside `2xx`.               |
| `timeout`            | `seconds`       | `failed`: the delivery ran past the bound.                   |
| `connect_failed`     |                 | `failed`: the endpoint did not accept the connection.        |
| `delivery_failed`    |                 | `failed`: another transport failure.                         |
| `egress_denied`      |                 | `failed`: the egress policy refuses the endpoint.            |
| `secret_unsealable`  |                 | `failed`: the secret cannot be unsealed; check `ROLTER_KEK`. |
| `secret_invalid`     |                 | `failed`: the secret is not a valid header value.            |
| `client_unavailable` |                 | `failed`: the webhook client could not be built.             |
| `no_channel`         |                 | `skipped`: the rule has no channel.                          |
| `channel_disabled`   |                 | `skipped`: the channel is disabled.                          |

A row written for a window with no data adds `"no_data": true` to its
`detail_params`.

`value` and `threshold` record the reading the row was written for and the rule's
threshold at that moment, in the signal's API unit (see [Delivery](#delivery)).
`value` is `null` for a no-data transition and on older rows. The **Evaluate
now** response carries the same fields on its `notification`.

The API takes these optional query parameters:

| Parameter         | Meaning                                                         |
| ----------------- | --------------------------------------------------------------- |
| `rule_id`         | One rule's rows. Leave out for every rule.                      |
| `state`           | `firing` or `resolved`.                                         |
| `delivery_status` | `delivered`, `failed` or `skipped`.                             |
| `limit`           | Page size, default `100`, clamped to `1`–`500`.                 |
| `before`          | Cursor `<sent_at>,<id>`: the rows strictly older than that row. |

The filters combine. The list is newest first, ordered by `sent_at` then `id`, so
rows written in the same instant keep a stable order. To read the next page, pass
the last row's `sent_at` and `id` as `before`; an empty page, or one shorter than
`limit`, is the end. A value outside the lists above, or a malformed `before`,
answers `400` with code `invalid_field`.

```bash
curl -s "https://control.example.com/api/v1/alert-notifications?delivery_status=failed&limit=50&before=2026-09-27T10:15:02.184Z,0d9c7a52-5f1e-4a8b-9d0e-3c2b1a0f9e8d" \
  -H "Authorization: Bearer $ROLTER_TOKEN"
```

## Scopes and ownership

A rule can watch one **scope** instead of the deployment: an organization, a
team, a project, a virtual key, a business unit or a customer. It then
evaluates only the request-log rows of that scope, so a team lead's error-rate
rule sees their team's traffic and nothing else. The scope is set when the rule
is created (`scope_type` and `scope_id` in the body, given together) and cannot
be changed afterwards; create a new rule to watch another scope.
`provider_health_flaps` is recorded per organization, so it can only watch an
organization or the whole deployment. An organization's rule counts the health
events of the providers that organization owns; providers defined in the
gateway's own config file belong to no organization and are counted only by a
deployment-wide rule. A rule whose team, project or key has
since been deleted reports a scope error (`scope_gone`) and does not fire;
delete it.

Creating, editing, deleting and evaluating a rule takes **admin at its scope**:
an organization admin for an organization rule, a team admin for a team's, and
so on. A rule for a business unit or a customer is the organization admin's. A
rule with no scope, which reads every tenant, stays superadmin-only.

A **channel** belongs to one organization or to the deployment. An organization
admin owns the channels of their organization (`org_id` in the create body); a
deployment channel is created by a superadmin and can be named by any rule. A
rule can notify only a channel of its own organization or a deployment channel,
and a deployment-wide rule only a deployment channel. The control plane refuses
anything else with a `400` on `channel_id`, answering an unknown channel the
same way so the id cannot be probed, and it checks again at delivery. Names
are unique per owner, so two organizations can each have a rule called
`prod errors`.

The lists are narrowed to what the caller may use. Rules and notification
history show only the scopes the caller administers. Channels show the
caller's organizations' channels and the deployment's, but a deployment
channel's address is withheld (`endpoint` is empty and `manageable` is false),
as it may carry a token. A caller who administers nothing gets `403`.

## Permissions and audit

Every operation is decided at the owner of the row, as described above, and the
dashboard shows the Alerting group to anyone who administers a scope. Deployment
rows and channels are for a superadmin or an operator (a [deployment role](/security/rbac#deployment-roles)), like the other deployment-wide settings in
[RBAC](/security/rbac).

| Endpoint                                    | Capability                     |
| ------------------------------------------- | ------------------------------ |
| `GET`, `POST /api/v1/alert-channels`        | `alert_channel` read, create   |
| `PUT`, `DELETE /api/v1/alert-channels/{id}` | `alert_channel` update, delete |
| `GET`, `POST /api/v1/alert-rules`           | `alert_rule` read, create      |
| `PUT`, `DELETE /api/v1/alert-rules/{id}`    | `alert_rule` update, delete    |
| `POST /api/v1/alert-rules/{id}/evaluate`    | `alert_history` create         |
| `GET /api/v1/alert-notifications`           | `alert_history` read           |

Evaluating a rule by hand counts as creating history, because it can write a
row.

Every create, update and delete of a channel or rule writes an audit entry
(`alert.channel.create`, `alert.rule.delete` and so on). Create and update
entries carry the name, the enabled flag and, for a rule, the signal and scope. A channel
update entry also records `endpoint_changed`, `secret_replaced` and
`secret_cleared`, as booleans. Neither a channel secret nor its endpoint ever
appears in one, since an endpoint's query string can carry a token.

All of these operations are in the control plane's
[OpenAPI document](/api/control-plane-openapi) under the `alerting` tag.