> ## Documentation index
> Fetch the complete documentation index at: https://rolter-ai-rolter.docs7.io/llms.txt
> Ask it with GET https://rolter-ai-rolter.docs7.io/ask?q=<keywords or question>. Short keywords rank best.
> Use these to discover all available pages before exploring further.

# Performance Tuning: Retries, Timeouts and the Admission Queue

> Tune Rolter's upstream retry budget, connect and request timeouts, and the per-provider bounded admission queue from the dashboard or the management API.

One deployment-wide runtime policy governs how Rolter talks to upstreams under load: how hard it retries, how long it waits, and how many requests it admits at once. Edit it from **Settings → Performance Tuning** in the dashboard, or through the management API.

---

## Retries

| Setting             | Range    | Meaning                                   |
| ------------------- | -------- | ----------------------------------------- |
| `retry_max_retries` | 0–10     | Retry budget for a single client request. |
| `retry_base_ms`     | 0–60000  | First backoff interval.                   |
| `retry_max_ms`      | 0–600000 | Ceiling the exponential backoff grows to. |

The cap cannot be lower than the base — an incoherent pair is rejected rather than silently clamped.

---

## Timeouts

`timeout_connect_s` (0–300) bounds establishing the upstream connection; `timeout_request_s` (0–3600) bounds the wait for the upstream's response headers. Once the headers arrive the body is not timed, so a streaming response that has started runs to completion however long it takes. Zero disables the corresponding timeout.

---

## Admission queue

Rolter creates an **independent bounded queue and worker set per provider**, so one slow provider cannot consume the admission capacity that healthy providers need.

| Setting              | Range                      | Meaning                                                                    |
| -------------------- | -------------------------- | -------------------------------------------------------------------------- |
| `queue_capacity`     | 1–100000                   | Requests that may wait for a worker.                                       |
| `queue_workers`      | 1–2048                     | Requests per provider waiting for response headers (does not cap streams). |
| `queue_backpressure` | `drop` / `block` / `error` | What happens when the queue is full.                                       |
| `queue_block_ms`     | 0–120000                   | How long `block` waits for capacity.                                       |

On the dashboard the queue fields are disabled while the queue is off, and the block timeout unless the policy is `block`. A disabled field is not validated, so an out-of-range value already stored never blocks **Save**, and saving keeps it unchanged. It is checked again once the field is enabled.

The dashboard shows each problem under the field it belongs to, all at once, and the backpressure dropdown offers `Drop`, `Block` and `Error` for the wire values below. **Save** and **Discard changes** wait for an edit; a panel holding one is marked **Edited**, the footer counts the unsaved settings, and leaving the screen with unsaved edits asks first.

The three backpressure policies:

- **`drop`** — shed the request immediately with an overload response.
- **`block`** — wait for capacity until `queue_block_ms` expires. This requires a non-zero timeout; blocking forever would park callers indefinitely, so the combination is rejected.
- **`error`** — return a structured `queue_full` error immediately. This is the default.

Shed requests never fail over to a sibling target and never retry: admission
backpressure is a signal to the caller, not an upstream fault. Each policy has a
fixed wire shape:

| Policy              | HTTP status | `error.code`    | `error.type`       |
| ------------------- | ----------- | --------------- | ------------------ |
| `drop`              | 503         | `queue_dropped` | `overloaded_error` |
| `block` (timed out) | 429         | `queue_timeout` | `rate_limit_error` |
| `error`             | 429         | `queue_full`    | `rate_limit_error` |

```json
{
  "error": {
    "message": "provider queue full",
    "type": "rate_limit_error",
    "param": null,
    "code": "queue_full"
  }
}
```

Rejections and timeouts are counted separately on the gateway metrics endpoint as
`rolter_provider_queue_rejections_total` and
`rolter_provider_queue_timeouts_total`, so backpressure is visible as a real
counter rather than only as a client-side status code.

---

## API

Both endpoints are for a **superadmin** or an **operator** (a [deployment role](/security/rbac#deployment-roles)). Every update is audited.

```bash
curl -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" \
  https://control.example.com/api/v1/runtime-policy

curl -X PUT -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "retry_max_retries": 2,
        "retry_base_ms": 200,
        "retry_max_ms": 5000,
        "timeout_connect_s": 10,
        "timeout_request_s": 120,
        "queue_enabled": true,
        "queue_capacity": 1024,
        "queue_workers": 32,
        "queue_backpressure": "error",
        "queue_block_ms": 0
      }' \
  https://control.example.com/api/v1/runtime-policy
```

Send the full policy, not a patch. A successful update bumps `config_version`, so gateways pick it up on their next snapshot poll without a restart.