What to Look At in Rolter Telemetry

Collecting telemetry is easy; using it is the part that gets skipped. This page is the short list — the queries worth running, and what a healthy answer looks like, so a number means something the first time you see it.

Two questions, two data sources:

• Why was a request slow? → traces in SigNoz (Tracing)
• Is a dashboard screen usable? → the ui_events table in ClickHouse (UX Events)

Both carry trace_id, so the two are one join apart rather than two systems apart.

Performance: why was that request slow?#

Where the time actually goes#

Rolter emits one span per pipeline stage, so a slow request is attributable to a named stage rather than merely slow. In SigNoz, open Traces, filter service.name = rolter-gateway, and sort by duration descending.

Read the waterfall top-down and ask which stage is wide:

p95 latency by model, from the request log#

Traces are sampled, and the request log is complete while its sample_rate is 1, the default. For “is this actually getting worse”, use the log. Below 1 the percentiles and the error share below are estimates from the logged requests, and requests counts only those; the dashboard’s analytics endpoints scale their counts and sums by each row’s sample_weight, a raw query does not.

SELECT    model,    count()                                    AS requests,    round(quantile(0.50)(latency_ms))          AS p50_ms,    round(quantile(0.95)(latency_ms))          AS p95_ms,    round(100 * countIf(status >= 400) / count(), 2) AS error_pctFROM request_logsWHERE ts > now() - INTERVAL 24 HOURGROUP BY modelORDER BY requests DESCLIMIT 20;

Healthy shape: p95_ms within roughly 3× of p50_ms. A much wider gap means a slow minority — usually queueing or retries, and the trace waterfall will say which.

UX: is the dashboard usable?#

Which screens are slow to become usable#

time_to_interactive is emitted when a screen’s primary data lands, so this is time-to-usable, not time-to-first-paint.

SELECT    screen,    count()                           AS views,    round(quantile(0.50)(duration_ms)) AS p50_ms,    round(quantile(0.95)(duration_ms)) AS p95_msFROM ui_eventsWHERE action = 'time_to_interactive'  AND ts > now() - INTERVAL 7 DAYGROUP BY screenHAVING views > 20ORDER BY p95_ms DESCLIMIT 20;

Healthy shape: p95 under ~1s for list screens. Anything over 3s is a screen people will describe as broken, whatever the p50 says.

Which screens people back out of#

A back-out is leaving via the browser’s back button rather than by acting on the screen. The dwell time is what makes it meaningful.

SELECT    screen,    count()                            AS back_outs,    round(quantile(0.50)(duration_ms)) AS p50_dwell_msFROM ui_eventsWHERE action = 'back_out'  AND ts > now() - INTERVAL 7 DAYGROUP BY screenORDER BY back_outs DESCLIMIT 20;

How to read it: a short dwell (under ~5s) with many back-outs is a navigation problem — people are landing somewhere they did not intend, so the route in is mislabelled. A long dwell is a content problem — they meant to be there, tried, and gave up.

Which forms get abandoned#

SELECT    screen,    target                                          AS form,    countIf(action = 'form_submit')                 AS submits,    countIf(action = 'form_abandon')                AS abandons,    round(100 * countIf(action = 'form_abandon')          / nullIf(countIf(action IN ('form_submit','form_abandon')), 0), 1) AS abandon_pct,    round(quantileIf(0.50)(duration_ms, action = 'form_abandon')) AS p50_dwell_msFROM ui_eventsWHERE action IN ('form_submit', 'form_abandon')  AND ts > now() - INTERVAL 30 DAYGROUP BY screen, targetHAVING submits + abandons > 10ORDER BY abandon_pct DESC;

How to read it: high abandon_pct with a short dwell means the form was opened by mistake — usually an ambiguous button. High abandon with a long dwell means people tried and could not finish, which is the one worth fixing.

Which validation rules actually fire#

target is the rule’s name; the value that failed it is never recorded.

SELECT screen, target AS rule, count() AS hitsFROM ui_eventsWHERE action = 'validation_error'  AND ts > now() - INTERVAL 30 DAYGROUP BY screen, ruleORDER BY hits DESCLIMIT 20;

A rule at the top of this list is usually a field whose format is not explained before it is enforced.

Dead states and hot error states#

A state we wrote and nobody ever sees is dead code; one we ship and everyone hits is a bug.

SELECT    screen,    target                                    AS region,    countIf(action = 'empty_state')           AS empty_impressions,    countIf(action = 'error_state')           AS error_impressionsFROM ui_eventsWHERE action IN ('empty_state', 'error_state')  AND ts > now() - INTERVAL 30 DAYGROUP BY screen, regionORDER BY error_impressions DESC, empty_impressions DESC;

Both columns come from the dashboard’s shared placeholders: every empty state records empty_state and every load-failure alert records error_state, each time it appears, under the screen it is on. region is the part of the screen that came back empty or failed, and an empty state and the alert for the same list carry the same region, so one row reads both. A screen that shows several alerts at once records one row for each.

A region ending in -stale (dashboard-spend-stale) is a different thing: the card still shows its earlier data, and a refresh of it failed. There is no alert, so the figures are going out of date unseen. Each appearance of the “refresh failed” line records one error_state under the card’s region with -stale added, and polls that keep failing under the same line add nothing.

Sustained error_impressions on one screen is a bug report nobody filed. High empty_impressions on a screen that should have data usually means a scope or permission problem, not an authoring one.

Save-to-confirmation latency#

How long the user waits between pressing save and being told it worked.

SELECT    screen,    target                              AS form,    count()                             AS saves,    round(quantile(0.95)(duration_ms))  AS p95_msFROM ui_eventsWHERE action = 'save_confirmed'  AND ts > now() - INTERVAL 7 DAYGROUP BY screen, formORDER BY p95_ms DESC;

Healthy shape: under ~1s. Past about 2s people press save twice, so this number predicts duplicate writes.

Where people get stuck#

Three actions record what somebody could not do, which is the part a support ticket usually starts from.

SELECT    action,    screen,    target,    count() AS hitsFROM ui_eventsWHERE action IN ('retry_submit', 'refused_click', 'abandon_dirty')  AND ts > now() - INTERVAL 7 DAYGROUP BY action, screen, targetORDER BY hits DESCLIMIT 20;

How to read it. A retry_submit is somebody who did not understand why the first save failed — pair it with validation_error on the same form to see whether the message was the problem. An abandon_dirty is somebody who filled the form in and gave up, which is a much stronger signal than the plain form_abandon beside it. A refused_click is a reach for a control the caller’s role denies, and its target is <control>:<resource>:<action>, so the capability is right there:

SELECT    splitByChar(':', target)[-2] || ':' || splitByChar(':', target)[-1] AS capability,    count() AS refusalsFROM ui_eventsWHERE action = 'refused_click'  AND ts > now() - INTERVAL 30 DAYGROUP BY capabilityORDER BY refusals DESC;

A capability at the top of that list is one your role model puts in people’s way repeatedly. The row deliberately carries no identity beyond the session and scope ids every event has, so it answers “which boundary is in the way”, not “who kept trying”.

Joining the two: a slow screen and the request behind it#

This is the reason both streams carry trace_id. When a screen is slow, this says whether the dashboard or the gateway is responsible.

SELECT    e.screen,    e.duration_ms          AS screen_ms,    r.model,    r.latency_ms           AS gateway_ms,    r.statusFROM ui_events AS eINNER JOIN request_logs AS r ON r.trace_id = e.trace_idWHERE e.action = 'time_to_interactive'  AND e.duration_ms > 2000  AND e.ts > now() - INTERVAL 24 HOURORDER BY e.duration_ms DESCLIMIT 50;

How to read it: if gateway_ms is a small fraction of screen_ms, the backend was fine and the time went on the client — rendering, or requests the screen makes in series that could be parallel. If the two are close, fix the gateway path first.

Before you trust any of it#

Three checks, in order — each one explains an empty result you would otherwise misread as good news.

1. Are events arriving at all?

   SELECT action, count() FROM ui_eventsWHERE ts > now() - INTERVAL 1 HOUR GROUP BY action ORDER BY 2 DESC;

   Nothing at all means clickhouse_url is unset, logging.ui_events = false, the store is refusing the writes, or nobody has used the dashboard in an hour. A refused write is the one to rule out first: it shows up as rolter_control_ingest_failures rising, and as a telemetry ingest failed warning in the control-plane log at most once a minute, naming the store’s own error.

2. Is a screen missing on purpose? screen_view, navigate, back_out, empty_state and error_state are wired centrally and cover every screen. time_to_interactive is per-screen. A screen absent from it is un-instrumented, not fast.

3. Compare like with like. app_version is on every row. After a deploy, filter to one version before concluding a screen got slower:

   ... WHERE app_version = '1.4.2'

Retention#

ui_events carries the same 90-day TTL and daily partitioning as request_logs, so none of this needs its own retention policy. For anything you want to keep beyond that, roll it into a summary table on your own schedule — these queries all aggregate cleanly by day.