Alert on symptoms, not causes

TL;DR: Page on the thing a user would notice: requests failing, latency past the threshold, data going stale. One symptom alert covers dozens of causes including the ones nobody thought of, and it stays quiet when a cause is present but users are fine, which is most of the time.

Why cause alerts multiply

A cause alert names an internal condition: CPU above 80 percent, disk above 70, a pod restarted, replication lag above ten seconds, queue depth above a thousand. Each was added after an incident where that condition mattered, which is why they accumulate and never leave.

The problem is that these conditions are true constantly in a healthy system. Something is always at 80 percent somewhere. A pod restarts. A queue backs up for ninety seconds and drains. None of it reaches a user, all of it pages someone, and within a quarter the rotation has learned that pages are usually noise.

That learning is rational, and it is the actual danger. The responder who ignores the twentieth CPU alert this week will ignore the real one at the same rate.

What a symptom alert covers

Alert on the SLI: successful requests over total, latency inside the objective, freshness of the data a consumer reads. That single alert fires for every cause that matters, including a cause nobody anticipated, which is the class that produces the worst incidents.

It also stays silent for causes that do not matter. A node at 95 percent CPU while latency is fine is a capacity planning input, not a page. Moving it to a dashboard loses nothing.

The test a page has to pass

Three conditions, all of them:

1. Urgent. The system will not recover on its own.
2. Actionable. A person can do something about it now.
3. Needs a human now. It cannot wait until morning.

Anything failing one goes somewhere else. Not deleted, which is how information gets lost and how the next incident review recreates it, but moved to a ticket queue or a channel where it still exists and nobody is woken by it.

Burn rate is how a symptom alert gets urgency

A raw error-rate threshold has no notion of how much budget remains, so it fires the same way for a brief blip and a sustained outage. Burn-rate alerting fixes that: alert when the current consumption rate would exhaust the error budget faster than acceptable, evaluated over two windows so a short spike does not page and a slow bleed does.

For each alert, require the burn threshold in both a longer observation window and a shorter confirmation window. Different pairs can represent different urgencies: for example, a 14.4× threshold over one hour and five minutes can page, while a lower sustained threshold can create a ticket. Combine the windows within each pair with AND; do not page on the short window alone.

Where cause alerts still earn a place

They are diagnostics and they are useful as such: linked from the runbook, on the dashboard, present in the incident channel. They also earn a page in one narrow case, when the condition is a reliable leading indicator with enough lead time to act and no user-visible symptom yet. Certificate expiry in seven days normally warrants a ticket and escalation before the safe remediation deadline, rather than an overnight page. A disk forecast to fill in four hours may warrant a page if intervention cannot wait. GPU memory pressure needs an admission or capacity action tied to the imminent impact.

The distinction is prediction, not internality. If it predicts material harm before routine response can act, and a human can prevent that harm now, it can page. If it merely describes the system's mood, it cannot.

Self-check

A certificate expires next week and renewal takes one hour during working time. Should it page tonight? Usually create an owned ticket with a deadline and escalation. Page if the remaining safe remediation window makes waiting dangerous, not merely because the condition predicts a failure.

Sources: Google SRE multiwindow alerts.