TL;DR: Delete most of them. An alert may page only if it is urgent, actionable, and a human must act now; everything else is a ticket or a dashboard. Replace cause-based threshold alerts with symptom-based burn-rate alerts against your SLO, using a fast and a slow window so severity tracks consequence.
How to approach it
State the rule for what may page before describing any mechanism, because the mechanism is downstream of the rule. Then give the multi-window burn rate concretely, with numbers.
A strong answer
The failure is structural. Alerts accumulate: every incident produces a new one, none are ever removed, and most fire on causes (CPU above 80 percent, disk above 70, pod restarted) rather than on whether users are affected. Cause alerts fire constantly because systems are always slightly unhealthy somewhere, and once a page is usually noise, on-call learns to defer all of them, including the real one. That behaviour is rational, which is why exhorting people to take alerts seriously never works.
The rule I would apply, and enforce in review: a page requires all three of urgent, actionable, and needing a human now. If automatic recovery will arrive before unacceptable user harm, a page may be unnecessary; a long self-recovering outage can still need intervention. If there is no action, it is a dashboard. If it can wait until morning, it is a ticket. Anything failing the test gets downgraded, not deleted quietly but moved to a channel or a queue where it still exists.
Then change what is measured. Alert on symptoms, meaning the SLI a user would recognise: successful requests, latency under a threshold. One symptom alert covers dozens of causes, including the ones you have not thought of, and it does not fire when a cause is present but users are fine, which is most of the time.
Burn rate is the mechanism. Error budget is the allowance for a window; burn rate is how fast you are spending it. Rate 1 means you exactly exhaust it by the window's end. For a 30-day budget window, this example pairs each long window with a short confirmation window:
| Burn rate | Window | Budget spent | Action |
|---|---|---|---|
| 14.4x | 1 hour and 5 minutes | 2% | page |
| 6x | 6 hours and 30 minutes | 5% | page |
| 3x | 1 day and 2 hours | 10% | ticket |
| 1x | 3 days and 6 hours | 10% | ticket |
The fast window catches a sharp outage in minutes. The slow window catches a gradual degradation that a threshold alert never notices. Requiring a shorter confirmation window alongside each stops a brief spike paging anyone, which is what removes most of the remaining noise.
Around it: include a runbook link where available, but judge actionability by what the responder can do; a link alone proves nothing. Related alerts group into one notification rather than one per pod. And review the page volume monthly with the same seriousness as an incident, because the trend is the health metric for the whole system. A reasonable target is fewer than two pages per on-call shift, and if you are far above that the answer is not more resilience training.
What interviewers probe next
"What would you still alert on that is not a symptom?" A small set of predictive ones where the lead time matters: certificate expiry, disk filling at a rate that reaches full in hours, quota approaching a hard limit. All are actionable and none are recoverable after the fact.
"How do you handle a service with too little traffic for an SLO?" Burn rate is noisy at low volume. Use synthetic probes to create a baseline, or accept longer windows and lower confidence.
"Who owns the alert?" The team that owns the service. A central team writing alerts for services it does not operate is how the noise accumulated originally.
Common mistakes
Tuning thresholds instead of changing what is measured. It reduces volume for a quarter and it comes back.
Paging on cause metrics, which is the root of the problem.
Deleting alerts with no replacement and no owner, which trades noise for blindness.
References