If your on-call phone buzzes five times a night and four of those you’ve learned to swipe away without looking, your alerting isn’t protecting production — it’s training you to ignore it. That’s the actual failure mode: the alert that matters arrives indistinguishable from the ones that don’t, and gets swiped away with the rest.

The test for every alert: does it need a human, tonight?

Not “is this worth knowing.” Not “is this a real problem.” Specifically: does a human need to act on this right now, or can it wait until business hours, or does it need no action at all?

• CPU at 85% for two minutes, self-resolved: not a page. Log it, dashboard it, don’t wake anyone.
• Disk usage climbing toward full over the next six hours: a page during business hours, not at 3am — there’s runway to act on it in the morning.
• API error rate above 10% for five minutes, ongoing: a page, right now, any hour.

If you can’t articulate what a human does differently at 3am versus 9am for a given alert, it’s not a page — downgrade it to a dashboard metric or a daytime-only ticket.

As a decision:

flowchart TD
    A["Alert fires"] --> B{"Does a human need\nto act right now?"}
    B -->|"No action needed"| C["Dashboard only"]
    B -->|"Can wait for business hours"| D["Warning: Slack,\ndaytime-reviewed"]
    B -->|"Yes, tonight"| E["Page: phone call/SMS,\nany hour"]
  

Warning vs. page, as separate channels

Collapsing every severity into one notification channel is the single most common cause of fatigue. Split them explicitly:

If “disk 70% full” and “API is fully down” arrive through the same phone alert, your brain stops distinguishing them by the third night, and by the tenth it swipes both away identically.

Deduplication: one incident, one alert

A database going down often triggers a dozen downstream alerts simultaneously — every service that depends on it starts failing its own healthcheck. Without deduplication, that’s a dozen pages for one root cause, and after enough of those nights you start ignoring bursts.

Most alerting tools (PagerDuty, Better Stack, even Uptime Kuma with the right config) support grouping alerts by a shared key within a time window:

# conceptual — actual config depends on your tool
alert_grouping:
  key: root_cause_tag
  window: 5m

Tag alerts that share a likely root cause (same downstream dependency) so they collapse into one notification instead of ten.

SLOs instead of static thresholds

A static threshold (“page if latency > 500ms”) pages you for every transient blip, because real-world latency is noisy. An SLO-based alert pages you when you’re burning error budget fast enough to actually blow your target over a meaningful window — which filters out the noise that a flat threshold can’t.

Target: 99.9% of requests under 500ms, over 30 days
Alert: page if the current burn rate would exhaust the monthly error budget within 6 hours

This takes more setup than a flat threshold, but it’s the difference between “paged because of a 90-second blip that self-resolved” and “paged because something’s actually degrading fast enough to blow the target.”

Escalation policies stop the same person from absorbing everything

If one page goes unacknowledged for 10-15 minutes, it should escalate to a second person — not silently retry the same phone forever. Without escalation, a missed page (phone on silent, in the shower) means the incident just sits, unaddressed, until someone happens to check.

The audit worth doing this week

Pull your last 30 days of pages. For each one, write one line: what did the responder actually do? If the answer is “nothing, it self-resolved” or “waited until morning” for more than a small fraction of them, those alerts are miscategorized — they’re warnings wearing a page’s severity, and every one of them is spending down your team’s trust in the phone that’s supposed to wake them up for the alert that actually matters.

Takeaway

Fewer, better-targeted pages beat comprehensive coverage. An alert that doesn’t require immediate human action is a dashboard entry, not a page — sorting your alerts into that split, honestly, is most of the fix.