Monitoring and Alerting: Knowing When a Service Needs Attention

Production systems produce more signals than any person can watch continuously.

Monitoring collects and evaluates those signals. Alerting interrupts someone when a condition requires timely human action.

An alert should be urgent, actionable, user-relevant, and worth interrupting a person.

Everything else may belong in a dashboard, ticket, report, or automated response.

A concrete example: checkout failures

One application instance restarts and the load balancer replaces it automatically.

No user-visible failure occurs, so a page is unnecessary.

Later, checkout success falls below 98 percent for ten minutes across healthy instances. The alert pages the on-call responder with:

• impact,
• start time,
• region,
• recent deployment,
• dashboard,
• and runbook.

Monitoring

Monitoring observes known system questions:

• Is the service reachable?
• Are users succeeding?
• Is latency within target?
• Is capacity approaching a limit?
• Are backups completing?
• Is a queue falling behind?

Observability provides evidence for novel investigation; monitoring continually evaluates selected conditions.

Symptom-based alerts

Alert on user-visible symptoms where possible:

• high failed-checkout rate,
• slow API objective,
• report jobs missing deadlines,
• or authentication unavailable.

Cause-based alerts such as high CPU are useful when they require direct action before user impact, but often belong as diagnostic context.

Black-box monitoring

Black-box checks observe the service from outside.

Examples:

• DNS resolution,
• TLS connection,
• homepage request,
• synthetic sign-in,
• or API transaction.

They approximate user experience and can catch failures that internal health metrics miss.

White-box monitoring

White-box signals come from inside:

• error rate,
• saturation,
• thread pools,
• replication lag,
• queue age,
• or cache hit rate.

They explain cause and provide earlier warning. Use both external symptoms and internal evidence.

The four golden signals

A common framework watches:

• latency,
• traffic,
• errors,
• saturation.

It is a starting point, not a substitute for domain measures such as payment confirmation or message-delivery deadline.

Threshold alerts

A static threshold triggers when a measurement crosses a value:

checkout errors > 2 percent for 10 minutes

Choose duration and volume carefully. Two failures out of two requests are statistically different from 2,000 out of 100,000.

Rate and ratio

Ratios often represent user impact better than raw counts.

One hundred failures may be severe at low traffic and minor at massive traffic. Include minimum request volume to avoid unstable ratios during quiet periods.

Count-based alerts remain useful for events where one occurrence matters, such as unauthorized key use.

Anomaly detection

Anomaly systems compare current behavior with learned history.

They help with seasonal signals but can:

• learn bad behavior as normal,
• alert on legitimate launches,
• or miss entirely new failure patterns.

Use anomaly evidence with explicit objectives and ownership.

Alert severity

Define levels such as:

• page now,
• urgent ticket,
• business-hours review,
• informational dashboard.

Severity should reflect response deadline and user impact, not which component emitted the signal.

An "error" log does not automatically deserve a page.

Actionability

An alert should state:

• what is affected,
• how large the impact is,
• what changed recently,
• where to investigate,
• which immediate actions are safe,
• and who owns the service.

If no responder action exists, automate the response or change the signal.

Alert fatigue

Frequent low-value pages train people to ignore all pages.

Track:

• alerts per shift,
• repeated alerts,
• false positives,
• pages with no action,
• and time to acknowledge.

Every page should receive review as a cost imposed on the team.

Flapping

An alert flaps when it repeatedly opens and closes around a threshold.

Use:

• sustained windows,
• different trigger and recovery thresholds,
• aggregation,
• and stabilization.

Do not hide real intermittent failure; understand its user effect.

Deduplication and grouping

One database failure can trigger hundreds of service alerts.

Group related signals by incident and suppress downstream symptoms where reliable dependency context exists. Preserve evidence so responders can see scope without receiving one page per instance.

Over-aggressive suppression can hide independent failure.

Maintenance windows

Planned work may intentionally violate normal signals.

Silence narrowly by:

• service,
• environment,
• time,
• and known change.

Broad permanent mute creates blind spots. Verify alerts return after maintenance.

On-call ownership

Every page needs a reachable team with:

• access,
• training,
• runbooks,
• authority to mitigate,
• and escalation.

Paging a person who cannot act only delays response.

Runbooks

A runbook should include:

• meaning of the alert,
• confirmation steps,
• common causes,
• safe mitigations,
• verification,
• escalation,
• and links to evidence.

Runbooks should be tested during drills and updated after incidents.

Synthetic monitoring

Synthetic checks perform controlled transactions periodically.

Use dedicated accounts and prevent real charges, emails, or destructive effects. Run from representative locations and distinguish synthetic traffic in logs.

A shallow health endpoint should not be the only external check.

Alerting on objectives

Service-level objectives can drive alerts based on error-budget consumption.

A fast burn indicates severe current impact; a slow burn shows a smaller issue that will violate the objective if sustained.

Multi-window alerts balance urgency and noise.

Testing alerts

Verify:

• the condition can trigger,
• notification reaches the right route,
• message contains context,
• runbook access works,
• escalation occurs,
• and recovery closes correctly.

An alert configured but never exercised is an untested production feature.

Knowledge check

1. How do monitoring and observability differ?
2. Why are symptom-based alerts often better than instance alerts?
3. What makes a ratio alert unreliable at low volume?
4. How does alert fatigue affect incident response?
5. Why should alert routing and runbooks be tested?

The one idea to remember

Monitoring evaluates known service signals; alerting asks a person to act. Page on sustained, meaningful user impact or imminent risk, include actionable context, control noise, assign capable ownership, and test the entire response path.