Metrics and Alerts

RED (Rate/Errors/Duration) provides an external perspective on the server side, while USE (Utilization/Saturation/Errors) offers an internal perspective on resources—the two are complementary. SLI is the actual measured value, SLO is the committed target, and SLA is the consequence of breach. The first principle of alert design: page only for things requiring immediate human action; everything else goes to a dashboard. False positives train on-call engineers to ignore alerts, which is more dangerous than missing alerts.

RED and USE Methodologies

Two complementary perspectives:

RED (Rate, Errors, Duration)

Focused on services—each service endpoint:

• Rate: Requests per second—understand traffic patterns
• Errors: Failure rate—500s, timeouts, connection errors
• Duration: Latency distribution (P50/P90/P99/P99.9)—user experience

RED is suitable for all online services (HTTP/gRPC workers); each endpoint should have at least 3 metrics.

USE (Utilization, Saturation, Errors)

Focused on resources—CPU, memory, disk, network:

• Utilization: Resource usage (CPU%, memory bytes)
• Saturation: Resource wait queues (CPU run queue, memory OOM score, IO await)
• Errors: Resource errors (disk IO errors, NIC errors, dropped packets)

USE is suitable for all physical/virtual resources—detect issues before they hit usage limits.

Prometheus HA

Prometheus is stateless—scrape on each instance. High availability relies on two identical instances scraping independently:

Grafana queries Thanos → automatically deduplicates the same time series from A and B (using Prometheus's external_labels for distinction).

SLI → SLO → SLA

SLI (Service Level Indicator): The actually measured metric
  Example: GET /api success rate over the past 28 days = 99.95%

SLO (Service Level Objective): The target we commit to
  Example: GET /api success rate >= 99.9%
  
SLA (Service Level Agreement): The consequence of missing the target
  Example: If SLO is not met → 10% credit refund

Error budget = 100% - SLO = 0.1%
  Can be "burned": If you really need to release a quick hotfix (which might temporarily break the SLO), the error budget is your "quota"

Alert Design Principles

1. Page only on user-visible SLO violation: If metrics show P99 latency > 500ms but users can still use the service, do not page. Only page when 4xx/5xx rate > SLO
2. Symptom over cause: Alerts should be "users see 500 errors", not "CPU is at 70%"
3. Eliminate noise: Every alert must have a clear, actionable runbook; otherwise → downgrade to a ticket
4. Tune for sensitivity vs specificity: for: 5m (fire only if condition persists for at least 5 minutes) filters out brief spikes. Longer for duration → later detection, but fewer false positives

Managing False Positives

Common causes of false positives:
  - Fixed thresholds (P99=200ms), but traffic patterns differ between day and night → use percentile over a larger window
  - Alerts trigger during deployments → add maintenance windows or suppress during rollout
  - Too many false positives → alerts are ignored → real incidents are missed → this is called alarm fatigue

Remediation: Regularly review alert fires—which alerts triggered but had no corresponding user impact? → Raise thresholds or convert to tickets.

References

• Google SRE Book: Chapter 6 — Monitoring Distributed Systems
• USE Method: brendangregg.com/usemethod.html
• RED Method: grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services

Keywords: RED, USE, SLI, SLO, SLA, error budget, Prometheus HA, alert design, alarm fatigue