Four Golden Signals

SRE · Observability

Studied

Problem

A service dashboard is a hodgepodge: CPU, memory, log counts, random business metrics. During an incident it's unclear where to look first. When reviewing dashboards across teams, each has its own philosophy and there are no shared KPIs.

Solution

The Google SRE Book: "The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four."

Signal	SRE Book definition	Example for an HTTP service
Latency	The time it takes to service a request — track successful and failed requests separately	p50 / p95 / p99 response time
Traffic	How much demand is placed on the system, in a high-level system-specific metric	RPS, queue messages per second
Errors	The rate of requests that fail — explicitly, implicitly or by policy	Error rate = 5xx / total
Saturation	How "full" the service is, with emphasis on the most constrained resources	CPU %, worker queue, heap utilization

Rule of thumb: if these four graphs look normal, users aren't suffering. If one turns "red", that's where to look.

Example

Prometheus + Grafana for an HTTP service with http_requests_total and http_request_duration_seconds:

# Latency — p95 over successful requests
histogram_quantile(0.95,
  sum by (le) (rate(http_request_duration_seconds_bucket{status!~"5.."}[5m]))
)

# Traffic
sum(rate(http_requests_total[5m]))

# Errors — share of 5xx
sum(rate(http_requests_total{status=~"5.."}[5m]))
  /
sum(rate(http_requests_total[5m]))

# Saturation — pod CPU utilization
sum by (pod) (rate(container_cpu_usage_seconds_total[5m]))
  /
sum by (pod) (kube_pod_container_resource_limits{resource="cpu"})

Bonus — recording rules, so heavy queries aren't recomputed on every dashboard refresh:

# prometheus/rules/golden.yml
groups:
  - name: golden-signals
    interval: 30s
    rules:
      - record: job:http_request_duration_seconds:p95_5m
        expr: histogram_quantile(0.95,
                sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
      - record: job:http_errors:ratio_5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
                /
              sum by (job) (rate(http_requests_total[5m]))

When to use

• Always, as soon as a service has a user (internal or external).
• Before any "pretty" business dashboards. Golden signals first, revenue/funnel later.

Source

• Google SRE Book — Monitoring Distributed Systems
• dastergon/awesome-sre — the Monitoring & Observability section
• binhnguyennus/awesome-scalability — write-ups of real dashboards

How to apply it yourself

1. For every user-facing service — a dedicated Grafana dashboard page with 4 panels.
2. Move the Prometheus queries into recording rules.
3. Alert only on the two signals that are worth waking a human for: errors > X% and latency p95 > Y ms.
4. Once the on-call engineer localizes a real incident in 1–2 minutes using the golden signals — status Studied.