The Four Golden Signals, and the Two Everyone Forgets

The four golden signals are famous enough to be on a poster: latency, traffic, errors, saturation. Most dashboards cover the first three well and the fourth badly, and the fourth is where incidents live.

The canonical four, briefly

• Latency — how long requests take, measured at the percentiles users feel (p95, p99), not the mean.
• Traffic — how much demand the system is under; your denominator for everything else.
• Errors — the rate of failed requests, including the ones that fail slowly or silently.
• Saturation — how full the constrained resources are: CPU, memory, disk, connection pools, queue depth.

The mean is a lie you tell yourself. A p50 of 40 ms with a p99 of 4 s means one user in a hundred is having a terrible time, and your "healthy" dashboard is green.

The two everyone forgets

Saturation of the hidden resources. Everyone graphs CPU. Far fewer graph database connection pool utilisation, goroutine counts, or the length of the retry queue. Those saturate first and loudest, and they are the actual cause of "mystery" latency cliffs.

Change events overlaid on the signals. A signal tells you that something changed; an annotation tells you what you did. Overlaying deploys, config pushes and scaling events on the same timeline turns a guessing game into a five-second diagnosis. Grafana annotations exist precisely for this and are chronically underused.

From signals to a number you can argue about

Signals become useful when they collapse into a service-level objective someone owns:

# 99% of requests under 300ms, over a 30 day window
slo:
  name: api-latency
  target: 0.99
  window: 30d
  indicator:
    good: latency < 300ms
    total: all requests

An SLO converts four graphs into one budget. "We have spent 80% of this month's error budget" is a sentence that changes a roadmap. "Latency looks a bit high" is not.

Monitor the signals, but make a decision out of them. A dashboard nobody acts on is decoration.

Start with the four, add the two you forgot, and put a budget on top. That is most of observability, before anyone buys a vendor.