Observability on a Budget: What to Keep When You Cannot Keep Everything

Every team hits the moment where the observability bill exceeds the compute bill. The instinct is to cut evenly across the board, which is exactly wrong: it degrades the signals you need during incidents while preserving the ones you never look at.

Cut in order of value, not of volume.

Keep first: the cheap, high-signal stuff

• Metrics are aggregates. They are tiny, permanent and answer "is it broken and since when?". Never cut these.
• Synthetic checks from a few regions cost almost nothing and catch outages your users would otherwise report.
• Error logs, sampled at 100%. Errors are rare by definition; keeping all of them is cheap and priceless.

Sample aggressively: the verbose stuff

• Success logs at info level. This is usually the single largest line item. Sampling 1–10% of happy-path logs loses almost nothing diagnostically.
• Traces. Head-based sampling at 1–10% is fine for most teams. If you need the rare slow request, use tail-based sampling to keep traces that crossed a latency or error threshold and drop the boring ones.
• Debug logs. Off by default, toggleable per component at runtime. Shipping them always is burning money on data nobody reads.

A concrete ordering

Make retention a decision, not a default

Thirty days of traces is rarely worth thirty times the cost of three. Metrics can live a year at low resolution; high-cardinality traces cannot. Set retention per signal, and revisit it when the bill moves.

The goal is not less telemetry. It is telemetry whose cost you chose.

Tools like Prometheus for metrics and a sampled pipeline for traces get most teams to a bill they can defend. The discipline — keep the aggregates, sample the floods — matters more than the vendor.