Building a Monitoring Stack with Prometheus and Grafana

Monitoring is not about collecting every metric you can — it is about collecting the right metrics, alerting on what matters, and giving engineers the dashboards they need to diagnose problems quickly.

The Four Golden Signals

Google SRE defines four golden signals: latency, traffic, errors, and saturation. Every service should have dashboards for these four. They give you an immediate sense of whether a service is healthy and where the problem is when it is not.

Recording Rules

Pre-compute frequently used queries with Prometheus recording rules. This improves dashboard performance and reduces load on Prometheus. Define recording rules for your key SLIs and use them in dashboards and alerts.

SLO-Based Alerting

Alert on error budget burn rate, not on raw thresholds. This reduces alert fatigue and focuses attention on issues that actually impact users. Define SLOs with stakeholders, calculate error budgets, and alert when the budget is being consumed too quickly.

Dashboards That Help

A good dashboard answers a question. Build dashboards for specific scenarios: a service health dashboard for on-call, a capacity planning dashboard for scaling decisions, and an executive dashboard for high-level system health. Avoid dashboards that just show every metric on one screen — they look impressive but are not useful during an incident.