Grafana — Dashboards and Alerts

Grafana is an open-source dashboard tool that stores no metrics but queries data sources such as Prometheus; panels use PromQL, with ready-made dashboards, alert rules and contact points.

Grafana builds dashboards from many data sources (Prometheus, Loki, Elasticsearch, MySQL, clouds…). Grafana stores no metrics: on every refresh it queries the data source, redraws panels and evaluates alert rules.

flowchart LR
  NE["node_exporter"] --> P[("Prometheus<br/>stores metrics")]
  APP["App /metrics"] --> P
  G["Grafana"] -- "PromQL on each refresh" --> P
  G --> D["Dashboard<br/>CPU, RAM, latency panels"]
  G -- "alert rule" --> CP["Contact point<br/>email, Slack, Discord"]

Step	Details
Run	`docker run -d --name grafana -p 3000:3000 grafana/grafana`, open `http://localhost:3000` (first login `admin`/`admin`, change it at once)
Data source	Connections → Data sources → Prometheus; the URL is Prometheus' address as seen from Grafana
Dashboard	Build panels with PromQL, or Import a ready-made dashboard by ID (for example Node Exporter Full)
Alerts	Alerting → Alert rules (query + threshold + duration), Contact points (email, Slack…), Notification policies
Access	Users, teams, organizations (multi-tenancy), dashboard folders with permissions

When Grafana runs in another container, localhost:9090 points at the Grafana container itself; use the service/container name on a shared Docker network (for example http://prometheus:9090).
Keep dashboards as JSON in Git (or provisioning) so they survive container removal; mount a volume for /var/lib/grafana.
A good service dashboard puts the Four Golden Signals on top and resource details below.

Traps: thinking Grafana stores metrics; using localhost:9090 as the data source when Grafana runs in a container; keeping the default admin password.

Command	Meaning
`docker run -d --name grafana -p 3000:3000 grafana/grafana`	Run Grafana
`curl -s localhost:3000/api/health`	Check Grafana

Making dashboards useful

Variables (Dashboard settings → Variables) add drop-downs such as $instance or $namespace, filled by a query like label_values(up, instance), so one dashboard serves every server or service.
Panel types: time series for trends, stat and gauge for current values, bar gauge for comparisons, table for lists, heatmap for latency distributions.
Units and thresholds: set the unit (bytes, seconds, percent) and colour thresholds so a glance shows what is normal.
Annotations mark deployments or incidents on graphs, often pushed automatically by the CI/CD pipeline through the Grafana API, making cause and effect visible.
Link panels to more detailed dashboards (drill-down) instead of putting everything on one screen.

Dashboards as code

Provisioning loads data sources and dashboards from files at startup:

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  url: http://prometheus:9090
  isDefault: true

Dashboard JSON files placed in a provisioned folder appear automatically, so a new Grafana instance starts fully configured. Tools such as Terraform's Grafana provider or Grafonnet generate dashboards programmatically.

The rest of the LGTM stack

Grafana Labs builds a full open-source observability stack around Grafana: Loki for logs, Tempo for traces, Mimir for scalable long-term metrics, and Alloy as a collector. Grafana can correlate them: jump from a latency spike to the matching logs and traces for the same time and service.

Running it in production

Use an external database (PostgreSQL or MySQL) instead of the default SQLite when running several Grafana replicas, configure single sign-on (OAuth or SAML) rather than local accounts, and put it behind HTTPS. Managed options include Grafana Cloud, Azure Managed Grafana, and Amazon Managed Grafana.

Dashboard design tips

Start with the question the dashboard answers ("is checkout healthy?"), keep the most important panels at the top left, use consistent colours across dashboards, and delete dashboards nobody opens.