PromQL and Alerting | DevOps Peach

PromQL is a read-only query language for metrics — rate() for counters, aggregations like sum by, histogram_quantile for percentiles; alert rules have for, labels and annotations and should be symptom-based.

Metric type	Behaviour	How to use
Counter (`_total`)	Only goes up (resets on restart)	Always wrap in `rate()` / `increase()`
Gauge	Goes up and down	Use directly: `node_memory_MemAvailable_bytes`
Histogram (`_bucket`)	Counts per bucket	`histogram_quantile(0.95, ...)`

up                                                          # target alive (1) or dead (0)
http_requests_total{method="POST"}                          # filter by label
sum by (handler) (rate(http_requests_total[5m]))            # requests/s per handler
topk(3, rate(http_requests_total[5m]))                      # the 3 highest series
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
100 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100   # % CPU used

[5m] makes a range vector; without it you get an instant vector. rate() = average per-second increase; increase() = total increase over the range.
Operators: + - * / % ^, comparisons == != > < >= <=, set operators and or unless. Aggregations: sum min max avg count topk bottomk quantile stddev… with by (...) / without (...).
PromQL is read-only — it cannot write or delete data.

Alert rule (file alerts.yml, listed under rule_files):

groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels: { severity: page }
        annotations:
          summary: "5xx > 5% for 10 minutes"
          runbook_url: https://wiki.example.com/runbooks/high-error-rate

flowchart LR
  R["Rule: expr true"] --> PE["Pending (waits for: 10m)"] --> FI["Firing"] --> AM["Alertmanager<br/>group, route, silence"] --> N["Slack / email / ticket"]

Principles: alert on symptoms users see (errors, latency) rather than causes (high CPU); avoid alert fatigue (warnings become tickets, page only when action is needed now); give every alert a runbook; practise outages.

Command	Meaning
`promtool check rules alerts.yml`	Validate a rule file
`curl -s localhost:9090/api/v1/query --data-urlencode 'query=up{job="node"}'`	Run PromQL through the API
`curl -X POST localhost:9090/-/reload`	Reload the config (needs `--web.enable-lifecycle`)

Alertmanager

Prometheus evaluates alert rules, but Alertmanager decides who hears about them:

Feature	Purpose
Routing	Send alerts to receivers (email, Slack, Microsoft Teams, PagerDuty, webhooks) by labels such as `severity` or `team`
Grouping	Combine related alerts, for example all instances of one service, into one notification
Inhibition	Mute dependent alerts while a bigger one fires, such as node alerts while the whole datacenter is down
Silences	Mute matching alerts during planned maintenance
Deduplication	Accept the same alert from several Prometheus replicas but notify once

route:
  receiver: team-web
  group_by: [alertname, service]
  routes:
  - matchers: [severity="critical"]
    receiver: pager
receivers:
- name: team-web
  slack_configs: [{ channel: '#web-alerts', api_url: 'https://hooks.slack.example.com/services/T000/B000/XXXX' }]
- name: pager
  webhook_configs: [{ url: 'https://oncall.example.com/hook' }]

More PromQL you will need

Expression	Use
`absent(up{job="api"})`	Alert when a job disappears completely
`predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0`	Disk predicted to fill within 4 hours
`rate(http_requests_total[5m] offset 1d)`	Compare with the same time yesterday
`sum without (instance) (...)`	Aggregate while dropping a label
`label_replace(...)`	Rename or build labels for joining

Always use rate() over a range at least four times the scrape interval, so a missed scrape does not create gaps.

Writing better alert rules

Use for: 5m so short spikes do not page anyone.
Add labels (severity, team) for routing and annotations (summary, description, runbook_url) that explain what to do.
Test rules with promtool test rules and sample series before deploying them.
Prefer SLO-based burn-rate alerts for user-facing services: they fire when the error budget is being spent too fast, catching both sudden outages and slow degradations.