Alerting Policies and Notification Channels

An alerting policy = one or more conditions + notification channels + documentation. When a condition is met, Cloud Monitoring opens an incident and notifies people.

flowchart LR
  M["Metric: Cloud Run 5xx ratio"] --> C["Condition: > 2% for 5 min"]
  C --> I["Incident opened"]
  I --> N1["Slack #shop-oncall"]
  I --> N2["PagerDuty"]
  I --> D["Runbook link in documentation"]
  C -- "back to normal" --> R["Incident closed"]
  %% step M: Pick the signal users feel, such as error ratio or latency, not only CPU.
  %% step C: The condition needs the value above the threshold for a duration, which filters out short spikes.
  %% step I: When the condition holds, an incident opens and stays open until it recovers or someone closes it.
  %% step N1: Notification channels deliver the alert, here to the on-call Slack channel.
  %% step N2: Critical alerts page the on-call engineer through PagerDuty.
  %% step D: The documentation field explains what the alert means and links to the runbook.
  %% step R: When the metric returns below the threshold, the incident closes automatically.

Condition types

Use aggregation (mean, sum, percentile) and group-by carefully: “any VM above 90%” and “average of all VMs above 90%” are very different alerts.

Notification channels

Email, SMS, Google Cloud mobile app, Slack, PagerDuty, webhooks, and Pub/Sub (to automate a response, for example restart a job). Use snooze during planned maintenance.

Good alerting practice

• Alert on symptoms users notice (errors, latency, availability), and keep cause metrics (CPU) on dashboards.
• Every alert must be actionable: if nobody needs to do anything, it is noise.
• Add severity, documentation and a runbook link.
• Tune durations to avoid flapping; review noisy alerts every sprint.
• Use separate channels for critical (page) and warning (ticket/Slack).

# cpu-policy.yaml
displayName: "VM CPU > 90% for 10 min"
combiner: OR
conditions:
- displayName: CPU high
  conditionThreshold:
    filter: metric.type="compute.googleapis.com/instance/cpu/utilization" AND resource.type="gce_instance"
    comparison: COMPARISON_GT
    thresholdValue: 0.9
    duration: 600s
    aggregations: [{alignmentPeriod: 60s, perSeriesAligner: ALIGN_MEAN}]
notificationChannels: ["projects/dl-shop-prod/notificationChannels/1234567890"]
documentation: {content: "Runbook: https://wiki.example.com/runbooks/vm-cpu"}

gcloud monitoring policies create --policy-from-file=cpu-policy.yaml

Traps: a threshold of 0.9 for CPU utilization means 90% (the metric is a ratio); an alert with no notification channel only shows in the console; alerts on every brief spike train people to ignore pages.