DevOps Peach

GitOps with Argo CD · Argo CD Security and Operations · Lesson 3 of 4

Notifications and monitoring

Argo CD Notifications sends messages on sync and health events, and Prometheus metrics such as argocd_app_info feed dashboards and alerts.

Practice interactively →

Notifications

The notifications controller (part of a standard install) watches Applications and sends messages when a trigger fires. Everything is configured in argocd-notifications-cm and secrets in argocd-notifications-secret.

apiVersion: v1
kind: ConfigMap
metadata: { name: argocd-notifications-cm, namespace: argocd }
data:
  service.slack: |
    token: $slack-token                      # key in argocd-notifications-secret
  template.app-sync-failed: |
    message: |
      :x: {{.app.metadata.name}} failed to sync to {{.app.status.operationState.syncResult.revision}}
      {{.app.status.operationState.message}}
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [app-health-degraded]

Teams subscribe with an annotation on their Application (or AppProject):

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: shop-alerts
    notifications.argoproj.io/subscribe.on-health-degraded.slack: shop-alerts

The catalog of ready-made triggers and templates (on-deployed, on-sync-succeeded, on-sync-status-unknown…) can be installed from the Argo CD docs. Services include Slack, Microsoft Teams, e-mail, webhooks, GitHub commit statuses and more. A GitHub commit status on the config repo shows reviewers whether a merged change actually deployed.

Metrics

Each component exposes Prometheus metrics:

Component Port Useful metrics
application-controller 8082 argocd_app_info{sync_status,health_status}, argocd_app_sync_total{phase}, argocd_app_reconcile
repo-server 8084 argocd_git_request_total, argocd_git_request_duration_seconds
argocd-server 8083 API request counts and latency 
# apps that are not Synced
count by (name, project) (argocd_app_info{sync_status!="Synced"})
# apps that are Degraded
argocd_app_info{health_status="Degraded"} == 1
# failed syncs per hour
sum(increase(argocd_app_sync_total{phase=~"Error|Failed"}[1h]))

With the Prometheus Operator, enable the ServiceMonitors in the Helm chart (controller.metrics.serviceMonitor.enabled=true, and the same for repo server and server). A community Grafana dashboard for Argo CD shows sync status, reconcile time and Git request errors.

Alerts worth having

Alert Why
App OutOfSync for more than 30 minutes A change is stuck, or someone is fighting self-heal
App Degraded for more than 10 minutes The deployed version is broken
Sync failures increasing Admission webhooks, quotas or bad manifests
Git request errors Expired credentials or the Git host is down

Keep alerts actionable: OutOfSync for a few minutes is normal during a deploy; use for: and route warnings to a chat channel, real outages to on-call.

Practice interactively →