Anomaly Management | DevOps Peach

Cost anomalies are unexpected spend changes; managing them means detecting early with ML or threshold alerts, routing alerts to the owning team, investigating the root cause and closing the loop.

A forgotten load test, a logging loop or a leaked access key can burn a month's budget in a weekend. Anomaly management finds unexpected spend fast and gets it to someone who can act.

flowchart LR
  DET["Detect: ML monitor or threshold"] --> AL["Alert the owner"]
  AL --> INV["Investigate: service, account, usage type"]
  INV --> DEC{"Expected?"}
  DEC -->|yes| DOC["Document and adjust forecast"]
  DEC -->|no| FIX["Fix: stop, roll back, rotate keys"]
  FIX --> LEARN["Prevent: policy, budget, guardrail"]
  %% step DET: Cost anomaly detection compares spend with the expected pattern per service, account or tag.
  %% step AL: Alerts go to the owning team through email, chat or tickets — not only to the FinOps team.
  %% step INV: Drill down by service, account, region and usage type to find the root cause.
  %% step DEC: Many anomalies are expected: a launch, a migration or a planned test.
  %% step FIX: Unexpected ones are stopped quickly; a security incident follows the incident process.
  %% step LEARN: Close the loop with a guardrail so the same anomaly cannot repeat.

Detection options

Method	Good for	Weakness
ML-based (AWS Cost Anomaly Detection, Azure anomaly alerts, GCP anomaly detection)	Catching unusual patterns without tuning	Learns over time; can miss slow growth
Budget alerts on actual or forecast	Hard limits per team or account	Late for daily spikes; monthly view
Custom thresholds on daily cost per service	Specific known risks	Needs maintenance

AWS Cost Anomaly Detection uses monitors (what to watch: all services, a linked account, a cost category or a tag) and subscriptions (who to alert, how often, above which impact threshold).

Triage questions

What changed — service, account, region, usage type?
When did it start, and does it line up with a deployment?
Who owns it (tag, account mapping)?
Is it still running? What is the cost per day if we do nothing?

Make it routine

Route each monitor to the owning team's channel.
Track anomalies like incidents: time to detect, time to resolve, cost impact.
Feed expected changes (launches, migrations) into the forecast so they do not alert.

Traps: one central inbox that nobody reads at weekends; alert thresholds so low that people mute the channel; and treating a sudden spike in an unused region as a cost issue when it may be compromised credentials.