DevOps Peach"Hope is not a strategy" — without monitoring you only learn the system is broken when customers complain. Monitoring is for: - alerting on incidents and dashboards of the current state; - long-term trend analysis (how long until the disk fills?), comparing before/after a release; - retrospective analysis during incident investigation: what is broken and why.
| Layer | Metrics |
|---|---|
| Host | CPU, memory, disk I/O, network, filesystem |
| Container | CPU, memory, I/O, network, restart count, CPU throttling |
| Application | Throughput (requests/s), latency, error rate |
Four Golden Signals (Google SRE) for every user-facing service:
flowchart LR U["User request"] --> S["Service"] S --> L["Latency<br/>how long (p95, p99)"] S --> T["Traffic<br/>how many requests/s"] S --> E["Errors<br/>share of failed requests"] S --> SA["Saturation<br/>how full are resources"]
| Signal | Question | Example metric |
|---|---|---|
| Latency | How long does a request take? | p95 response time |
| Traffic | How much load arrives? | HTTP requests/s |
| Errors | How many requests fail? | % HTTP 5xx |
| Saturation | How "full" are resources? | CPU, memory, queues, disk |
- Track latency of failed requests separately — fast errors can pull the average down and hide problems.
- Use percentiles (p95, p99) rather than averages.
Traps: confusing Saturation (resources running out) with Traffic (request volume); monitoring hosts only and ignoring the app metrics users actually feel.
node_exporter exposes host metrics on port 9100 at /metrics.
| Command | Meaning |
|---|---|
docker run -d --name node-exporter -p 9100:9100 prom/node-exporter |
Run node_exporter |
curl -s localhost:9100/metrics \| head |
Show a few metric lines |
Monitoring and observability
Monitoring watches known signals and alerts when they cross thresholds: it answers questions you thought of in advance. Observability is the ability to understand new problems from the data a system emits, built on three pillars:
| Pillar | What it is | Typical tools |
|---|---|---|
| Metrics | Numbers over time, cheap to store and query | Prometheus, Azure Monitor, CloudWatch |
| Logs | Timestamped event records with detail | Loki, ELK/OpenSearch, cloud log services |
| Traces | The path of one request across services, with timing per step | OpenTelemetry, Jaeger, Tempo |
OpenTelemetry is the vendor-neutral standard for producing all three from applications.
Other useful methods
- USE for resources: Utilisation, Saturation, Errors (for CPU, disks, network interfaces).
- RED for services: Rate, Errors, Duration (a simpler form of the golden signals).
SLI, SLO, and SLA
| Term | Meaning | Example |
|---|---|---|
| SLI (indicator) | A measured value of service quality | % of requests served in under 300 ms |
| SLO (objective) | The internal target for the SLI | 99.5% over 30 days |
| SLA (agreement) | A promise to customers with penalties | 99.9% availability or service credits |
The gap between 100% and the SLO is the error budget: while budget remains, teams may ship faster; when it is spent, reliability work comes first.
Black-box and white-box
White-box monitoring uses internal metrics exposed by the application. Black-box monitoring tests from outside like a user, for example the Prometheus blackbox_exporter probing HTTP, TCP, DNS, or ICMP endpoints, or synthetic checks from several regions. You need both: internal metrics explain why, external checks prove whether users are affected.
On-call basics
Alerts that page someone should be urgent, actionable, and rare. Rotate on-call duty, document runbooks, and review every page afterwards: was it needed, was the runbook enough, can it be automated?