Prometheus — Architecture and Data Model

Prometheus is a monitoring system plus time-series database that scrapes metrics from exporters on a schedule; each time series is a metric name plus labels; alerts go out through Alertmanager.

Prometheus (started at SoundCloud, inspired by Google's Borgmon, now a CNCF project) = monitoring system + time-series database: instrumentation, metric collection and storage, querying, alerting and graphing — built for dynamic cloud systems.

flowchart LR
  subgraph Targets
    NE["node_exporter :9100"]
    APP["App /metrics"]
    PG["Pushgateway<br/>(short jobs)"]
  end
  PS["Prometheus server<br/>scrape + TSDB"] -- "pull every 15s" --> NE
  PS -- "pull" --> APP
  PS -- "pull" --> PG
  PS -- "alerts" --> AM["Alertmanager<br/>email, Slack"]
  G["Grafana"] -- "PromQL" --> PS

Pull-based: the server scrapes each target's /metrics endpoint. Systems that do not expose metrics get an exporter (node_exporter for hosts, mysqld_exporter…). Short-lived jobs push metrics to the Pushgateway.
Data model: a time series = metric name + labels: http_requests_total{method="POST", handler="/login"}. Every distinct label combination is its own series.
It does not do: raw logs/events (use Loki, ELK), tracing (use OpenTelemetry/Jaeger), durable long-term storage, horizontal scaling (use Thanos/Mimir), user management.

# prometheus.yml
global:
  scrape_interval: 15s
rule_files:
  - alerts.yml
scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['localhost:9100']

The UI is on :9090; Status → Targets shows whether targets are UP or DOWN; promtool (shipped with Prometheus) validates config files.

Traps: thinking Prometheus is push-based or stores logs; creating high-cardinality labels (user_id, request_id) that explode the series count.

Command	Meaning
`docker run -d --name prometheus -p 9090:9090 -v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml" prom/prometheus`	Run Prometheus
`promtool check config prometheus.yml`	Validate the config
`curl -s localhost:9090/-/healthy`	Health check

Finding targets automatically

Static target lists do not work when containers come and go. Prometheus service discovery builds the target list from a platform's API:

Discovery	Finds
`kubernetes_sd_configs`	Pods, Services, endpoints, nodes in a cluster
`ec2_sd_configs`, `azure_sd_configs`	Cloud VMs, filtered by tags
`consul_sd_configs`	Services registered in Consul
`file_sd_configs`	Targets from JSON or YAML files that another tool writes

Relabelling (relabel_configs) then filters targets and turns metadata, such as a pod's labels, into Prometheus labels.

Instrumenting your own application

Client libraries for Go, Java, Python, Node.js, and .NET expose a /metrics endpoint from your code with counters, gauges, and histograms. Many frameworks have ready middleware for HTTP metrics. OpenTelemetry SDKs can also export to Prometheus.

Storage and retention

Prometheus stores data in a local time-series database (TSDB) on disk, in two-hour blocks that are compacted over time. Retention defaults to 15 days and is set with --storage.tsdb.retention.time=30d or by size. For longer history, high availability, or many clusters, remote write sends samples to systems such as Thanos, Grafana Mimir, or a managed service (Amazon Managed Service for Prometheus, Azure Monitor managed Prometheus).

Prometheus on Kubernetes

The kube-prometheus-stack Helm chart installs Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and default dashboards and alerts. It uses the Prometheus Operator, where a ServiceMonitor resource next to your app tells Prometheus what to scrape:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: web, labels: { release: monitoring } }
spec:
  selector: { matchLabels: { app: web } }
  endpoints: [{ port: http, path: /metrics, interval: 30s }]

Recording rules

A recording rule precomputes an expensive query on a schedule and stores the result as a new series, such as job:http_requests:rate5m, so dashboards and alerts load fast.