DevOps Peach

AWS Basics · Monitoring – CloudWatch, CloudTrail, and AWS Config · Lesson 2 of 5

CloudWatch Logs and Agent

Collect logs and OS metrics with the unified agent.

Practice interactively →

Groups, streams, and retention

A log group contains logs for a workload and owns retention, KMS encryption and access settings. A log stream commonly represents a source such as an instance, container, or file. Confusing a stream with a group leads to queries and retention at the wrong scope.

flowchart LR
  E["EC2 + CloudWatch Agent<br/>role CloudWatchAgentServerPolicy"] --> G["Log group<br/>/myapp/prod, 30 days"]
  G --> S1["Stream<br/>i-0123/app.log"]
  G --> S2["Stream<br/>i-0456/app.log"]
  G --> MF["Metric filter<br/>ERROR count"]
  MF --> AL["Alarm<br/>errors > 10 in 5 min"]
  G --> LI["Logs Insights<br/>query"]
  %% step E: The agent tails files on the instance and needs an IAM role allowed to write logs.
  %% step G: The group is the unit of retention, encryption and permissions.
  %% step S1: Each instance or file writes its own stream inside the group.
  %% step S2: A new instance simply adds a new stream.
  %% step MF: A metric filter counts matching lines and turns them into a metric.
  %% step AL: An alarm on that metric alerts the team when errors spike.
  %% step LI: Logs Insights searches and aggregates across all streams in one or more groups.

The CloudWatch Agent

EC2 does not automatically send application or OS logs. The unified CloudWatch Agent needs configuration and an IAM role with suitable permissions (the managed policy CloudWatchAgentServerPolicy); a running agent without permissions still cannot publish. Install it with Systems Manager or the package manager, generate a config with the wizard, and store the configuration in SSM Parameter Store so every instance in a fleet uses the same file.

The agent collects logs plus memory, disk, swap, process, and network metrics (namespace CWAgent). Lambda, ECS with the awslogs driver, API Gateway, VPC Flow Logs, RDS and many other services can send logs to CloudWatch without an agent.

Retention and cost

By default a log group never expires data. Set retention rather than keeping high-volume logs forever, for example 30 days for app logs and longer for audit logs, and export or archive to S3 when needed. Use the Infrequent Access log class for logs you rarely query.

Querying and routing

Logs Insights queries data in one or more groups with a pipe-based language:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) by bin(5m)

A metric filter turns matching events into a metric (for alarms and dashboards). A subscription filter streams data in near real time to Lambda, Kinesis Data Streams, Firehose or OpenSearch, while S3 export is batch-oriented and can take hours. Live Tail shows new events as they arrive.

Structured logs

Write logs as JSON ({"level":"error","route":"/pay","ms":812}) so Logs Insights can filter by field without regex. Never log secrets, tokens, or full card numbers; data protection policies can detect and mask sensitive data in a log group.

Command Purpose
aws logs create-log-group --log-group-name /myapp/prod Create a group
aws logs put-retention-policy --log-group-name /myapp/prod --retention-in-days 30 Set retention
aws logs tail /myapp/prod --follow --filter-pattern ERROR Follow logs
aws logs put-metric-filter --log-group-name /myapp/prod --filter-name errors --filter-pattern ERROR --metric-transformations metricName=AppErrors,metricNamespace=MyApp,metricValue=1 Count errors as a metric

Traps: an agent with no IAM role; leaving retention at "never expire"; setting retention on a stream instead of the group.

Practice interactively →