AWS Monitoring Fundamentals

Level: Beginner–Intermediate

Table of Contents

1. Module 1: Understanding CloudWatch
   • What is CloudWatch?
   • CloudWatch Metrics
   • Metric Retention and Resolution
   • Common AWS Namespaces
   • CloudWatch Agent
   • CloudWatch Logs
   • CloudWatch Logs Insights
2. Module 2: Alerting in CloudWatch
   • CloudWatch Alarms – Concepts
   • Composite Alarms
   • Complete Alert Flow
   • Demo: Create Your First CloudWatch Alarm
3. Module 3: Advanced Observability
   • CloudWatch Container Insights
   • CloudWatch Application Insights
   • AWS X-Ray – Distributed Tracing
   • Amazon EventBridge Rules
4. Practical Code Examples
   • AWS CLI
   • Python boto3
   • CloudFormation YAML
5. Reference Architectures
6. Summary and Key Takeaways
7. Additional Resources

Module 1: Understanding CloudWatch

What is CloudWatch?

Telemetry monitoring is an automated process of collecting data from a remote source into a centralized location. In the AWS context, this refers primarily to Amazon CloudWatch.

Data is collected at predefined time intervals and is used to:

• Identify system performance
• Detect and resolve problems (troubleshooting)
• Optimize workloads
• Feed proactive alerts to operational teams

Services That Send Data to CloudWatch

CloudWatch Ecosystem Overview

flowchart TD
    subgraph Sources["Data Sources"]
        EC2["EC2 Instances"]
        RDS["RDS / Aurora"]
        Lambda["Lambda Functions"]
        ELB["Elastic Load Balancer"]
        ECS["ECS / EKS Containers"]
        OnPrem["On-Premises Servers\n(via CloudWatch Agent)"]
    end

    subgraph CW["Amazon CloudWatch"]
        Metrics["CloudWatch Metrics\n(namespaces, dimensions)"]
        Logs["CloudWatch Logs\n(log groups / streams)"]
        Insights["Logs Insights\n(queries)"]
        Alarms["CloudWatch Alarms\n(threshold / composite)"]
        Dashboard["Dashboards"]
        ContainerI["Container Insights"]
        AppI["Application Insights"]
    end

    subgraph Actions["Actions and Notifications"]
        SNS["SNS Topic\n(email, SMS, HTTP)"]
        AutoScale["Auto Scaling Policy"]
        LambdaAction["Lambda Function"]
        SSM["Systems Manager\nAutomation"]
        EB["EventBridge"]
    end

    Sources --> Metrics
    Sources --> Logs
    Logs --> Insights
    Metrics --> Alarms
    Metrics --> Dashboard
    Alarms --> SNS
    Alarms --> AutoScale
    Alarms --> LambdaAction
    EB --> SSM
    CW --> EB

CloudWatch Metrics

Metrics are time-series data points published to CloudWatch. They form the foundation of all AWS monitoring.

Metric Hierarchy

Namespace  (e.g.: AWS/EC2)
  └── Dimension(s)  (e.g.: InstanceId = i-0abc123)
        └── Metric Name  (e.g.: CPUUtilization)
              └── Data Points  (value + timestamp + unit)

Core Concepts

EC2 Monitoring Types

Detailed Monitoring use cases: production workloads, reactive auto-scaling, troubleshooting requiring high granularity.

Metric Retention and Resolution

CloudWatch automatically aggregates data points over time to optimize storage:

Metrics automatically expire after 15 months without new data.

Common AWS Namespaces

Key Metrics for EC2

Important: CloudWatch does not collect RAM and OS-level disk usage metrics by default. You must install the CloudWatch Agent.

CloudWatch Agent

The CloudWatch Agent is a downloadable package that enables collection of custom metrics and logs from:

• EC2 instances (Linux and Windows)
• Containers (ECS, EKS)
• On-premises servers

Additional Metrics Available via the Agent

Installation Methods

Option 1: Manual Installation via CLI (Amazon Linux 2)

# Install the package
sudo yum install amazon-cloudwatch-agent -y

# Launch the interactive configuration wizard
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

# Start the agent with the generated configuration file
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
    -a fetch-config \
    -m ec2 \
    -s \
    -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json

Option 2: Installation via AWS Systems Manager (recommended for production)

flowchart LR
    Admin["Administrator"] --> |"1. Configure agent"| SSM_PS["SSM Parameter Store\n(AmazonCloudWatch-Config)"]
    SSM_PS --> |"2. Distributes via SSM Run Command"| EC2A["EC2 Instance A"]
    SSM_PS --> |"2. Distributes via SSM Run Command"| EC2B["EC2 Instance B"]
    SSM_PS --> |"2. Distributes via SSM Run Command"| OnPrem["On-Premises Server\n(SSM Agent required)"]
    EC2A --> |"3. Sends metrics and logs"| CW["CloudWatch"]
    EC2B --> |"3. Sends metrics and logs"| CW
    OnPrem --> |"3. Sends metrics and logs"| CW

AWS Systems Manager advantage: Single configuration stored in Parameter Store, automatically deployed to all instances — major time-saver in environments with hundreds of instances.

CloudWatch Agent Configuration Example (JSON)

{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
  },
  "metrics": {
    "namespace": "CWAgent",
    "metrics_collected": {
      "mem": {
        "measurement": ["mem_used_percent"],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": ["disk_used_percent"],
        "resources": ["/", "/data"],
        "metrics_collection_interval": 60
      },
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
        "metrics_collection_interval": 60,
        "totalcpu": true
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/messages",
            "log_group_name": "/ec2/system-logs",
            "log_stream_name": "{instance_id}/messages",
            "retention_in_days": 30
          },
          {
            "file_path": "/var/log/nginx/access.log",
            "log_group_name": "/ec2/nginx/access",
            "log_stream_name": "{instance_id}",
            "retention_in_days": 7
          }
        ]
      }
    }
  }
}

CloudWatch Logs

CloudWatch Logs enables you to centralize all log files from your cloud resources in a single location. It provides search, querying, and alerting capabilities on log data.

Log Structure in CloudWatch

Log Group  (e.g.: /aws/lambda/order-processor)
  └── Log Stream  (log stream from a specific resource)
        └── Log Events  (individual log entries with timestamp + message)

Log Classes

Key Features of the Standard Class

Log Types

1. Vended Logs

• Predefined logs that can be enabled from AWS services
• No custom configuration needed, but activation may be required

2. Custom Logs

• Managed and configured by the user
• Collected via the CloudWatch Agent for system and application logs
• Examples: nginx logs, Java/Python application logs, Windows Event logs

Retention Policy

By default, logs are retained indefinitely. It is recommended to set a retention policy:

# Set 30-day retention on a log group
aws logs put-retention-policy \
  --log-group-name "/aws/lambda/order-processor" \
  --retention-in-days 30

CloudWatch Logs Insights

CloudWatch Logs Insights is an interactive query tool for analyzing log data in seconds, even on terabytes of data.

Main Commands

Logs Insights Query Examples

Most Frequent Errors (Lambda)

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc
| limit 20

Top 10 5xx Errors on API Gateway

fields @timestamp, status, resourcePath, httpMethod
| filter status >= 500
| stats count(*) as errorCount by resourcePath, httpMethod
| sort errorCount desc
| limit 10

Average Lambda Request Latency

fields @timestamp, @duration, @memorySize, @maxMemoryUsed
| stats avg(@duration) as avgDuration,
        max(@duration) as maxDuration,
        min(@duration) as minDuration,
        pct(@duration, 95) as p95Duration
  by bin(1h)
| sort @timestamp desc

Field Extraction with parse (regex)

fields @timestamp, @message
| parse @message "user=* action=* status=*" as user, action, status
| filter status = "FAILED"
| stats count(*) as failCount by user
| sort failCount desc

VPC Flow Logs Analysis – Top Sources of Rejected Traffic

fields srcAddr, dstAddr, dstPort, action
| filter action = "REJECT"
| stats count(*) as rejectedConnections by srcAddr, dstAddr, dstPort
| sort rejectedConnections desc
| limit 20

Auto-Generated Fields by CloudWatch Logs Insights

Module 2: Alerting in CloudWatch

CloudWatch Alarms – Concepts

A CloudWatch Alarm monitors a single metric over a defined period and triggers actions when the value crosses a configured threshold.

Alarm States

Alarm Configuration Parameters

Missing Data Treatment

Available Actions from an Alarm

flowchart LR
    Alarm["CloudWatch Alarm\n(ALARM state)"] --> SNS["SNS Topic\n(Email, SMS, HTTP, Lambda)"]
    Alarm --> AS["Auto Scaling\n(Scale Out / Scale In)"]
    Alarm --> EC2["EC2 Action\n(Stop, Terminate, Reboot, Recover)"]
    Alarm --> SSM["Systems Manager\nOpsItem"]

Composite Alarms

Composite Alarms allow combining multiple individual alarms with logical operators (AND, OR, NOT). They reduce false positives and enable more precise modeling of alert states.

Composite Logic Example

ALARM("HighCPU") AND ALARM("HighNetworkOut") -> "PotentialDDoS"
ALARM("DatabaseError") OR ALARM("DatabaseTimeout") -> "DatabaseCritical"

flowchart TD
    A1["Alarm: CPUUtilization > 90%\n(state: ALARM)"]
    A2["Alarm: NetworkOut > 1GB\n(state: ALARM)"]
    A3["Alarm: MemoryUsed > 85%\n(state: OK)"]
    A4["Composite Alarm:\nA1 AND A2\n(state: ALARM)"]
    A5["Composite Alarm:\nA4 AND A3\n(state: OK)"]

    A1 --> A4
    A2 --> A4
    A4 --> A5
    A3 --> A5
    A4 --> |"ALARM notifies SNS"| SNS["SNS: PagerDuty / Email"]

Complete Alert Flow

sequenceDiagram
    participant EC2 as EC2 Instance
    participant CW as CloudWatch Metrics
    participant Alarm as CloudWatch Alarm
    participant SNS as SNS Topic
    participant Email as Email / SMS
    participant Lambda as Lambda Function
    participant AS as Auto Scaling

    EC2->>CW: Publishes CPUUtilization = 95%
    CW->>Alarm: Evaluates threshold > 80% over 3 periods
    Alarm->>Alarm: State changes to ALARM
    Alarm->>SNS: Publishes notification
    SNS->>Email: Sends email/SMS to the team
    SNS->>Lambda: Triggers automatic remediation
    Alarm->>AS: Triggers Scale Out policy
    AS->>EC2: Launches new instances

Demo: Create Your First CloudWatch Alarm

This demo walks through step-by-step creation of a CloudWatch alarm via the AWS console, monitoring the CPU of an EC2 instance and sending notifications by email via SNS.

Step 1: Launch a Test EC2 Instance

1. In the AWS console, search for EC2
2. Click Launch Instance
3. Name the instance: web-app-monitoring-test
4. AMI: Amazon Linux 2023
5. Instance type: t3.micro
6. Key pair: Proceed without a key pair
7. Leave other settings as default → Launch Instance

Step 2: Create an SNS Topic

1. Search for SNS in the console → Simple Notification Service
2. Click Topics → Create topic
3. Type: Standard
4. Name: ops-alarm-notifications
5. Display name: ops-alarm-notifications
6. Click Create topic
7. Subscriptions tab → Create subscription
8. Protocol: Email
9. Endpoint: your email address
10. Click Create subscription
11. Confirm the verification email received

Step 3: Create the CloudWatch Alarm

1. Search for CloudWatch in the console
2. Left menu → Alarms → All Alarms → Create alarm
3. Click Select metric
4. Choose EC2 → Per-Instance Metrics
5. Search for web-app-monitoring-test → select CPUUtilization
6. Parameters:
   • Period: 5 minutes
   • Statistic: Average
   • Threshold type: Static
   • Condition: Greater than 70 (%)
   • Datapoints to alarm: 3 out of 3
7. Notification:
   • Alarm state trigger: In Alarm
   • SNS topic: ops-alarm-notifications
8. Name the alarm: HighCPUAlarm-web-app-monitoring-test
9. Click Create alarm

Step 4: Test the Alarm

To generate CPU load on the instance (via SSM Session Manager or EC2 Instance Connect):

# Generate artificial CPU load
stress --cpu 4 --timeout 600

Important: Delete resources after testing (EC2 instance, SNS topic, alarm) to avoid charges.

Module 3: Advanced Observability

CloudWatch Container Insights

CloudWatch Container Insights automatically collects detailed metrics and logs for ECS, EKS, and self-managed Kubernetes workloads.

Collected Metrics

Enabling on ECS

# Enable Container Insights on an ECS cluster
aws ecs update-cluster-settings \
  --cluster MyECSCluster \
  --settings name=containerInsights,value=enabled

Enabling on EKS via Helm

helm repo add aws-observability https://aws-observability.github.io/helm-charts
helm install aws-cloudwatch-metrics aws-observability/aws-cloudwatch-metrics \
  --namespace amazon-cloudwatch \
  --set clusterName=MyEKSCluster

CloudWatch Application Insights

CloudWatch Application Insights automates discovery and monitoring of .NET and SQL Server applications on EC2, automatically creating relevant dashboards and alarms.

Features

• Auto-discovery of application components (IIS, SQL Server, .NET)
• Automatic configuration of metrics, logs, and alarms
• Problem detection with severity levels
• Automatic creation of OpsItems in Systems Manager
• Correlation between metric anomalies and log errors

Supported Applications

AWS X-Ray – Distributed Tracing

AWS X-Ray enables end-to-end tracing of requests through distributed architectures (microservices, serverless), making it easier to troubleshoot latency and error issues.

X-Ray Concepts

X-Ray Architecture

flowchart LR
    Client["HTTP Client"] --> ALB["Application\nLoad Balancer"]
    ALB --> API["API Gateway\n(X-Ray enabled)"]
    API --> Lambda["Lambda Function\n(X-Ray SDK)"]
    Lambda --> DDB["DynamoDB\n(automatic segment)"]
    Lambda --> S3["S3\n(automatic segment)"]
    Lambda --> ExtAPI["External API\n(manual subsegment)"]
    Lambda --> |"Sends traces"| XRay["AWS X-Ray\nService"]
    XRay --> |"Service Map and Traces"| Console["X-Ray Console\nand CloudWatch"]

Enabling X-Ray on Lambda (Python)

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

# Instrument all AWS libraries automatically
patch_all()

def lambda_handler(event, context):
    with xray_recorder.in_subsegment('process-shipment') as subsegment:
        subsegment.put_annotation('shipmentId', event.get('shipmentId'))
        subsegment.put_metadata('payload', event)
        result = process_shipment(event)
        return result

Amazon EventBridge Rules

Amazon EventBridge (formerly CloudWatch Events) enables real-time reaction to events from AWS services, your applications, or partner sources.

EventBridge Concepts

EventBridge Rule Example for EC2

{
  "source": ["aws.ec2"],
  "detail-type": ["EC2 Instance State-change Notification"],
  "detail": {
    "state": ["terminated", "stopped"]
  }
}

EventBridge Flow – Notification and Remediation

flowchart LR
    CW_Alarm["CloudWatch Alarm\n(ALARM state)"] --> |"Event generated"| EB["EventBridge\nDefault Event Bus"]
    EB --> |"Rule: alarm state = ALARM"| Lambda["Lambda\n(remediation)"]
    EB --> |"Rule: alarm state = ALARM"| SNS["SNS Topic\n(notification)"]
    EB --> |"Rule: alarm state = ALARM"| SSM["SSM Automation\n(automatic runbook)"]
    EC2_Terminated["EC2 Instance\nTerminated"] --> |"Event generated"| EB
    EB --> |"Rule: state = terminated"| Lambda2["Lambda\n(cleanup resources)"]

Scheduling Rule Examples

# Trigger a Lambda every 5 minutes
aws events put-rule \
  --name "Every5Minutes" \
  --schedule-expression "rate(5 minutes)" \
  --state ENABLED

# Trigger a Lambda every day at 08:00 UTC (cron)
aws events put-rule \
  --name "DailyAt8AM" \
  --schedule-expression "cron(0 8 * * ? *)" \
  --state ENABLED

Practical Code Examples

AWS CLI

Metrics

# List all available namespaces
aws cloudwatch list-metrics --query "Metrics[].Namespace" --output json

# List metrics for EC2
aws cloudwatch list-metrics --namespace "AWS/EC2" --metric-name CPUUtilization

# Retrieve statistics for a metric
aws cloudwatch get-metric-statistics \
  --namespace "AWS/EC2" \
  --metric-name "CPUUtilization" \
  --dimensions Name=InstanceId,Value=i-0fedcba9876543210 \
  --start-time 2024-01-01T00:00:00Z \
  --end-time 2024-01-01T23:59:59Z \
  --period 3600 \
  --statistics Average Maximum

# Publish a custom metric
aws cloudwatch put-metric-data \
  --namespace "WebOrderService" \
  --metric-name "TransactionsProcessed" \
  --value 42 \
  --unit Count \
  --dimensions Environment=Production,Service=CheckoutAPI

Alarms

# Create an alarm on CPUUtilization
aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPUAlarm" \
  --alarm-description "CPU utilization > 80% for 3 consecutive periods" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --dimensions Name=InstanceId,Value=i-0fedcba9876543210 \
  --period 300 \
  --evaluation-periods 3 \
  --datapoints-to-alarm 3 \
  --threshold 80 \
  --comparison-operator GreaterThanOrEqualToThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:MyAlarmTopic \
  --ok-actions arn:aws:sns:us-east-1:123456789012:MyAlarmTopic

# Create a Composite Alarm
aws cloudwatch put-composite-alarm \
  --alarm-name "CriticalApplicationAlarm" \
  --alarm-rule "ALARM(\"HighCPUAlarm\") AND ALARM(\"HighMemoryAlarm\")" \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:PagerDutyTopic

# List all alarms and their state
aws cloudwatch describe-alarms \
  --alarm-types MetricAlarm CompositeAlarm \
  --query "MetricAlarms[].{Name:AlarmName,State:StateValue}"

# Temporarily disable alarm actions
aws cloudwatch disable-alarm-actions \
  --alarm-names "HighCPUAlarm"

CloudWatch Logs

# Create a log group with tags
aws logs create-log-group \
  --log-group-name "/myapp/production" \
  --tags Environment=production,Team=backend

# Set retention (30 days)
aws logs put-retention-policy \
  --log-group-name "/myapp/production" \
  --retention-in-days 30

# Start a Logs Insights query
QUERY_ID=$(aws logs start-query \
  --log-group-name "/aws/lambda/order-processor" \
  --start-time $(date -d '1 hour ago' +%s) \
  --end-time $(date +%s) \
  --query-string "fields @timestamp, @message | filter @message like /ERROR/ | limit 20" \
  --query "queryId" \
  --output text)

# Retrieve results
aws logs get-query-results --query-id "$QUERY_ID"

# Create a Metric Filter (logs to metrics)
aws logs put-metric-filter \
  --log-group-name "/myapp/production" \
  --filter-name "ErrorCount" \
  --filter-pattern "[timestamp, requestId, level=ERROR, ...]" \
  --metric-transformations \
    metricName=ApplicationErrors,metricNamespace=MyApp,metricValue=1,defaultValue=0

Python boto3

import boto3
from datetime import datetime, timedelta, timezone
import time

cloudwatch = boto3.client('cloudwatch', region_name='us-east-1')
logs_client = boto3.client('logs', region_name='us-east-1')

def publish_custom_metric(namespace: str, metric_name: str, value: float,
                           dimensions: list, unit: str = 'Count') -> None:
    """Publishes a custom metric to CloudWatch."""
    cloudwatch.put_metric_data(
        Namespace=namespace,
        MetricData=[{
            'MetricName': metric_name,
            'Dimensions': dimensions,
            'Value': value,
            'Unit': unit,
            'Timestamp': datetime.now(timezone.utc)
        }]
    )

# Example: publish an error metric
publish_custom_metric(
    namespace='EcommercePortal',
    metric_name='OrderErrors',
    value=1.0,
    dimensions=[
        {'Name': 'Environment', 'Value': 'production'},
        {'Name': 'Region', 'Value': 'us-east-1'}
    ]
)

def create_cpu_alarm(instance_id: str, threshold: float, sns_arn: str) -> None:
    """Creates a CloudWatch alarm on the CPU of an EC2 instance."""
    cloudwatch.put_metric_alarm(
        AlarmName=f'HighCPU-{instance_id}',
        AlarmDescription=f'CPU > {threshold}% for 3 periods of 5 min',
        MetricName='CPUUtilization',
        Namespace='AWS/EC2',
        Statistic='Average',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        Period=300,
        EvaluationPeriods=3,
        DatapointsToAlarm=3,
        Threshold=threshold,
        ComparisonOperator='GreaterThanOrEqualToThreshold',
        TreatMissingData='notBreaching',
        AlarmActions=[sns_arn],
        OKActions=[sns_arn]
    )

def get_metric_stats(instance_id: str, hours: int = 24) -> list:
    """Retrieves CPU statistics for an EC2 instance."""
    end_time = datetime.now(timezone.utc)
    start_time = end_time - timedelta(hours=hours)

    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/EC2',
        MetricName='CPUUtilization',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        StartTime=start_time,
        EndTime=end_time,
        Period=3600,
        Statistics=['Average', 'Maximum']
    )
    return response['Datapoints']

def run_logs_insights_query(log_group: str, query: str, hours: int = 1) -> list:
    """Executes a Logs Insights query and waits for results."""
    end_time = int(datetime.now(timezone.utc).timestamp())
    start_time = int((datetime.now(timezone.utc) - timedelta(hours=hours)).timestamp())

    response = logs_client.start_query(
        logGroupName=log_group,
        startTime=start_time,
        endTime=end_time,
        queryString=query
    )
    query_id = response['queryId']

    while True:
        result = logs_client.get_query_results(queryId=query_id)
        if result['status'] in ('Complete', 'Failed', 'Cancelled'):
            break
        time.sleep(1)

    return result.get('results', [])

# Example: top Lambda errors over the last hour
results = run_logs_insights_query(
    log_group='/aws/lambda/order-processor',
    query="""
        fields @timestamp, @message
        | filter @message like /ERROR/
        | stats count(*) as errorCount by bin(1h)
        | sort errorCount desc
        | limit 10
    """
)

for row in results:
    fields = {f['field']: f['value'] for f in row}
    print(f"Period: {fields.get('bin(1h)')} -> Errors: {fields.get('errorCount')}")

CloudFormation YAML

AWSTemplateFormatVersion: '2010-09-09'
Description: 'Complete CloudWatch monitoring stack'

Parameters:
  InstanceId:
    Type: String
    Description: ID of the EC2 instance to monitor
  AlertEmail:
    Type: String
    Description: Email address for alerts
  EnvironmentName:
    Type: String
    Default: production

Resources:

  AlertSNSTopic:
    Type: AWS::SNS::Topic
    Properties:
      TopicName: !Sub '${EnvironmentName}-monitoring-alerts'
      DisplayName: !Sub '${EnvironmentName} Monitoring Alerts'
      Subscription:
        - Protocol: email
          Endpoint: !Ref AlertEmail

  HighCPUAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub '${EnvironmentName}-HighCPU-${InstanceId}'
      AlarmDescription: 'CPU > 80% for 3 consecutive 5-minute periods'
      Namespace: AWS/EC2
      MetricName: CPUUtilization
      Dimensions:
        - Name: InstanceId
          Value: !Ref InstanceId
      Statistic: Average
      Period: 300
      EvaluationPeriods: 3
      DatapointsToAlarm: 3
      Threshold: 80
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching
      AlarmActions:
        - !Ref AlertSNSTopic
      OKActions:
        - !Ref AlertSNSTopic

  StatusCheckFailedAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmName: !Sub '${EnvironmentName}-StatusCheckFailed-${InstanceId}'
      AlarmDescription: 'Status check failed - instance potentially compromised'
      Namespace: AWS/EC2
      MetricName: StatusCheckFailed
      Dimensions:
        - Name: InstanceId
          Value: !Ref InstanceId
      Statistic: Maximum
      Period: 60
      EvaluationPeriods: 2
      DatapointsToAlarm: 2
      Threshold: 1
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: breaching
      AlarmActions:
        - !Ref AlertSNSTopic
        - !Sub 'arn:aws:automate:${AWS::Region}:ec2:recover'

  CriticalInstanceAlarm:
    Type: AWS::CloudWatch::CompositeAlarm
    DependsOn:
      - HighCPUAlarm
      - StatusCheckFailedAlarm
    Properties:
      AlarmName: !Sub '${EnvironmentName}-CriticalInstance-${InstanceId}'
      AlarmDescription: 'High CPU AND failed status check = critical incident'
      AlarmRule: !Sub >
        ALARM("${EnvironmentName}-HighCPU-${InstanceId}") AND
        ALARM("${EnvironmentName}-StatusCheckFailed-${InstanceId}")
      AlarmActions:
        - !Ref AlertSNSTopic

  ApplicationLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub '/ec2/${EnvironmentName}/${InstanceId}/application'
      RetentionInDays: 30
      Tags:
        - Key: Environment
          Value: !Ref EnvironmentName

  ApplicationErrorMetricFilter:
    Type: AWS::Logs::MetricFilter
    Properties:
      LogGroupName: !Ref ApplicationLogGroup
      FilterName: ApplicationErrorCount
      FilterPattern: '[timestamp, requestId, level=ERROR, ...]'
      MetricTransformations:
        - MetricName: ApplicationErrorCount
          MetricNamespace: !Sub 'CustomApp/${EnvironmentName}'
          MetricValue: '1'
          DefaultValue: 0
          Unit: Count

  ApplicationErrorAlarm:
    Type: AWS::CloudWatch::Alarm
    DependsOn: ApplicationErrorMetricFilter
    Properties:
      AlarmName: !Sub '${EnvironmentName}-ApplicationErrors-High'
      AlarmDescription: 'High application error rate'
      Namespace: !Sub 'CustomApp/${EnvironmentName}'
      MetricName: ApplicationErrorCount
      Statistic: Sum
      Period: 300
      EvaluationPeriods: 2
      DatapointsToAlarm: 2
      Threshold: 10
      ComparisonOperator: GreaterThanOrEqualToThreshold
      TreatMissingData: notBreaching
      AlarmActions:
        - !Ref AlertSNSTopic

  MonitoringDashboard:
    Type: AWS::CloudWatch::Dashboard
    Properties:
      DashboardName: !Sub '${EnvironmentName}-EC2-Monitoring'
      DashboardBody: !Sub |
        {
          "widgets": [
            {
              "type": "metric",
              "x": 0, "y": 0, "width": 12, "height": 6,
              "properties": {
                "title": "CPU Utilization",
                "metrics": [["AWS/EC2", "CPUUtilization", "InstanceId", "${InstanceId}"]],
                "period": 300,
                "stat": "Average",
                "view": "timeSeries"
              }
            },
            {
              "type": "metric",
              "x": 12, "y": 0, "width": 12, "height": 6,
              "properties": {
                "title": "Network In/Out",
                "metrics": [
                  ["AWS/EC2", "NetworkIn", "InstanceId", "${InstanceId}"],
                  ["AWS/EC2", "NetworkOut", "InstanceId", "${InstanceId}"]
                ],
                "period": 300,
                "stat": "Average",
                "view": "timeSeries"
              }
            },
            {
              "type": "alarm",
              "x": 0, "y": 6, "width": 24, "height": 3,
              "properties": {
                "title": "Alarm States",
                "alarms": [
                  "arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:${EnvironmentName}-HighCPU-${InstanceId}",
                  "arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:${EnvironmentName}-StatusCheckFailed-${InstanceId}"
                ]
              }
            }
          ]
        }

Outputs:
  SNSTopicArn:
    Description: ARN of the alert SNS topic
    Value: !Ref AlertSNSTopic
    Export:
      Name: !Sub '${EnvironmentName}-AlertSNSTopic'

  DashboardURL:
    Description: URL of the CloudWatch dashboard
    Value: !Sub 'https://${AWS::Region}.console.aws.amazon.com/cloudwatch/home#dashboards:name=${EnvironmentName}-EC2-Monitoring'

Reference Architectures

Complete Monitoring Architecture (Production)

flowchart TD
    subgraph VPC["Production VPC"]
        subgraph EC2_Group["EC2 Instances (Auto Scaling Group)"]
            App1["App Server 1\n+ CloudWatch Agent"]
            App2["App Server 2\n+ CloudWatch Agent"]
        end
        RDS["RDS Multi-AZ"]
        ALB["Application Load Balancer"]
    end

    subgraph ECS_Cluster["ECS Cluster (Containers)"]
        Svc1["API Service\n(Container Insights enabled)"]
        Svc2["Worker Service\n(Container Insights enabled)"]
    end

    Lambda_Fn["Lambda Functions\n(X-Ray enabled)"]

    subgraph CW["Amazon CloudWatch"]
        Metrics_CW["Metrics\n(AWS/EC2, AWS/RDS, AWS/ALB, CWAgent)"]
        Logs_CW["Logs\n(app logs, system logs, VPC Flow Logs)"]
        Alarms_CW["Alarms\n(metric + composite)"]
        Dashboard_CW["Dashboards"]
        ContI["Container Insights"]
    end

    XRay_Svc["AWS X-Ray\n(Service Map + Traces)"]

    subgraph Notification["Notification and Remediation"]
        SNS_Topic["SNS Topic"]
        Email["Email / SMS\n(ops team)"]
        Lambda_Rem["Lambda\n(auto remediation)"]
        ASPolicy["Auto Scaling\nPolicy"]
    end

    EB["Amazon EventBridge\n(Rules)"]

    App1 --> Metrics_CW
    App1 --> Logs_CW
    App2 --> Metrics_CW
    App2 --> Logs_CW
    RDS --> Metrics_CW
    RDS --> Logs_CW
    ALB --> Metrics_CW
    Svc1 --> ContI
    Svc2 --> ContI
    Lambda_Fn --> Metrics_CW
    Lambda_Fn --> XRay_Svc
    ContI --> Metrics_CW
    Metrics_CW --> Alarms_CW
    Alarms_CW --> Dashboard_CW
    Alarms_CW --> EB
    EB --> SNS_Topic
    SNS_Topic --> Email
    EB --> Lambda_Rem
    Alarms_CW --> ASPolicy

Automatic Detection and Remediation Flow

sequenceDiagram
    participant CWAgent as CloudWatch Agent
    participant CW_M as CloudWatch Metrics
    participant Alarm as CloudWatch Alarm
    participant EB as EventBridge
    participant Lambda as Lambda Remediation
    participant SNS as SNS Topic
    participant Ops as Ops Team
    participant EC2 as EC2 Instance

    CWAgent->>CW_M: Publishes mem_used_percent = 95%
    CW_M->>Alarm: Threshold > 90%, 3 periods
    Alarm->>Alarm: State -> ALARM
    Alarm->>EB: CloudWatch Alarm State Change Event
    EB->>Lambda: Rule HighMemory
    EB->>SNS: Rule NotifyOps
    SNS->>Ops: Email/SMS alert
    Lambda->>EC2: Memory cleanup script
    CW_M->>Alarm: mem_used_percent drops to 65%
    Alarm->>Alarm: State -> OK
    Alarm->>SNS: Alarm cleared notification
    SNS->>Ops: Email: Alarm cleared

Summary and Key Takeaways

Summary by Service

Best Practices

1. Enable Detailed Monitoring on production EC2 instances for 1-minute granularity
2. Install the CloudWatch Agent on all instances to collect RAM, disk, and application metrics
3. Define Retention Policies on Log Groups (no unlimited free retention)
4. Use Composite Alarms to avoid false positives and unnecessary notifications
5. Configure missing data treatment (TreatMissingData) according to context
6. Tag CloudWatch resources (Log Groups, Alarms) for cost management
7. Limit Logs Insights query scope (select minimum Log Groups, narrowest possible time range)
8. Enable Container Insights on all production ECS/EKS clusters
9. Use X-Ray with sampling to avoid excessive costs on high-traffic APIs
10. Create consolidated CloudWatch Dashboards per application or team

Decision Architecture: Which Solution to Choose?

flowchart TD
    Q1{"What do you\nneed to monitor?"}
    Q1 --> |"System metrics\nCPU, network, storage"| Q2{"Source?"}
    Q1 --> |"Application or\nsystem logs"| Q3{"Access\nfrequency?"}
    Q1 --> |"Distributed requests\nmicroservices"| XRay["AWS X-Ray"]
    Q1 --> |"AWS Events\nstate changes"| EB["EventBridge Rules"]

    Q2 --> |"Standard AWS metric"| CW_Basic["CloudWatch Metrics\n(Basic or Detailed)"]
    Q2 --> |"RAM, disk, custom"| Agent["CloudWatch Agent\n+ Custom Namespace"]
    Q2 --> |"ECS/EKS containers"| ContI["Container Insights"]
    Q2 --> |".NET / SQL Server app"| AppI["Application Insights"]

    Q3 --> |"Frequently accessed"| Standard["Log Class: Standard"]
    Q3 --> |"Archives, audit logs"| IA["Log Class: Infrequent Access"]

    CW_Basic --> Alarm["CloudWatch Alarm\n(metric threshold)"]
    Agent --> Alarm
    Standard --> Insights["Logs Insights\n(queries)"]
    Standard --> MetricFilter["Metric Filter\n(logs to metrics)"]
    MetricFilter --> Alarm
    Alarm --> |"False positives?"| Composite["Composite Alarm"]
    Alarm --> |"Direct action"| Actions["SNS / Auto Scaling\n/ EC2 Action"]

Additional Resources

Search Terms

aws · monitoring · fundamentals · core · services · amazon · web · cloudwatch · alarm · insights · logs · metrics · concepts · ec2 · eventbridge · x-ray · agent · alarms · architecture · enabling · flow · log · additional · available