DevOps Troubleshooting: Using Monitoring and Logging

Course: DevOps Troubleshooting — Using Monitoring and Logging Demo environment: Ubuntu 24.04

Table of Contents

1. Observability Overview
2. Module 1 — Setup Monitoring Tools
   • Why Monitoring Is Essential
   • Prometheus and Node Exporter
   • Grafana — Installation and Configuration
   • Grafana Dashboards
3. Module 2 — Implement Logging Solutions
   • Logging Fundamentals
   • The ELK Stack
   • Installing the ELK Stack
   • Configuring Logstash
   • Analyzing Logs in Kibana
4. Module 3 — Integration into CI/CD Pipelines
   • Monitoring CI/CD Pipelines
   • Jenkins + Prometheus
   • Automating Application Observability
5. Architecture Diagrams
   • Observability Stack
   • ELK Pipeline
   • Prometheus / Grafana Pipeline
   • Alerting Flow
   • Distributed Tracing
6. YAML Configurations / Reference Configs
7. Observability Tools Reference Tables
8. Demo Files — Source Code

1. Observability Overview

The Problem Without Monitoring

Imagine an e-commerce website during a promotional campaign. Without monitoring or logging, the Ops team is blind to:

• The gradual increase in server load
• Bottlenecks as they appear
• System errors multiplying

Monitoring in DevOps is analogous to a car’s dashboard instruments: the speedometer, fuel gauge, and tire pressure sensor. Each instrument provides critical information about the system’s state. Without this data, you’re driving blind.

The Three Pillars of Observability

graph TD
    O[Observability] --> L[Logs]
    O --> M[Metrics]
    O --> T[Traces]
    L --> L1[Discrete Events\nErrors, access, audits]
    M --> M1[Numerical Measurements\nCPU, memory, latency]
    T --> T1[Execution Paths\nDistributed tracing]

2. Module 1 — Setup Monitoring Tools

2.1 Why Monitoring Is Essential

Monitoring is crucial for the following reasons:

Proactive Problem Detection Identify potential issues before they cause downtime or user impact. By tracking key metrics, you identify bottlenecks and degradation zones.

Incident Analysis (Root Cause Analysis) When an incident occurs, monitoring data provides the essential context for diagnosing the root cause.

Performance Optimization Identify bottlenecks and areas for improvement in infrastructure or code.

Reliability and Availability (SLA/SLO/SLI) By proactively addressing issues and optimizing performance, monitoring directly contributes to increased uptime.

Data-Driven Scalability Monitoring data guides decisions on infrastructure scaling, code optimization, and growth planning.

Regulatory Compliance Some industries require demonstrating that systems operate within specific parameters — monitoring provides that evidence.

2.2 Prometheus and Node Exporter

Prometheus Architecture

graph LR
    NE[Node Exporter\n:9100/metrics] -->|scrape| P[Prometheus Server\n:9090]
    APP[Application\n/metrics] -->|scrape| P
    JENKINS[Jenkins\n:8080/prometheus] -->|scrape| P
    P --> G[Grafana\n:3000]
    P --> AM[Alertmanager]
    AM --> EMAIL[Email]
    AM --> SLACK[Slack]
    AM --> PAGER[PagerDuty]

Key Prometheus Components

Installing Prometheus on Ubuntu 24.04

#!/bin/sh
# Install Prometheus and Node Exporter via apt
sudo apt-get install prometheus -y

# Verify that Node Exporter is active
sudo systemctl status prometheus-node-exporter --no-pager -n 0

# Verify that Prometheus is active
sudo systemctl status prometheus --no-pager -n 0

Verification: Access http://localhost:9100/metrics to see the metrics exposed by Node Exporter.

Example metric: node_memory_Active_bytes — number of bytes of memory currently in use.

prometheus.yml Configuration — Basic Structure

global:
  scrape_interval: 15s       # Default collection frequency
  evaluation_interval: 15s   # Alert rule evaluation frequency

scrape_configs:
  # Job for Node Exporter (system metrics)
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

  # Job for Prometheus itself (self-monitoring)
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Job for Jenkins (CI/CD metrics)
  - job_name: 'jenkins'
    scrape_interval: 5s
    metrics_path: '/prometheus'
    static_configs:
      - targets: ['localhost:8080']

Essential PromQL Queries

2.3 Grafana — Installation and Configuration

Installation on Ubuntu 24.04

#!/bin/bash
# Create a directory for GPG keys
sudo mkdir -p /etc/apt/keyrings/

# Download and save the Grafana key
wget -q -O - https://packages.grafana.com/gpg.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg

# Add the stable Grafana repository
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] \
  https://packages.grafana.com/oss/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list

# Update and install
sudo apt-get update
sudo apt-get install grafana -y

# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

# Verification
sudo systemctl status grafana-server

UI Access: http://localhost:3000 Default credentials: admin / admin (must be changed on first login)

Configuring the Prometheus Datasource in Grafana

1. Left menu → Connections → Data sources
2. Add data source → select Prometheus
3. Name field: prometheus (default)
4. Connection URL: http://localhost:9090
5. Save & test → verify the green confirmation banner

2.4 Grafana Dashboards

Creating a Custom Dashboard

1. Menu → Dashboards → Create dashboard
2. Add visualization → select the prometheus datasource
3. Queries tab → choose the metric (e.g.: node_memory_Active_bytes)
4. Run queries → verify the graph
5. Save dashboard → name the dashboard

Available Visualization Types

Importing a Community Dashboard

Node Exporter dashboards are available at grafana.com/grafana/dashboards. Import procedure:

1. Download the JSON from the Grafana website
2. In the UI → Dashboards → New → Import
3. Drag and drop the JSON
4. Select the Prometheus datasource
5. Click Import

3. Module 2 — Implement Logging Solutions

3.1 Logging Fundamentals

Logs are the black boxes of your software. In complex IT environments, applications generate massive amounts of data. Logs provide a historical record of events:

• What happened
• When it occurred
• Why (sometimes) it happened

Logging Use Cases

Structure of a Typical JSON Log

{
  "ip": "10.0.1.72",
  "user_id": 3915,
  "timestamp": "2025-03-10T09:45:22.654321Z",
  "method": "GET",
  "url": "/course/devops",
  "status": 200,
  "response_time_ms": 112,
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}

3.2 The ELK Stack

ELK is an acronym for three complementary open-source components:

graph LR
    SOURCES["Log Sources\n(Applications, Servers,\nContainers, Services)"] --> LS["Logstash\nCollect · Transform · Send"]
    LS --> ES["Elasticsearch\nStore · Index · Search"]
    ES --> K["Kibana\nVisualize · Analyze · Dashboard"]
    style SOURCES fill:#f9f,stroke:#333
    style LS fill:#ffcc00,stroke:#333
    style ES fill:#00bcd4,stroke:#333
    style K fill:#4caf50,stroke:#333

Why the ELK Stack?

• Scalability: Elasticsearch is distributed, designed to handle terabytes of data
• Flexibility: Logstash accepts any log source
• Search power: Near-real-time full-text queries
• Rich visualizations: Kibana offers dashboards, geographic maps, and advanced analytics

3.3 Installing the ELK Stack

Prerequisite: Java 17

Elasticsearch requires Java 17 at minimum.

sudo apt-get install openjdk-17-jdk -y

Adding Elastic Repositories

#!/bin/bash
# Create directory for GPG keys
sudo mkdir -p /etc/apt/keyrings/

# Download and save the Elastic key
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | \
  sudo gpg --dearmor -o /etc/apt/keyrings/elasticsearch-keyring.gpg

# Add the stable repository
echo "deb [signed-by=/etc/apt/keyrings/elasticsearch-keyring.gpg] \
  https://artifacts.elastic.co/packages/8.x/apt stable main" | \
  sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Update packages
sudo apt-get update

# Install all three components
sudo apt-get install elasticsearch logstash kibana -y

# Enable and start
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch logstash kibana
sudo systemctl start elasticsearch logstash kibana

# Verify statuses
sudo systemctl status elasticsearch --no-pager -n 0
sudo systemctl status logstash --no-pager -n 0
sudo systemctl status kibana --no-pager -n 0

ELK Stack Ports and URLs

Kibana ↔ Elasticsearch Connection

After installation, Kibana must be connected to Elasticsearch via an enrollment token:

# Reset the elastic user password
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic

# Generate an enrollment token for Kibana
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana

# Get the Kibana verification code
sudo /usr/share/kibana/bin/kibana-verification-code

3.4 Configuring Logstash

Logstash uses configuration files in /etc/logstash/conf.d/ with three main blocks:

Structure of a Logstash Configuration File

input  → filter → output

Config: Web Access Log Ingestion

File: /etc/logstash/conf.d/access.conf

# /etc/logstash/conf.d/access.conf

input {
  file {
    path => "/var/log/access.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "access"
    user => "elastic"
    password => "elastic"
    ssl_enabled => true
    ssl_verification_mode => "none"
  }
  stdout {
    codec => "rubydebug"
  }
}

Config: Custom Application Log Ingestion

File: /etc/logstash/conf.d/app-metrics.conf

input {
  file {
    path => "/var/log/app-metrics.log"
    type => "app-metrics"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "app-metrics-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "elastic"
    ssl_enabled => true
    ssl_verification_mode => "none"
  }
  stdout {
    codec => "rubydebug"
  }
}

Key Logstash Configuration Parameters

Logstash Configuration Script

#!/bin/sh
# Display the configuration
cat logstash-access-logs.conf

# Copy to the configuration directory
sudo cp logstash-access-logs.conf /etc/logstash/conf.d/

# Adjust permissions for the logstash user
sudo chmod 644 /etc/logstash/conf.d/logstash-access-logs.conf

# Restart to apply changes
sudo systemctl restart logstash

# Verification
sudo systemctl status logstash --no-pager -n 0

# Verify that the Elasticsearch index was created
curl -k -u elastic:elastic https://localhost:9200/_cat/indices

3.5 Analyzing Logs in Kibana

Creating a Data View in Kibana

1. Stack Management → Index Patterns (Data Views)
2. Create a pattern matching the index: access*
3. Select the timestamp field: @timestamp
4. Save

Creating a Kibana Dashboard

Example: Web access log analysis dashboard

1. Home → Dashboards → Create a dashboard
2. Create visualization for each panel:

Panel 1 — Total requests (Metric)

• Type: Metric
• Metric: Count of Records
• Name: “Total number of requests”

Panel 2 — Distribution by status code (Pie Chart)

• Type: Pie
• Slice by: Top values of status field (Top 10)
• Color palette: Status palette
• Metric: Count of status

Reading the pie chart: The majority of requests should be 200 OK. 4xx and 5xx codes indicate errors worth investigating.

4. Module 3 — Integration into CI/CD Pipelines

4.1 Monitoring CI/CD Pipelines

Integrating monitoring into CI/CD pipelines offers two levels of value:

For CI/CD Pipelines Themselves

For Deployed Applications

4.2 Jenkins + Prometheus

Jenkins Observability Architecture

sequenceDiagram
    participant DEV as Developer
    participant GIT as Git Repository
    participant JENKINS as Jenkins CI/CD
    participant PROM as Prometheus
    participant GRAFANA as Grafana

    DEV->>GIT: git push
    GIT->>JENKINS: Webhook trigger
    JENKINS->>JENKINS: Run pipeline stages
    JENKINS-->>PROM: Expose /prometheus metrics
    PROM->>JENKINS: Scrape every 5s
    PROM-->>GRAFANA: Metrics available
    GRAFANA->>GRAFANA: Update dashboards

Installing the Prometheus Plugin in Jenkins

1. Manage Jenkins → Plugins → Available plugins tab
2. Search for: Prometheus metrics
3. Check Prometheus metrics
4. Install → check Restart Jenkins when installation is complete

Verification: http://localhost:8080/prometheus/

Key exposed metric: jenkins_job_count_value — number of pipelines currently running.

Adding Jenkins as a Prometheus Target

Configuration script (1-configure-jenkins-metrics.sh):

#!/bin/sh
config_file="/etc/prometheus/prometheus.yml"
new_job_name="jenkins"

# Check if jenkins config already exists
if grep -q 'jenkins' "$config_file"; then
    echo "Scrape configuration for job '$new_job_name' already exists."
else
    # Add the new job
    echo "  - job_name: '$new_job_name'" >> $config_file
    echo "    scrape_interval: 5s" >> $config_file
    echo "    metrics_path: '/prometheus'" >> $config_file
    echo "    static_configs:" >> $config_file
    echo "    - targets: ['localhost:8080']" >> $config_file
    echo "Scrape configuration added for '$new_job_name'."
fi

# Restart Prometheus to apply changes
sudo systemctl restart prometheus
sudo systemctl status prometheus --no-pager -n 0

Verification: Access http://localhost:9090/classic/targets to confirm that Jenkins appears as an active target.

4.3 Automating Application Observability

Python Application with Prometheus Client

The app-metrics application demonstrates instrumenting a Python application with prometheus_client:

import logging
import os
import time
from prometheus_client import Counter, generate_latest
from http.server import HTTPServer, BaseHTTPRequestHandler
from threading import Thread

# Define a custom Counter metric
requests_processed = Counter(
    'webapp_processed_requests_total',
    'The total number of processed web requests'
)

# Configure logging to file
log_file_path = "/var/log/app-metrics.log"
logging.basicConfig(
    filename=log_file_path,
    level=logging.INFO,
    format='%(asctime)s - %(message)s'
)
logger = logging.getLogger(__name__)

def process_requests():
    """Increments the counter and logs every second."""
    while True:
        requests_processed.inc()
        timestamp = time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
        logger.info(f"Processed request at {timestamp}")
        time.sleep(1)

class MetricsHandler(BaseHTTPRequestHandler):
    """HTTP handler exposing Prometheus metrics."""
    def do_GET(self):
        if self.path == '/metrics':
            self.send_response(200)
            self.send_header('Content-Type', 'text/plain')
            self.end_headers()
            self.wfile.write(generate_latest())
        else:
            self.send_response(404)
            self.end_headers()

def run_http_server():
    httpd = HTTPServer(('', 2112), MetricsHandler)
    print("Serving metrics on port 2112...")
    httpd.serve_forever()

if __name__ == "__main__":
    # Background processing thread
    processing_thread = Thread(target=process_requests)
    processing_thread.daemon = True
    processing_thread.start()

    # HTTP server on the main thread
    run_http_server()

Key points:

• Port 2112 for metrics exposure (/metrics)
• Counter: metric that only increases (total operations, errors, requests)
• generate_latest(): serializes all metrics in Prometheus text format

Jenkins Pipeline — Deployment with Integrated Observability

pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                sh '''
                    cd /home/jenkins/app-metrics
                    python3 -m venv venv
                    source venv/bin/activate
                    pip install -r requirements.txt
                '''
            }
        }
        stage('Deploy') {
            steps {
                sh '''
                    # Copy the systemd service
                    sudo cp app-metrics.service /etc/systemd/system/
                    sudo systemctl daemon-reload
                    sudo systemctl enable app-metrics
                    sudo systemctl restart app-metrics
                    sudo systemctl status app-metrics --no-pager -n 0
                '''
            }
        }
        stage('Configure Monitoring') {
            steps {
                sh '''
                    # Add the app as a Prometheus target
                    echo "  - job_name: 'app-metrics'" >> /etc/prometheus/prometheus.yml
                    echo "    static_configs:" >> /etc/prometheus/prometheus.yml
                    echo "    - targets: ['localhost:2112']" >> /etc/prometheus/prometheus.yml
                    sudo systemctl restart prometheus
                '''
            }
        }
        stage('Configure Logging') {
            steps {
                sh '''
                    # Deploy Logstash config
                    sudo cp app-metrics-logstash.conf /etc/logstash/conf.d/
                    sudo systemctl restart logstash
                '''
            }
        }
    }
}

Web Access Log Generator (Python)

import json
import time
import random
from datetime import datetime

def generate_log():
    status_codes = [200] * 20 + [201] * 2 + [204] * 2 + [401] * 4 + [404] * 3 + [500] * 1
    urls = ['/course/python'] * 50 + ['/course/kubernetes'] * 35 + \
           ['/course/devops'] * 25 + ['../../etc/passwd'] * 2  # Attack simulation

    log = {
        "ip": f"{random.randint(0,255)}.{random.randint(0,255)}.{random.randint(0,255)}.{random.randint(0,255)}",
        "user_id": random.randint(1000, 9999),
        "timestamp": datetime.now().strftime("%Y-%m-%dT%H:%M:%S.%fZ"),
        "method": random.choice(["GET", "POST", "PUT", "DELETE", "PATCH"]),
        "url": random.choice(urls),
        "status": random.choice(status_codes),
        "response_time_ms": random.randint(20, 2000),
        "user_agent": "Mozilla/5.0 ..."
    }

    # Simulating an intrusion attempt (path traversal)
    if log['url'] == '../../etc/passwd':
        log['status'] = 500
        log['ip'] = '203.0.113.88'  # Suspicious IP

    return log

def main():
    while True:
        log = generate_log()
        with open("/var/log/access.log", "a") as f:
            f.write(json.dumps(log) + "\n")
        time.sleep(random.randint(1, 10))

Security note: This generator simulates a path traversal attack (../../etc/passwd). In Kibana, alerts can be created for suspicious URLs or HTTP 5xx codes from a specific IP.

5. Architecture Diagrams

5.1 Observability Stack

graph TB
    subgraph Sources["Data Sources"]
        APP[Applications\nMicroservices]
        INFRA[Infrastructure\nVMs / Containers]
        PIPE[CI/CD Pipelines\nJenkins / GitLab CI]
        NET[Network\nLoad Balancers]
    end

    subgraph Collection["Collection Layer"]
        NE[Node Exporter\nSystem metrics]
        PC[Prometheus Client\nApp metrics]
        BT[Beats / Agents\nFilebeat, Metricbeat]
        LS[Logstash\nLog pipeline]
        OT[OpenTelemetry\nCollector]
    end

    subgraph Storage["Storage Layer"]
        PROM[Prometheus\nTSDB — Metrics]
        ES[Elasticsearch\nLogs & Events]
        TEMPO[Tempo / Jaeger\nDistributed Traces]
    end

    subgraph Visualization["Visualization & Alerting Layer"]
        G[Grafana\nUnified Dashboards]
        K[Kibana\nLog analytics]
        AM[Alertmanager\nAlert management]
    end

    subgraph Notification["Notifications"]
        EMAIL[Email]
        SLACK[Slack]
        PAGER[PagerDuty]
        OPSGENIE[OpsGenie]
    end

    APP --> PC
    APP --> BT
    APP --> OT
    INFRA --> NE
    INFRA --> BT
    PIPE --> PC
    NET --> BT

    NE --> PROM
    PC --> PROM
    BT --> LS
    LS --> ES
    OT --> PROM
    OT --> ES
    OT --> TEMPO

    PROM --> G
    ES --> K
    ES --> G
    TEMPO --> G
    PROM --> AM

    G --> AM
    AM --> EMAIL
    AM --> SLACK
    AM --> PAGER
    AM --> OPSGENIE

5.2 ELK Pipeline

flowchart LR
    subgraph Input["Input — Filebeat / Logstash"]
        F1["/var/log/access.log"]
        F2["/var/log/app.log"]
        F3["Syslog / Journal"]
        F4["Docker logs"]
    end

    subgraph Logstash["Logstash Pipeline"]
        direction TB
        IN[Input Plugin\nfile / syslog / beats] --> FIL[Filter Plugin\njson / grok / mutate]
        FIL --> OUT[Output Plugin\nelasticsearch / stdout]
    end

    subgraph ES["Elasticsearch Cluster"]
        IDX1[Index: access-*]
        IDX2[Index: app-*]
        IDX3[Index: syslog-*]
    end

    subgraph Kibana["Kibana"]
        DV[Data Views\nIndex Patterns]
        DISC[Discover\nLog search]
        DASH[Dashboards\nVisualizations]
        ALERT[Alerts &\nRules]
    end

    F1 --> IN
    F2 --> IN
    F3 --> IN
    F4 --> IN

    OUT --> IDX1
    OUT --> IDX2
    OUT --> IDX3

    IDX1 --> DV
    IDX2 --> DV
    IDX3 --> DV
    DV --> DISC
    DV --> DASH
    DV --> ALERT

5.3 Prometheus / Grafana Pipeline

flowchart TD
    subgraph Targets["Targets — /metrics Exposure"]
        NE["Node Exporter\nlocalhost:9100/metrics"]
        JENKINS["Jenkins\nlocalhost:8080/prometheus"]
        APP["App Metrics\nlocalhost:2112/metrics"]
        PSELF["Prometheus\nlocalhost:9090/metrics"]
    end

    subgraph Prometheus["Prometheus Server (TSDB)"]
        SC["Scrape Manager\nScraping every N seconds"]
        TSDB["Time Series DB\nLocal storage"]
        RULES["Rules Engine\nAlerting Rules"]
        PAPI["HTTP API\n/api/v1/query"]
    end

    subgraph Grafana["Grafana"]
        DS["Datasource\nPrometheus"]
        PANELS["Panels\nPromQL queries"]
        DASH["Dashboards"]
        GALERT["Unified Alerting"]
    end

    AM["Alertmanager\n:9093"]
    NOTIF["Notifications\nEmail / Slack / PagerDuty"]

    NE -->|"HTTP GET /metrics"| SC
    JENKINS -->|"HTTP GET /prometheus"| SC
    APP -->|"HTTP GET /metrics"| SC
    PSELF -->|"HTTP GET /metrics"| SC

    SC --> TSDB
    TSDB --> RULES
    TSDB --> PAPI
    RULES -->|"FIRING alerts"| AM
    AM --> NOTIF

    PAPI --> DS
    DS --> PANELS
    PANELS --> DASH
    DASH --> GALERT
    GALERT --> AM

5.4 Alerting Flow

stateDiagram-v2
    [*] --> Inactive: Metric within normal thresholds

    Inactive --> Pending: Threshold exceeded\n(alert rule triggered)
    note right of Pending
        Waiting for delay\nfor: 5m
    end note

    Pending --> Firing: Threshold exceeded\nfor the minimum duration
    Firing --> Inactive: Metric returned\nwithin thresholds

    Firing --> AlertManager: Alert sent
    AlertManager --> Routing: Receiver selection\nby labels
    Routing --> Grouping: Grouping\nsimilar alerts
    Grouping --> Inhibition: Suppressing\nredundant alerts
    Inhibition --> Silencing: Silencing\n(planned maintenance)
    Silencing --> Notification: Final notification
    Notification --> [*]

Prometheus Alert Rule (Example)

# /etc/prometheus/rules/alerts.yml
groups:
  - name: infrastructure
    rules:
      # Alert if a server has been down for more than 2 minutes
      - alert: InstanceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."

      # Alert if memory usage exceeds 85%
      - alert: HighMemoryUsage
        expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory on {{ $labels.instance }}"
          description: "Memory usage is at {{ $value }}%"

      # Alert if CPU exceeds 90%
      - alert: HighCpuUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"

Alertmanager Configuration

# /etc/alertmanager/alertmanager.yml
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s       # Wait before sending first group
  group_interval: 5m    # Interval between sends for an existing group
  repeat_interval: 12h  # Repeat if alert remains active
  receiver: 'default'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
    - match:
        severity: warning
      receiver: 'slack'

receivers:
  - name: 'default'
    email_configs:
      - to: 'ops-team@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.example.com:587'

  - name: 'slack'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/TOKEN'
        channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'pagerduty'
    pagerduty_configs:
      - routing_key: 'YOUR_PAGERDUTY_INTEGRATION_KEY'

5.5 Distributed Tracing

sequenceDiagram
    participant USER as User
    participant GW as API Gateway
    participant AUTH as Auth Service
    participant ORDER as Order Service
    participant DB as Database
    participant CACHE as Redis Cache
    participant OTEL as OpenTelemetry\nCollector
    participant JAEGER as Jaeger / Tempo

    USER->>GW: POST /api/orders [Trace-ID: abc123]
    GW->>AUTH: Validate token [Span: auth-validate]
    AUTH-->>GW: Token valid
    GW->>ORDER: Create order [Span: order-create]
    ORDER->>CACHE: Check inventory [Span: cache-lookup]
    CACHE-->>ORDER: Cache miss
    ORDER->>DB: Query inventory [Span: db-query]
    DB-->>ORDER: Stock available
    ORDER->>DB: Insert order [Span: db-insert]
    DB-->>ORDER: Order created
    ORDER-->>GW: Order ID: 789
    GW-->>USER: 201 Created

    Note over GW,OTEL: Each service sends its spans via OTLP
    GW-)OTEL: Spans [Trace-ID: abc123]
    AUTH-)OTEL: Spans
    ORDER-)OTEL: Spans
    DB-)OTEL: Spans
    OTEL-)JAEGER: Complete traces

OpenTelemetry Collector Configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['0.0.0.0:8888']

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
    spike_limit_mib: 128

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  prometheus:
    endpoint: "0.0.0.0:8889"
  elasticsearch:
    endpoints: ["http://elasticsearch:9200"]
    logs_index: "otel-logs"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [elasticsearch]

6. YAML Configurations / Reference Configs

6.1 Prometheus — Complete Configuration

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: 'devops-monitor'
    environment: 'production'

# Load alert rules
rule_files:
  - "/etc/prometheus/rules/*.yml"

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node Exporter — system metrics
  - job_name: 'node'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  # Jenkins — CI/CD metrics
  - job_name: 'jenkins'
    scrape_interval: 5s
    metrics_path: '/prometheus'
    static_configs:
      - targets: ['localhost:8080']

  # Custom application
  - job_name: 'app-metrics'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:2112']

6.2 Grafana — Datasource as Code (Provisioning)

# /etc/grafana/provisioning/datasources/prometheus.yaml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://localhost:9090
    isDefault: true
    jsonData:
      timeInterval: "15s"
      queryTimeout: "60s"
      httpMethod: POST
    version: 1
    editable: true

  - name: Elasticsearch
    type: elasticsearch
    access: proxy
    url: http://localhost:9200
    basicAuth: true
    basicAuthUser: elastic
    secureJsonData:
      basicAuthPassword: elastic
    jsonData:
      index: "access-*"
      timeField: "@timestamp"
      logMessageField: message
      logLevelField: level
    version: 1
    editable: true

6.3 Loki — Lightweight Logging Stack (ELK Alternative)

# loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

6.4 Promtail — Collection Agent for Loki

# promtail-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # System logs
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          host: my-server
          __path__: /var/log/*.log

  # Web access logs
  - job_name: access-logs
    static_configs:
      - targets:
          - localhost
        labels:
          job: access
          __path__: /var/log/access.log
    pipeline_stages:
      - json:
          expressions:
            status: status
            url: url
            response_time: response_time_ms
      - labels:
          status:
          url:

6.5 Docker Compose — Complete Monitoring Stack

# docker-compose.monitoring.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:latest
    container_name: node-exporter
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    restart: unless-stopped
    depends_on:
      - prometheus

  loki:
    image: grafana/loki:latest
    container_name: loki
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yaml:/etc/loki/local-config.yaml
    command: -config.file=/etc/loki/local-config.yaml
    restart: unless-stopped

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    volumes:
      - /var/log:/var/log:ro
      - ./promtail-config.yaml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml
    restart: unless-stopped
    depends_on:
      - loki

volumes:
  prometheus_data:
  grafana_data:

7. Observability Tools Reference Tables

7.1 Metrics Solution Comparison

7.2 Logging Solution Comparison

7.3 Tracing Solution Comparison

7.4 Essential Metrics to Monitor

Infrastructure (Node Exporter)

Applications

CI/CD Jenkins

7.5 Logs — Severity Levels

7.6 HTTP Codes — Interpretation for Monitoring

8. Demo Files — Source Code

8.1 Demo File Structure

devops troubleshooting using monitoring and logging/
├── 01/
│   └── demos/
│       ├── 1-install-prometheus.sh     # Install Prometheus + Node Exporter
│       └── 2-install-grafana.sh        # Install Grafana
├── 02/
│   └── demos/
│       ├── 0-setup-log-generator.sh    # Set up log generator
│       ├── 1-install-ELK.sh            # Install Elasticsearch + Logstash + Kibana
│       ├── 2-setup-logstash.sh         # Configure Logstash
│       ├── logs-generator.py           # Python app generating web logs
│       └── logstash-access-logs.conf   # Logstash config for access logs
└── 03/
    └── demos/
        ├── 0-environment-setup.sh              # Full environment setup
        ├── 1-configure-jenkins-metrics.sh      # Configure Jenkins → Prometheus
        ├── 2-setup-app.sh                      # Deploy app-metrics
        └── app-metrics/
            ├── main.py                          # Python app with prometheus_client
            └── app-metrics-logstash.conf       # Logstash config for the app

8.2 Grafana Installation Script (Reference)

#!/bin/bash
# 2-install-grafana.sh

# Create directory for GPG keys
sudo mkdir -p /etc/apt/keyrings/

# Download and save the Grafana key
wget -q -O - https://apt.grafana.com/gpg.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg

# Add the stable repository
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] \
  https://packages.grafana.com/oss/deb stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list

sudo apt-get update
sudo apt-get install grafana -y

sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server --no-pager -n 0

8.3 Environment Variables and Security Best Practices

⚠️ Important — Credential Security

In the demos, passwords appear in plain text in Logstash configurations for simplicity. In production, use:

# Environment variables in Logstash
export ELASTIC_PASSWORD="$(vault kv get -field=password secret/elastic)"

# Reference in logstash.conf
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    user => "elastic"
    password => "${ELASTIC_PASSWORD}"
  }
}

Or use Logstash Keystore:

# Create the Logstash keystore
sudo -u logstash /usr/share/logstash/bin/logstash-keystore create

# Store the Elasticsearch password
sudo -u logstash /usr/share/logstash/bin/logstash-keystore add ELASTIC_PASSWORD

# Reference in config
password => "${ELASTIC_PASSWORD}"

Summary and Key Points

DevOps Observability Principles

1. Shift-left Observability: Integrate monitoring from development, not after deployment
2. Observability as Code: Version Prometheus, Grafana, and Alertmanager configs
3. Automated onboarding: Observability activates automatically with each CI/CD deployment
4. Correlation: Link logs, metrics, and traces for effective Root Cause Analysis
5. SLO/SLI/SLA: Base alerts on measurable objectives, not intuitions

Course completed — Generated from course transcripts and exercise files.

Search Terms

devops · troubleshooting · monitoring · logging · ci/cd · git · prometheus · configuration · observability · stack · elk · grafana · jenkins · kibana · log · logstash · application · architecture · comparison · dashboard · essential · installation · installing · pipeline