Node.js Microservices Monitoring and Logging

Table of Contents

1. Course presentation
2. Module 2 — Best logging practices for microservices

• 2.1 Verification of versions
• 2.2 Module introduction
• 2.3 Presentation of GloboTicket
• 2.4 Logging Example
• 2.5 Observability
• 2.6 Logging Libraries for Node.js
• 2.7 Logging Levels
• 2.8 Pino Example
• 2.9 Logging Formats
• 2.10 Logging best practices for microservices
• 2.11 Complete example of a log message
• 2.12 Demo — Logging in Express with Winston and Morgan

3. Centralized logging with the ELK stack

• 3.1 Module introduction
• 3.2 The ELK stack
• 3.3 Docker and Docker Compose
• 3.4 Demo — Starting ELK with Docker Compose
• 3.5 Log ingestion with Logstash
• 3.6 Demo — Logstash and Filebeat
• 3.7 Log storage with Elasticsearch
• 3.8 Demo — Elasticsearch
• 3.9 Viewing logs with Kibana
• 3.10 Demo — Kibana
• 3.11 ELK Tips and Best Practices
• 3.12 Demo — Debugging with ELK

4. Module 4 — Monitoring microservices with Prometheus and Grafana

• 4.1 Module introduction
• 4.2 Understanding monitoring
• 4.3 Prometheus and Grafana
• 4.4 Installation and configuration
• 4.5 Demo — Starting Prometheus and Grafana with Docker Compose
• 4.6 Instrumentation of Node.js microservices
• 4.7 Demo — Instrumentation of a Node.js application
• 4.8 Creating Grafana Dashboards
• 4.9 Demo — Grafana
• 4.10 Alerts configuration
• 4.11 Demo — Alerts
• 4.12 OpenTelemetry — Introduction

5. Distributed tracing with Jaeger and Zipkin

• 5.1 Module introduction
• 5.2 Why logging alone is not enough?
• 5.3 Demo — Deploy Jaeger and Zipkin with Docker Compose
• 5.4 Context Propagation
• 5.5 Demo — Tracing with Zipkin
• 5.6 Demo — Tracing with Jaeger
• 5.7 Demo — Tracing and monitoring with OpenTelemetry
• 5.8 Course Summary

6. Code Files — Full Reference

1. Course presentation

Hello everyone! My name is Julian Mateu, and welcome to my course, Node.js Microservices: Monitoring and Logging. I’m a Senior Software Engineer with nine years of experience, from small startups to large tech companies like Google and Amazon.

This course will teach you the concepts needed to navigate the world of observability in microservices using Node.js. In today’s technology landscape, effective monitoring and logging are essential for the health and performance of microservices. They help detect problems, optimize performance and maintain system reliability. Knowing how to configure and use popular observability tools can save you time and money when running your microservices in production.

Topics covered in the course

Here are the main topics covered in this course:

• Implementation and centralization of logging — using the ELK stack
• Monitoring with Prometheus and Grafana — metrics, dashboards and alerts
• Distributed tracing for efficient debugging with Jaeger and Zipkin
• Using OpenTelemetry for standardized and comprehensive observability

Learning Objectives

At the end of this course, you will have the skills and knowledge to design, implement and manage a robust observability strategy for your Node.js microservices, ensuring their performance and reliability.

Prerequisites

Before starting the course you should be familiar with:

• Basic Node.js programming
• Microservices Fundamentals
• General web development
• Basic knowledge of Docker is ideal, but not required

Recommended continuation

From here, you should feel comfortable diving deeper into observability and microservices with courses on the Elastic Stack, Prometheus and Grafana, Jaeger, Zipkin, and OpenTelemetry.

2. Logging Best Practices for Microservices

2.1 Verification of versions

This course was created using the following versions of the tools:

• Node.js
• Elastic Stack
• Prometheus
• Grafana
• Zipkin
• Jaeger

2.2 Module introduction

Welcome to the Node.js Microservices: Monitoring and Logging course. In this module, we explore many concepts related to logging and best practices for microservices.

Throughout the course, we use GloboTicket as a case study to understand how the concepts explored can be applied in hypothetical real-world scenarios.

Module plan:

1. Overview of the most commonly used logging libraries for Node.js
2. Discussion of logging levels and logging formats
3. Exploring logging best practices in microservices
4. Node.js logging demo

2.3 Presentation of GloboTicket

GloboTicket is an online platform where you can buy tickets for events like concerts and conferences. In the Node.js Microservices: The Big Picture course, we explored how GloboTicket transitioned from a monolithic architecture to a microservices architecture, and we looked at the challenges related to the distributed nature of microservices.

In this course we continue to use GloboTicket as a case study, but this time focusing on monitoring and logging. We’ll see how GloboTicket can monitor the health of its microservices and how to record important information to aid troubleshooting and debugging.

2.4 Logging Example

Understand the role of logging with a real-world example: Imagine that customers report problems with purchasing tickets, but the problem does not occur all the time — only in certain specific scenarios.

Without tools to understand what is happening, it would be very difficult to resolve the problem. You should rely on the customer’s description of the problem and try to reproduce it in a development environment. But even if you could reproduce the problem, it would be very difficult to understand what is happening without detailed information.

So let’s imagine that the development team included detailed logging in the ticket purchase process:

• Logging the different steps: when the customer selects the event, when he selects the number of tickets, when he enters payment information, etc.
• Logging the customer’s IP address, browser version, and other relevant information like timestamp of each step, ticket price, and relevant details.

After analyzing the logs, the team might discover that the problem occurs when a customer selects a specific event and attempts to purchase a certain number of tickets, or only when using a specific browser version.

Complexity in a microservices architecture

In a monolithic application, it is easy to enable this verbose logging and reproduce the issue. But in a microservices architecture, the problem is more complex:

• The issue could occur in one of the many microservices that are part of the ticket purchase process
• We need to find a way to correlate the logs of all these services to understand the problem
• Log volume is much higher in a microservices architecture
• Need to find a way to filter and search logs to find relevant information

2.5 Observability

When working with microservices, it is important to have observability. Observability is the ability to understand the internal state of a system by examining its outputs. In other words, it is the ability to understand what is happening inside the system by looking at its logs, metrics and traces.

The three pillars of observability are:

In this module we focus on logs, but we also explore metrics and traces in subsequent modules.

2.6 Logging libraries for Node.js

In Node.js, there are many logging libraries. The most commonly used are:

In this course, we use Winston to demonstrate logging in Node.js. Winston is a very popular logging library for Node.js, widely used in the industry. It has many features: logging levels, logging formats, and ability to log to different transports (console, files, external services).

2.7 Logging Levels

When logging in your application, it is important to use different levels to categorize logs. The most commonly used levels are:

Usefulness of levels

Using different logging levels allows you to filter logs based on their importance. For example, you can configure your logging system to only log messages of error level or higher, which excludes debug and info messages. This is useful for reducing the volume of logs and focusing on the most important messages.

Tip: Use string-based logging levels rather than numeric values ​​because different libraries may use different values.

2.8 Pino Example

Here’s a quick example of using different logging levels with Pino:

npm install pino pino-pretty

// module-2/01-logging-with-pino/index.js
import pino from 'pino'
import pretty from 'pino-pretty'

const logger = pino(pretty())

logger.level = process.env.LOG_LEVEL || 'trace'

logger.fatal('fatal message')
logger.error('error message')
logger.warn('warn message')
logger.info('info message')
logger.debug('debug message')
logger.trace('trace message')

In this example:

• We import pino and pino-pretty to format the logs with a timestamp, information and colors
• We create a logger instance with Pino
• We use the logger instance to log messages with different levels
• We set logger.level to trace to log all messages, but we can change the level to only log messages of a specific level

2.9 Logging formats

When logging in your application, it is important to use a consistent logging format. A logging format is the structure of the log message.

Unstructured logs

The simplest format. Messages are logged in plain text. Easy to read, but difficult to parse and analyze. It is not recommended to use unstructured logs because they are difficult to search, filter, and add context.

There are standardized unstructured formats like Common Log Format (CLF) and Combined Log Format, but teams often define their own formats.

Example:

192.168.1.1 - - [2024-01-15 10:30:45] "GET /events HTTP/1.1" 200 1024

Structured logs

The most recommended format. Messages are logged as structured data (JSON, XML, etc.). Not directly human readable, but easy to process by machines, and many tools can help analyze structured logs.

Advantages:

• Easy to analyze and visualize
• Ability to add context (service name, timestamp, level, etc.)
• Improves ability to search and filter
• Extensible and self-describing (no need to know the structure in advance)

JSON example:

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "error",
  "service": "ticket-service",
  "message": "Failed to process payment",
  "correlationId": "abc-123-def",
  "userId": "user-456"
}

Semi-structured logs

A mix of structured and unstructured data. Example: LogFmt (key-value pairs).

timestamp=2024-01-15T10:30:45Z level=error service=ticket-service msg="Failed to process payment"

2.10 Logging best practices for microservices

When working with microservices, it is important to follow these best practices:

1. Use levels effectively

• Remember when to use each level
• Use string-based logging levels rather than numeric values
• Different libraries may use different numeric values

2. Use structured JSON

• Facilitates analysis, visualization and search
• JSON is extensible and self-describing
• No backwards compatibility problem if you add a new field
• Many tools can analyze it

3. Save timestamps in ISO-8601

• International standard for representing dates and times – Easy to analyze
• Unambiguous (avoids day/month/year, time zone confusions)

4. Standardize the context

• Avoid relying on log message to provide context
• Use structured logs and include relevant information (user IDs, etc.)
• Include detailed information about the log source (service name, file name, line number, function name)
• Include build version or commitHash to correlate logs with application version

5. Use correlation IDs and stack traces

• Include correlation ID in logs to understand the path a request takes
• Make sure stack traces are included in the error logs (with caution, as they can be very long)
• Correlation IDs allow you to correlate logs from different services

6. Security — Avoid Sensitive Data (PII)

• Never include sensitive data in logs (passwords, credit card numbers, etc.)
• Beware of string interpolation which could expose sensitive data
• Implement a secure toString() method for objects that might contain PII data

// module-2/03-selective-logging/index.js
class UnsafeUser {
  constructor (id, name, email, password) {
    this.id = id
    this.name = name
    this.email = email
    this.password = password
  }

  toString () {
    // Ne faites pas ça! Cela expose les données PII et sensibles de l'utilisateur.
    return `User(${this.id}, ${this.name}, ${this.email}, ${this.password})`
  }
}

class SafeUser {
  constructor (id, name, email, password) {
    this.id = id
    this.name = name
    this.email = email
    this.password = password
  }

  toString () {
    // Cela expose uniquement l'ID de l'utilisateur, qui n'est pas sensible.
    return `User(${this.id})`
  }
}

const unsafeUser = new UnsafeUser(123, 'John', 'john@example.com', 'sensitive-password')
const safeUser = new SafeUser(123, 'John', 'john@example.com', 'sensitive-password')

// Cela affichera le mot de passe dans la console!
console.log(`User logged in: ${unsafeUser}`)

// Cela affichera uniquement l'ID de l'utilisateur
console.log(`User logged in: ${safeUser}`)

2.11 Complete example of a log message

Here are all the best practices in action with an example of a microservice log message:

{
  "timestamp": "2024-01-15T10:30:45.123Z",
  "level": "error",
  "service": "ticket-service",
  "message": "Failed to process payment",
  "correlationId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "buildInfo": {
    "commitHash": "abc1234",
    "nodeVersion": "v20.11.0",
    "version": "2.1.0"
  },
  "source": {
    "file": "paymentProcessor.js",
    "line": 42,
    "function": "processPayment"
  },
  "stackTrace": "Error: Payment gateway timeout\n    at processPayment (paymentProcessor.js:42:15)\n    ..."
}

Analysis of this example:

• Timestamp in ISO-8601 format: easy to analyze
• Level in string: explicit (error rather than 0 or 10)
• Message in structured JSON with relevant information (service name, message, fields separated from the message)
• correlationId and build information (commitHash, nodeVersion)
• Source information (file name, line number, function name)
• Stack trace for error log

2.12 Demo — Logging in Express with Winston and Morgan

In this demo we see an example of logging in a Node.js microservice using Express, Morgan and Winston.

Installing Node.js with NVM

To install Node.js, two alternatives:

1. Go to the Node.js.org website and follow the instructions for your OS
2. (Recommended) Use NVM (Node Version Manager) which allows you to use multiple versions of Node.js simultaneously

For Linux/macOS users:

# Installer NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash

# Vérifier l'installation
nvm --version

# Lister les versions disponibles
nvm ls

# Installer une version spécifique
nvm install 20

# Utiliser une version
nvm use 20

For Windows, use nvm-windows (separate GitHub repository) or WSL.

Configuring the Winston logger

// module-2/04-express-logging/logger.js
import winston from 'winston'

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'info',
  defaultMeta: {
    service: 'express-logging-example',
    buildInfo: {
      nodeVersion: process.version,
      commitHash: process.env.COMMIT_HASH || 'local',
      version: process.env.VERSION || '1.0.0'
    }
  },
  transports: [
    new winston.transports.Console({
      format: winston.format.combine(
        winston.format.colorize(),
        winston.format.timestamp(),
        winston.format.printf(({ message, timestamp, level, service, buildInfo, ...others }) => {
          // Ignore service and buildInfo when logging to the console
          return `${timestamp} ${level}: ${message} ${JSON.stringify(others)}`
        })
      )
    }),
    new winston.transports.File({
      filename: 'log.ndjson',
      level: 'info',
      format: winston.format.combine(
        winston.format.timestamp(),
        winston.format.json()
      )
    })
  ]
})

export default logger

Morgan Middleware for Express

// module-2/04-express-logging/morganMiddleware.js
import morgan from 'morgan'
import logger from './logger.js'

const morganFormat = `
{
    "method": ":method",
    "url": ":url",
    "status": ":status",
    "response-time": ":response-time"
}`

function messageHandler (message) {
  logger.info('Request received', JSON.parse(message.trim()))
}

const morganMiddleware = morgan(
  morganFormat,
  {
    stream: { write: messageHandler }
  }
)

export default morganMiddleware

Complete Express Application

// module-2/04-express-logging/index.js
import express from 'express'
import morganMiddleware from './morganMiddleware.js'
import logger from './logger.js'
import cors from 'cors'
import bodyParser from 'body-parser'

const PORT = process.env.PORT || 3000
const app = express()
app.use(cors())
app.use(bodyParser.urlencoded({extended:true}))
/* Use the morgan middleware to log requests. Needs to go before the routes */
app.use(morganMiddleware)

app.post('/test-error', (req, res) => {
  logger.info(JSON.stringify(req.body))
  res.status(500).send('some problem')
})

app.post('/test', (req, res) => {
  logger.info(req)
  res.status(200).send('all good')
})

app.get('/', (req, res) => {
  const response = 'Hello World!'
  logger.debug('Sending message', { response })
  res.status(200).send(response)
})

app.get('/error', (req, res) => {
  throw new Error('Error route triggered!')
})

/* Error handler. Needs to go before app.listen to catch all errors */
app.use((err, req, res, next) => {
  logger.error(err.message, { error: err, stackTrace: err.stack })
  res.status(500).send('Something failed.')
})

app.listen(PORT, () => {
  logger.info('Server started', { port: PORT })
})

Winston example with custom colors and formats

// module-2/02-logging-with-winston/index.js
import winston from 'winston'

/*
 * Constants for Select Graphics Rendition (SGR) colors in the terminal
 */
const SGR_COLORS = {
  green: 32,
  yellow: 33,
  blue: 34,
  magenta: 35
}

const ESC = '\x1B'
const CSI = '['
const generateSGR = (sgrEffects) => `${ESC}${CSI}${sgrEffects.join(';')}m`
const SGR_RESET = 0
const colorize = (text, color) => generateSGR([SGR_COLORS[color]]) + text + generateSGR([SGR_RESET])

/* Custom format for console logs */
const myFormat = winston.format.printf(({ service, message, buildInfo, timestamp, level }) => {
  const timestampColor = colorize(timestamp, 'green')
  const serviceColor = colorize(service, 'magenta')
  const versionColor = colorize(buildInfo.version, 'blue')
  const nodeVersionColor = colorize(`node${buildInfo.nodeVersion}`, 'yellow')

  return `${timestampColor} [${serviceColor}-${versionColor}-${nodeVersionColor}] ${level}: ${message}`
})

const logger = winston.createLogger({
  level: process.env.LOG_LEVEL || 'silly',
  format: winston.format.combine(
    winston.format.colorize(),
    winston.format.timestamp(),
    myFormat
  ),
  defaultMeta: {
    service: 'winston-demo',
    buildInfo: {
      nodeVersion: process.version,
      version: '1.0.0'
    }
  },
  transports: [
    new winston.transports.Console(),
    new winston.transports.File({
      filename: 'error.log',
      level: 'error',
      format: winston.format.combine(
        winston.format.timestamp(),
        winston.format.uncolorize(),
        winston.format.json()
      )
    })
  ]
})

logger.error('error message')
logger.warn('warn message')
logger.info('info message')
logger.http('http message')
logger.verbose('verbose message')
logger.debug('debug message')
logger.silly('silly message')

3. Centralized logging with the ELK stack

3.1 Module Introduction

In the previous module, we laid the foundations for effective monitoring of microservices by generating logs from our applications. Now it’s time to take the next step: centralized logging with the powerful ELK stack.

Module plan:

1. Overview of the ELK stack and its main components
2. Using Docker to deploy these tools locally
3. Configuration of the ELK stack locally and integration with a Node.js application
4. Exploring the three phases: ingestion with Logstash, storage with Elasticsearch, visualization with Kibana
5. Good practices with these tools

3.2 The ELK stack

The ELK stack is a set of tools for centralizing and managing logs. It is made up of three main components:

Application Node.js
       |
       v
  [Logstash] -----> [Elasticsearch] <-----> [Kibana]
  (Ingestion)        (Stockage &               (Visualisation)
                      Recherche)

Note: Although we primarily explore these three tools, the ecosystem has grown to incorporate other tools, which is why it is now officially called the Elastic Stack, although the acronym ELK is still widely used in the industry.

Advantages of the ELK stack

3.3 Docker and Docker Compose

Why Docker for the ELK stack?

Ease of setup: Docker greatly simplifies the process of installing and configuring Elasticsearch, Logstash and Kibana. Pre-built images on Docker Compose streamline the creation of a ready-to-use environment.

Manageability: Docker provides a structured way to manage dependencies and configurations between these interconnected components, making updates and maintenance easier. For example, you can avoid conflicts and run different versions of the same tools simultaneously for different projects.

Ease of cleanup: Docker allows you to easily delete containers and their associated data, keeping your machine organized and avoiding resource conflicts.

Docker Compose vs Kubernetes

Kubernetes offers advanced features like auto-scaling, self-healing, and complex orchestration ideal for large-scale production environments, but also introduces significant complexity.

Docker Compose provides a more streamlined approach for this course. It allows us to focus on the fundamental concepts of centralized logging with the ELK stack without the overhead of understanding the broad Kubernetes ecosystem.

Docker Compose uses a YAML configuration file to define all aspects of our services, from the images used to network settings and volume configurations.

3.4 Demo — Starting ELK with Docker Compose

Structure of the docker-compose.yml file

# module-3/elk-stack/docker-compose.yml
version: '3'

services:

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    ports:
      - "9200:9200"
    networks:
      - elk

  logstash:
    image: docker.elastic.co/logstash/logstash:8.12.0
    container_name: logstash
    volumes:
      - ./logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - ./logstash/logstash.yml:/usr/share/logstash/config/logstash.yml
    ports:
      - "5000:5000"
      - "5001:5001"
    depends_on:
      - elasticsearch
    networks:
      - elk

  kibana:
    image: docker.elastic.co/kibana/kibana:8.12.0
    container_name: kibana
    ports:
      - "5601:5601"
    networks:
      - elk

  filebeat:
    build:
      context: filebeat/
    environment:
      - strict.perms=false
    container_name: filebeat
    volumes:
      - ./microservice-logs:/var/log/microservice:ro
    depends_on:
      - logstash
    networks:
      - elk

networks:
  elk:
    driver: bridge

File structure:

• Version 3: indicates the version of Docker Compose used
• Services: YAML table including Elasticsearch, Logstash, Kibana and Filebeat
• Networks: section with the ELK network defined by the bridge driver

Docker Basic Commands

# Vérifier l'installation Docker
docker ps -a

# Tester Docker avec l'image hello-world
docker run hello-world

# Démarrer le stack ELK en mode détaché
docker compose up -d

# Vérifier l'état des conteneurs
docker ps -a

# Supprimer un conteneur
docker rm <container_id>

# Arrêter et supprimer tous les conteneurs
docker compose down

Access to interfaces

3.5 Log ingestion with Logstash

Now that our ELK stack is up and running, it’s time to send log data. Logstash is the versatile data pipeline that collects and transforms our logs before sending them to Elasticsearch.

Two approaches to log ingestion

1. Direct Transport

This approach involves configuring your application to send logs directly to Elasticsearch. Although simple in appearance and ideal for quick demonstrations, it is not ideal for a production environment because:

• Our applications need to know the details of Elasticsearch
• Need client library and clean configuration
• More difficult to change configuration, upgrade or change tools

2. Writing to files (File-based)

In this approach, the application writes to the local disk, and a separate process (Logstash or Filebeat) retrieves these files, parses the contents, and passes them to Elasticsearch. This approach:

• Requires more configuration, but is more robust for a production system
• Application does not need to know logging system details
• The system is more scalable and resilient to outages or bottlenecks

Logstash Pipeline

The Logstash pipeline is made up of three main sections:

[Input] --> [Filter] --> [Output]

# module-3/elk-stack/logstash/logstash.conf
input {
  tcp {
    port => 5000
  }
  beats {
    port => 5001
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "app-%{+YYYY.MM.dd}"
  }
  stdout { }
}

• Input section: defines the input plugin to collect logs (TCP on port 5000, Beats on port 5001)
• Filter section: transforms the data (here, parses the JSON of the message field)
• Output section: sends logs to Elasticsearch with a date-based index, and also to stdout

Logstash Configuration

# module-3/elk-stack/logstash/logstash.yml
http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]

Filebeat Configuration

# module-3/elk-stack/filebeat/filebeat.yml
filebeat.inputs:
  - type: filestream
    id: microservice
    fields_under_root: true
    encoding: utf-8
    fields:
      event.dataset: microservice
    paths:
      - /var/log/microservice/*.log

setup.kibana:
  host: "kibana:5601"
output.logstash:
  hosts: 'logstash:5001'

Filebeat is configured to:

• Read log files in /var/log/microservice/*.log
• Send logs to Logstash on port 5001

3.6 Demo — Logstash and Filebeat

Direct transport to Elasticsearch

// module-3/01-elasticsearch-transport/index.js
import winston from 'winston'
import { Client } from '@elastic/elasticsearch'
import { ElasticsearchTransport } from 'winston-elasticsearch'

const serviceName = 'elasticsearch-transport-example'
const ELASTICSEARCH_URL = 'http://localhost:9200'

const esTransportOpts = {
  level: 'info',
  client: new Client({ node: ELASTICSEARCH_URL }),
  indexPrefix: 'app',
  format: winston.format.combine(
    winston.format.timestamp(),
    winston.format.json()
  ),
  transformer: logData => {
    const { meta, timestamp, ...other } = logData
    return {
      timestamp,
      ...meta,
      ...other
    }
  }
}
const elasticsearchTransport = new ElasticsearchTransport(esTransportOpts)

const logger = winston.createLogger({
  defaultMeta: {
    service: serviceName,
    buildInfo: {
      nodeVersion: process.version
    }
  },
  transports: [
    elasticsearchTransport
  ]
})

logger.info('Info from elasticsearch transport')
logger.error('Error from elasticsearch transport')

/* Close the logger after a second to allow the logs to be sent */
setTimeout(() => {
  logger.close()
  console.log('Sent logs to elasticsearch')
}, 1000)

// module-3/01-elasticsearch-transport/package.json (dépendances clés)
{
  "dependencies": {
    "winston": "^3.x",
    "winston-elasticsearch": "^x.x",
    "@elastic/elasticsearch": "^8.x"
  }
}

Transport via Logstash

// module-3/02-logstash-transport/index.js
import winston from 'winston'
import LogstashTransport from 'winston-logstash/lib/winston-logstash-latest.js'

const serviceName = 'logstash-transport-example'

const logstashTransport = new LogstashTransport({
  host: 'localhost',
  port: 5000
})

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(winston.format.timestamp(), winston.format.json()),
  defaultMeta: {
    service: serviceName,
    buildInfo: {
      nodeVersion: process.version
    }
  },
  transports: [
    logstashTransport
  ]
})

logger.info('Info from logstash transport')
logger.error('Error from logstash transport')

/* Close the logger after a second to allow the logs to be sent */
setTimeout(() => {
  logger.close()
  console.log('Sent logs to logstash')
}, 1000)

Writing to files for Filebeat

// module-3/03-filebeat-example/index.js
import winston from 'winston'

const serviceName = 'filebeat-file-example'

const logger = winston.createLogger({
  level: 'info',
  format: winston.format.combine(winston.format.timestamp(), winston.format.json()),
  defaultMeta: {
    service: serviceName,
    buildInfo: {
      nodeVersion: process.version
    }
  },
  transports: [
    new winston.transports.File({ filename: '../elk-stack/microservice-logs/combined.log' })
  ]
})

logger.info('Info from file')
logger.error('Error from file')

setTimeout(() => {
  logger.close()
  console.log('Sent logs to file')
}, 1000)

In this approach:

• The application writes JSON logs to a file on the local disk
• Filebeat retrieves these files and sends them to Logstash
• Logstash transforms logs and sends them to Elasticsearch
• We then access Kibana on http://localhost:5601 to create a Data View and explore the logs

3.7 Log storage with Elasticsearch

Now that our logs are flowing into the ELK stack, let’s focus on Elasticsearch, the heart of our centralized logging solution.

Features of Elasticsearch

Distributed in nature: Elasticsearch is designed to span multiple nodes within a cluster. This makes it possible to manage vast amounts of data and distribute the workload, ensuring high availability and increased robustness.

Scalable: Elasticsearch scales seamlessly. You can easily add or remove nodes to adjust your cluster capacity as needed.

Designed for search and analysis: Elasticsearch excels at full-text search. It builds powerful indices that make finding relevant entries incredibly fast, even with massive datasets.

Key Features

3.8 Demo — Elasticsearch

Using Kibana DevTools to interact with Elasticsearch:

# Obtenir tous les documents
GET /_search

# Voir les indices existants
GET /_cat/indices

# Créer un index avec un mapping
PUT /globoticket-logs
{
  "mappings": {
    "properties": {
      "timestamp": { "type": "date" },
      "level": { "type": "keyword" },
      "service": { "type": "keyword" },
      "message": { "type": "text" }
    }
  }
}

# Créer un document
POST /globoticket-logs/_doc
{
  "message": "Hello World",
  "service": "hello-service"
}

# Recherche par texte
GET /globoticket-logs/_search
{
  "query": {
    "match": {
      "message": "Hello"
    }
  }
}

# Recherche floue (fuzzy search) - tolère les fautes de frappe
GET /globoticket-logs/_search
{
  "query": {
    "fuzzy": {
      "message": {
        "value": "Helo",
        "fuzziness": 1
      }
    }
  }
}

The mapping is the schema of an Elasticsearch index. Although Elasticsearch can work with flexible schemas, using mappings effectively ensures that your cluster’s performance is optimized.

3.9 Viewing logs with Kibana

Kibana is your window to the world of logs. Through interactive dashboards, visualizations and powerful search tools, it helps you understand your data.

Main strengths of Kibana

Visualizations: Transform your data into various charts and diagrams. Visualize trends, distribution patterns, identify anomalies and gain insights at a glance.

Dashboards: Create custom dashboards that combine multiple visualizations, providing a comprehensive view of your microservices. Share these boards with your team to encourage collaboration.

Real-time analysis: As logs flow into Elasticsearch, Kibana dashboards update in near real-time. This allows you to monitor critical metrics, spot emerging issues, and proactively respond to incidents.

3.10 Demo — Kibana

In this demo, we use Kibana to view our logs. We run a script simulating GloboTicket traffic:

# Installer les dépendances
npm install

# Démarrer et générer des logs
npm start

The script simulates GloboTicket’s microservices infrastructure with concurrent users:

• frontEnd: user navigation
• logIn: authentication service
• searchEvents: search for events
• getEvents: viewing events
• purchaseTickets: purchase tickets
• createOrders: creating orders
• processPayments: payment processing (asynchronous)
• sendNotifications: sending notifications

Workflow in Kibana

1. Navigate to the Discover section in Kibana
2. Create a new Data View with the app-* pattern (prefix configured in Logstash)
3. Set timestamp field
4. Explore logs with filters and visualizations
5. Create custom Dashboards with visualizations

3.11 ELK Tips and Best Practices

Index policy

• Use consistent structure formats for your logs
• Include contextual information
• Use time-based indices for efficient searches and optimal data management
• Plan your number of shards and their sizes strategically (too many shards can overload Elasticsearch; too few can create bottlenecks)
• Consider an index rollover strategy to manage storage and optimize queries

Scaling

Vertical: Increase hardware resources (RAM, CPU, storage). Monitor key metrics to identify when this is needed.

Horizontal: Add nodes to your cluster to distribute load and provide replication to improve robustness and availability.

Using Kibana

• Always work with the correct time ranges in Kibana for smooth visualizations and accurate searches
• Start with a targeted period of time related to the specific problem you are investigating
• Master Kibana’s filtering capabilities to focus on the most relevant logs
• Use Elasticsearch query language for precise and targeted filters
• Combine filters to isolate specific events or patterns of interest

Logging levels and performance

• Manage logging levels efficiently
• Avoid excessive logging in production environment
• Enable debug logs selectively when necessary

3.12 Demo — Debugging with ELK

Using a pre-configured Kibana dashboard to identify a bug in GloboTicket microservices.

Dashboard import

# Importer le dashboard Kibana via l'API
curl -X POST "http://localhost:5601/api/saved_objects/_import" \
  -H "kbn-xsrf: true" \
  -F file=@kibana-dashboard.ndjson

Debug scenario

Problem: Some users were charged but never received their tickets via email.

Debug workflow:

1. Open the Debugging Example Dashboard in Kibana
2. Copy problematic user ID to filter
3. Refine the filter on the specific userId field (to avoid partial matches with correlationId)
4. Observe logs with this userId — each correlationId corresponds to a different operation
5. Find the correlationId of the relevant purchase transaction
6. Filter by this correlationId to see all logs for this specific operation
7. Analyze the sequence of events in different services

The dashboard includes useful visualizations like:

• Top 5 values of services and URLs
• Distribution of logs by level
• Response time percentiles for different endpoints

4. Microservices Monitoring with Prometheus and Grafana

4.1 Module Introduction

In this module, we dive into the essential world of monitoring our microservices using Prometheus and Grafana.

Module plan:

1. Understand the fundamental differences between monitoring and logging
2. Presentation of Prometheus and Grafana
3. Installation and configuration with Docker and Docker Compose
4. Efficient instrumentation of Node.js microservices
5. Creating meaningful dashboards in Grafana
6. Configuring alerts to monitor potential issues
7. Introduction to OpenTelemetry and its growing role in modern observability

4.2 Understanding monitoring

Monitoring is the process of collecting and analyzing real-time data to track system health, performance, and identify potential issues.

What are metrics?

Metrics are numerical values ​​of different variables that provide information about our system:

By tracking these metrics, we can gain valuable insights into the performance of our microservices, detect bottlenecks, and proactively address issues.

Monitoring vs Logging

Key Point: Monitoring gives you a real-time pulse of your system, while logging provides the details for in-depth investigations.

Why monitoring is crucial for microservices?

Increased complexity: Microservices architectures involve multiple services, each of which can contribute to performance issues or failures. Without monitoring, identifying the responsible department becomes extremely difficult.

Faster Diagnosis of Problems: With metric-based alerts, you can detect anomalies as they occur, often before they impact end users. This allows for faster response to incidents.

Resource Optimization: Monitoring reveals patterns in resource usage, allowing you to optimize resource allocation and scale intelligently.

4.3 Prometheus and Grafana

Prometheus

Prometheus is an open source monitoring system and alerting toolkit, originally built at SoundCloud. It efficiently stores and organizes metrics over time using a time series database.

Key Features:

Pull vs push model:

• Push: applications actively send their metrics to a central system
• Pull (Prometheus): Prometheus will periodically fetch metrics from /metrics endpoints exposed by applications

The pull model is well suited to dynamic or ephemeral environments.

Grafana

Grafana is an open source visualization platform that allows you to create highly customizable dashboards tailored to analyzing your most important metrics:

• Various types of visualizations: graphs, heatmaps, single stats, tables, etc.
• Supports many data sources (Prometheus, InfluxDB, Elasticsearch, etc.)
• Central hub to monitor different parts of your system

Prometheus + Grafana Bundle

Together, Prometheus and Grafana constitute a robust toolkit for monitoring and visualizing the health and performance of your microservices:

Microservices Node.js (/metrics)
          |
    [Prometheus scrape]
          |
    [Prometheus TSDB]
          |
      [Grafana]
    (Visualisation, Alerting)
          |
    [Alertmanager]
          |
  [Slack, PagerDuty, Webhook...]

4.4 Installation and configuration

docker-compose.yml file

# module-4/prometheus-grafana/docker-compose.yml
version: '3.7'
services:
  prometheus:
    image: prom/prometheus:v2.50.1
    container_name: prometheus
    volumes:
      - ./prometheus:/etc/prometheus:ro
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--storage.tsdb.retention.time=200h'
      - '--web.enable-lifecycle'
    ports:
      - 9090:9090
    extra_hosts:
    - "host.docker.internal:host-gateway"

  grafana:
    image: grafana/grafana:10.2.4
    container_name: grafana
    volumes:
      - ./datasource.yml:/etc/grafana/provisioning/datasources/datasource.yml:ro
      - ./dashboards:/etc/grafana/provisioning/dashboards:ro
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - 3000:3000
    extra_hosts:
    - "host.docker.internal:host-gateway"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    container_name: alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
    command:
      - '--config.file=/etc/alertmanager/alertmanager.yml'
    ports:
      - 9093:9093
    extra_hosts:
    - "host.docker.internal:host-gateway"

  webhook:
    image: coveros/webhook-tester:2.0.0
    container_name: webhook
    ports:
      - 8080:8080

Note: The webhook service is a container for testing that the webhook works without needing to configure a third-party service like Slack. The extra_hosts: host.docker.internal:host-gateway option allows containers to make requests to ports on the host machine, outside of the Docker network.

Prometheus configuration (prometheus.yml)

# module-4/prometheus-grafana/prometheus/prometheus.yml
global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'microservice'
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:8000']

rule_files:
    - '/etc/prometheus/alerting_rules.yml' 

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

• scrape_interval: frequency at which Prometheus collects metrics (here every 5 seconds)
• evaluation_interval: frequency of evaluation of alert rules
• scrape_configs: list of targets to scrape (our microservices on port 8000)
• rule_files: file containing alert rules
• alerting: configuration to send alerts to Alertmanager

Grafana Data Source Configuration

# module-4/prometheus-grafana/datasource.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    orgId: 1
    url: http://prometheus:9090
    basicAuth: false
    isDefault: true
    editable: true

4.5 Demo — Starting Prometheus and Grafana with Docker Compose

# Démarrer Prometheus, Grafana, Alertmanager et Webhook
docker compose up -d

# Vérifier que tout fonctionne
docker ps -a

Access to interfaces

4.6 Instrumentation of Node.js microservices

Instrumentation refers to the process of adding code or tools to our software application to collect and monitor data about its behavior, performance and overall health.

For us this means:

1. Add relevant client libraries
2. Expose a metrics endpoint that Prometheus can scrape

Installation

npm install prom-client

Metric Types

4.7 Demo — Instrumentation of a Node.js application

// module-4/01-metrics/index.js
import express from 'express'
import morgan from 'morgan'
import { register, collectDefaultMetrics, Counter, Histogram } from 'prom-client'

const PORT = process.env.PORT || 8000
const app = express()
app.use(morgan('combined'))

/* Collect some default Node.js metrics */
collectDefaultMetrics()

/* Define a counter to track the number of events processed */
const eventsProcessedCounter = new Counter({
  name: 'globoticket_events_processed_total',
  help: 'Total number of events processed by the Events service'
})
register.registerMetric(eventsProcessedCounter)

/* Define a histogram to track the duration of event processing */
const eventProcessingDuration = new Histogram({
  name: 'globoticket_event_processing_duration_seconds',
  help: 'Distribution of event processing times in seconds',
  buckets: [0.01, 0.05, 0.1, 0.5, 1]
})
register.registerMetric(eventProcessingDuration)

/* Expose the metrics at the /metrics endpoint */
app.get('/metrics', async (req, res) => {
  try {
    res.set('Content-Type', register.contentType)
    const metrics = await register.metrics()
    res.send(metrics)
  } catch (error) {
    console.log('Error fetching metrics:', error)
    res.status(500).send('Error fetching metrics')
  }
})

/* Simulate processing an event */
app.get('/process-event', async (req, res) => {
  await processEvent()
  res.send('Event processed')
})

/* Helper function to simulate a delay */
async function sleep (ms) {
  return new Promise(resolve => setTimeout(resolve, ms))
}

/* Decorator to track the duration of event processing */
function metricsDecorator (f) {
  return async function (req, res) {
    const end = eventProcessingDuration.startTimer()
    try {
      const result = await f(req, res)
      eventsProcessedCounter.inc()
      return result
    } finally {
      end()
    }
  }
}

/* Function to simulate processing an event */
const processEvent = metricsDecorator(async () => {
  await sleep(Math.random() * 1000)
  console.log('Processed event')
})

app.listen(PORT, '0.0.0.0', () => {
  console.log(`Server listening on port ${PORT}`)
})

Important points:

• collectDefaultMetrics(): automatically collect basic Node.js metrics (memory usage, event loop, etc.)
• Counter eventsProcessedCounter: tracks the total number of events processed
• Histogram eventProcessingDuration: measures the distribution of processing latencies
• register.registerMetric(): registers metrics in the registry
• The /metrics endpoint exposes all metrics in Prometheus format (text-based)
• The decorator pattern is useful for instrumenting functions without modifying their internal logic
• app.listen(PORT, '0.0.0.0', ...): necessary to accept requests coming from Prometheus from the Docker network

4.8 Creating Grafana Dashboards

Fundamentals

Focus on key metrics: Don’t try to put all metrics in one dashboard. Target specific audiences (developers, ops teams, business stakeholders) and adapt dashboards to their needs.

Use clear visualizations: Select visualizations that effectively convey the nature of the data:

• Time series: excellent for trends
• Single stat: highlights current values
• Tables: good for in-depth details

Hierarchical Design: Organize your metrics into panels, rows, and layouts to group related metrics together and make them easier to understand.

Contextual information: Provide clear labels, units and titles. Make your dashboards self-explanatory.

Configure alerts: Proactive monitoring means setting thresholds and configuring notifications (PagerDuty, Slack) to be alerted when critical metrics exceed specific values.

4.9 Demo — Grafana

Traffic simulation

# Générer du trafic continu vers le microservice
while true; do curl http://localhost:8000/process-event; done

Useful PromQL queries in Grafana

# Voir le taux d'événements traités (événements/seconde)
rate(globoticket_events_processed_total[5m])

# Voir la valeur brute du compteur (toujours croissant)
globoticket_events_processed_total

# Voir le 95e percentile des durées de traitement
histogram_quantile(0.95, globoticket_event_processing_duration_seconds_bucket)

# Voir les buckets de l'histogramme
globoticket_event_processing_duration_seconds_bucket

Note: The Counter is monotonically increasing, so we use rate() to see how it evolves over time. The histogram_quantile(0.95, ...) function allows you to see that 95% of requests are processed in less than a certain duration.

Importing a pre-configured dashboard

In Grafana, via the Import option, you can import a dashboard from a JSON file or from Grafana.com.

4.10 Configuring alerts

Alert flow

Prometheus (évalue PromQL) --> Alert fired --> Alertmanager
                                                     |
                          +------------------+-------+--------+
                          |                  |                |
                        Slack           PagerDuty         Webhook

Prometheus evaluates metrics with PromQL. If an alert rule condition is met, Prometheus sends an alert to Alertmanager.

Alertmanager manages:

• Grouping alerts
• Silencing notifications
• Routing alerts to the right teams

Prometheus alert rules

# module-4/prometheus-grafana/prometheus/alerting_rules.yml
groups:
- name: globoticket_alerts
  rules:
  - alert: LowEventProcessingRate
    expr: rate(globoticket_events_processed_total[1m]) < 0.5
    for: 10s  # Alert if condition persists for 10 seconds
    labels:
      severity: warning
    annotations:
      summary: "Globoticket event processing rate is too low"
      description: "Event processing rate has fallen below 0.5 events per second for the past 10 seconds. Investigate potential slowdown or bottleneck."

• expr: PromQL expression evaluated
• for: length of time the condition must persist before triggering the alert
• labels: metadata to categorize the alert
• annotations: additional information for notifications

Alertmanager configuration

# module-4/prometheus-grafana/alertmanager.yml
global:
  resolve_timeout: 5m 

route:
  group_by: ['service_name'] 
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 30m
  receiver: 'demo-webhook'

receivers:
- name: 'demo-webhook'
  webhook_configs:
    - url: 'http://webhook:8080/json-hook'

• group_by: groups alerts by service name
• group_wait: waiting time before sending first notification
• group_interval: delay between notifications for the same group
• repeat_interval: delay before repeating an unresolved alert
• receivers: list of notification destinations

Common Alertmanager integrations

Important: Design alert rules carefully to avoid alert noise and fatigue. Test your alert configuration thoroughly.

4.11 Demo — Alerts

Demo scenario

1. The loop that makes requests to our application is running
2. In the Prometheus interface → Alerts section, we see our LowEventProcessingRate alert in green (the requests are arriving)
3. We simulate a failure by stopping the request loop
4. After ~10 seconds, the alert enters FIRING state in Prometheus
5. Alertmanager receives the alert and sends a request to the test webhook
6. The webhook container displays the received notification

4.12 OpenTelemetry — Introduction

OpenTelemetry is a vendor-neutral standard for collecting metrics, traces and logs. It is quickly becoming the de facto standard for collecting telemetry data.

Advantages

We will see a practical example of OpenTelemetry in the last module of this course.

5. Distributed tracing with Jaeger and Zipkin

5.1 Module Introduction

In this module, we dive into the world of distributed tracing. By the end, you will have a solid understanding of using Jaeger and Zipkin to gain deep visibility into your microservices architectures.

Module plan:

1. Why traditional logging approaches are insufficient in microservices
2. Distributed Tracing Fundamentals
3. Presentation of Jaeger and Zipkin
4. Practical side: instrumentation of Node.js code and configuration of Jaeger and Zipkin
5. Context propagation
6. Demo with GloboTicket

5.2 Why is logging alone not enough?

As we built our microservices architectures, we likely used logging to understand system behavior. Unfortunately, this approach reaches its limits in a distributed world:

Logging limitations in microservices

The solution: distributed tracing

Distributed tracing offers a solution specifically tailored to microservices challenges. It’s like following the journey of a query through our systems, being able to see all the details of the services and operations involved with timing information and other useful metadata.

Fundamental concepts

Spans are connected, giving us a complete view of the full path of a request. This is similar to the correlation IDs used for logs, but more powerful because spans automatically include a lot of information. Additionally, these tracing tools offer visualizations and timing data, allowing us to identify slow operations or uncover bottlenecks.

Jaeger vs Zipkin

Jaeger and Zipkin are both open source distributed tracing systems used to monitor and troubleshoot microservices.

Both integrate with various programming languages ​​and frameworks, and both offer similar functionality.

5.3 Demo — Deploy Jaeger and Zipkin with Docker Compose

# module-5/jaeger-zipkin/docker-compose.yml
version: '3.7'

services:
  zipkin:
    container_name: zipkin
    image: openzipkin/zipkin:3.1
    ports:
      - 9411:9411

  jaeger:
    image: jaegertracing/all-in-one:1.54.0
    container_name: jaeger
    ports:
      - 6832:6832/udp
      - 4318:4318
      - 16686:16686

  prometheus:
    image: prom/prometheus:v2.50.1
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports:
      - 9090:9090
    extra_hosts:
    - "host.docker.internal:host-gateway"

Prometheus Configuration for OpenTelemetry:

# module-5/jaeger-zipkin/prometheus.yml
global:
  scrape_interval:     5s
  evaluation_interval: 5s

scrape_configs:
  - job_name: 'microservice'
    scrape_interval: 5s
    static_configs:
      - targets: 
        - 'host.docker.internal:9464'
        - 'host.docker.internal:9465'

# Démarrer Zipkin et Jaeger
docker compose up -d

# Accéder aux interfaces
# Zipkin: http://localhost:9411
# Jaeger: http://localhost:16686
# Prometheus: http://localhost:9090

Important ports:

5.4 Context Propagation

Context propagation is the crucial mechanism that allows distributed tracing to work.

Think of it as a relay race in a microservices system. For a distributed trace to depict the complete story of a request, there needs to be a way to pass the baton between services.

This stick is the trace context: a unique identifier for the trace and other metadata. This allows us to link spans into a single trace and, for example, correlate all operations that occur as a result of a single user interaction.

How it works

1. The first service receives a request and creates a new span with a unique traceId and spanId
2. When this service calls another service, it injects the trace context into the HTTP headers
3. The following service extracts the trace context from the headers and creates a new child span
4. This process repeats until the query is complete

Service A                    Service B
    |                            |
    |-- Span A1 (root) --------> |
    |   traceId: abc             |-- Span B1 (child de A1)
    |   spanId: 001              |   traceId: abc
    |                            |   spanId: 002
    |  HTTP Request avec         |   parentSpanId: 001
    |  headers: X-B3-TraceId,    |
    |  X-B3-SpanId               |
    |<---------------------------|
    |   Span A2 (child de A1)    |
    |   traceId: abc             |
    |   spanId: 003              |

Each service receiving a request must recognize this context, extract it, and include it transparently when making its own calls to downstream services. This can be handled automatically by the tools, but we usually need to add additional instrumentation or wrap the functions used to make queries to ensure these headers are included.

5.5 Demo — Tracing with Zipkin

Node.js microservices instrumentation with Zipkin. We simulate two microservices (Service A and Service B) which talk to each other.

Common code (server.js)

// module-5/01-zipkin-example/server.js
import { Tracer, BatchRecorder, jsonEncoder } from 'zipkin'
import { HttpLogger } from 'zipkin-transport-http'
import { expressMiddleware } from 'zipkin-instrumentation-express'
import CLSContext from 'zipkin-context-cls'
import wrapFetch from 'zipkin-instrumentation-fetch'
import express from 'express'
import morgan from 'morgan'

const ZIPKIN_URL = process.env.ZIPKIN_URL ?? 'http://localhost:9411'

const recorder = new BatchRecorder({
  logger: new HttpLogger({
    endpoint: `${ZIPKIN_URL}/api/v2/spans`,
    jsonEncoder: jsonEncoder.JSON_V2
  })
})

export default function getApp (localServiceName) {
  // CLSContext gère la propagation du contexte sans passer le contexte manuellement
  const ctxImpl = new CLSContext(localServiceName)
  const tracer = new Tracer({ ctxImpl, recorder, localServiceName })

  // Envelopper fetch pour propager automatiquement les headers de trace
  const zipkinFetch = wrapFetch(fetch, { tracer, remoteServiceName: '' })

  const app = express()
  app.use(morgan('combined'))
  app.use((req, res, next) => {
    console.log(req.headers)
    next()
  })

  // Middleware Zipkin pour Express - extrait le contexte des headers entrants
  app.use(expressMiddleware({ tracer }))

  function logToTrace (message) {
    ctxImpl.scoped(() => {
      tracer.recordMessage(message)
    })
  }

  async function callWithTraceFromRequest (req, name, fn) {
    return await tracer.letId(req._trace_id, async () => {
      return await tracer.local(name, async () => {
        return await fn()
      })
    })
  }

  return { app, zipkinFetch, tracer, logToTrace, callWithTraceFromRequest }
}

Service A

// module-5/01-zipkin-example/service-A.js
import getApp from './server.js'

const PORT = process.env.PORT || 8000
const CLIENT_PORT = process.env.CLIENT_PORT || 8001
const { app, zipkinFetch, logToTrace, callWithTraceFromRequest } = getApp('zipkin-example-A')

app.get('/a', async (req, res) => {
  // Appel au Service B avec propagation automatique du contexte
  const response = await zipkinFetch(`http://localhost:${CLIENT_PORT}/b`)
  logToTrace(`Received response from B: ${await response.text()}`)

  const processOutput = () => {
    const output = doSomething()
    logToTrace(`Processed output: ${output}`)
  }
  
  // Création de spans enfants pour chaque opération
  await callWithTraceFromRequest(req, 'process-output', processOutput)
  await callWithTraceFromRequest(req, 'process-output', processOutput)
  res.send('Hello from A')
})

function doSomething () {
  const iterations = Math.floor(100_000_000 * Math.random())
  for (let i = 0; i < iterations; i++) {
    // do nothing
  }
  return iterations
}

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`)
})

Service B

// module-5/01-zipkin-example/service-B.js
import getApp from './server.js'

const PORT = process.env.PORT || 8001
const { app, callWithTraceFromRequest } = getApp('zipkin-example-B')

app.get('/b', async (req, res) => {
  // Exécution de plusieurs opérations en parallèle
  const promises = []
  for (let i = 0; i < 3; i++) {
    promises.push(
      callWithTraceFromRequest(req, `call-database-${i}`, callDatabase)
    )
  }
  await Promise.all(promises)
  res.send('Hello from B')
})

function callDatabase () {
  const iterations = Math.floor(100_000_000 * Math.random())
  for (let i = 0; i < iterations; i++) {
    // do nothing
  }
}

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`)
})

Starting services

# Dans package.json, deux scripts:
# npm run start:A  -> démarre le Service A sur port 8000
# npm run start:B  -> démarre le Service B sur port 8001

# Terminal 1
npm run start:B

# Terminal 2
npm run start:A

# Faire une requête
curl http://localhost:8000/a

# Visualiser dans l'interface Zipkin
# http://localhost:9411

In the Zipkin interface we can see:

• The complete trace with all spans
• Timing information for each service and operation
• Parallel spans for Service B (call-database-0, call-database-1, call-database-2)

5.6 Demo — Tracing with Jaeger

Node.js microservices instrumentation with Jaeger and OpenTracing.

Jaeger tracer creator

// module-5/02-jaeger-example/jaegerTracer.js
import jaegerClient from 'jaeger-client'

const { JAEGER_AGENT_HOST, JAEGER_AGENT_PORT } = process.env

const config = {
  sampler: {
    type: 'const',
    param: 1
  },
  reporter: {
    logSpans: true,
    agentHost: JAEGER_AGENT_HOST ?? 'localhost',
    agentPort: parseInt(JAEGER_AGENT_PORT ?? '6832')
  }
}

function createJaegerTracer (serviceName, logger) {
  const options = {
    logger: {
      info: (message) => {
        console.info(`JAEGER: INFO: ${message}`)
      },
      error: (message) => {
        console.error(`JAEGER: ERROR: ${message}`)
      }
    }
  }

  return jaegerClient.initTracer({ ...config, serviceName }, options)
}

export default createJaegerTracer

• sampler.type: 'const' with param: 1: samples 100% of traces (in production you would use a lower sampling rate)
• reporter.agentHost/Port: address of the Jaeger agent to receive spans

Jaeger Middleware for Express

// module-5/02-jaeger-example/jaegerMiddleware.js
import { FORMAT_HTTP_HEADERS, Tags } from 'opentracing'

export const jaegerMiddleware = (jaegerTracer) => (req, res, next) => {
  // Extraire le contexte de trace depuis les headers HTTP entrants
  const extractedSpan = jaegerTracer.extract(FORMAT_HTTP_HEADERS, req.headers)
  
  // Créer un span enfant ou un span racine
  const span = jaegerTracer.startSpan(req.path, {
    childOf: extractedSpan,
    tags: {
      [Tags.HTTP_METHOD]: req.method,
      [Tags.HTTP_URL]: req.url,
      [Tags.PEER_HOSTNAME]: req.hostname,
      [Tags.PEER_PORT]: req.socket.remotePort,
      [Tags.PEER_SERVICE]: req.hostname,
      [Tags.COMPONENT]: 'express'
    }
  })

  req.span = span

  res.on('finish', () => {
    if (res.statusCode >= 500) {
      span.setTag(Tags.ERROR, true)
      span.log({ 'error.message': res.statusMessage })
    }
    span.setTag(Tags.HTTP_STATUS_CODE, res.statusCode)
    span.finish()
  })

  next()
}

Common server.js code (Jaeger)

// module-5/02-jaeger-example/server.js
import { jaegerMiddleware } from './jaegerMiddleware.js'
import createJaegerTracer from './jaegerTracer.js'
import { Span, FORMAT_HTTP_HEADERS } from 'opentracing'
import express from 'express'
import morgan from 'morgan'

export default function getApp (serverName) {
  const tracer = createJaegerTracer(serverName)

  // Wrapper fetch pour injecter manuellement les headers de trace
  const jaegerFetch = async (req, url) => {
    const jaegerHeaders = {}
    tracer.inject(req.span.context() ?? new Span(), FORMAT_HTTP_HEADERS, jaegerHeaders)
    return await fetch(url, { headers: jaegerHeaders })
  }

  const app = express()
  app.use(morgan('combined'))
  app.use((req, res, next) => {
    console.log(req.headers)
    next()
  })

  app.use(jaegerMiddleware(tracer))

  return { app, jaegerFetch, tracer }
}

Difference with Zipkin: Jaeger does not provide fetch instrumentation by default, so we must manually create jaegerFetch which injects the trace headers.

Service A (Jaeger)

// module-5/02-jaeger-example/service-A.js
import getApp from './server.js'

const PORT = process.env.PORT || 8000
const CLIENT_PORT = process.env.CLIENT_PORT || 8001
const { app, jaegerFetch, tracer } = getApp('jaeger-example-A')

app.get('/a', async (req, res) => {
  const response = await jaegerFetch(req, `http://localhost:${CLIENT_PORT}/b`)
  req.span.log({ message: `Received response from B: ${await response.text()}` })

  // Création manuelle de spans enfants
  let childSpan = tracer.startSpan('process-output', { childOf: req.span })
  let output = doSomething()
  childSpan.finish()

  req.span.log({ message: `Processed output: ${output}` })

  childSpan = tracer.startSpan('process-output', { childOf: req.span })
  output = doSomething()
  childSpan.finish()

  res.send('Hello from A')
})

function doSomething () {
  const iterations = Math.floor(100_000_000 * Math.random())
  for (let i = 0; i < iterations; i++) {
    // do nothing
  }
  return iterations
}

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`)
})

Service B (Jaeger)

// module-5/02-jaeger-example/service-B.js
import getApp from './server.js'

const PORT = process.env.PORT || 8001
const { app, tracer } = getApp('jaeger-example-B')

app.get('/b', async (req, res) => {
  // Opérations parallèles avec spans enfants
  const promises = []
  for (let i = 0; i < 3; i++) {
    promises.push(new Promise((resolve) => {
      const childSpan = tracer.startSpan(`call-database-${i}`, { childOf: req.span })
      setImmediate(() => {
        callDatabase()
        resolve()
        childSpan.finish()
      })
    }))
  }
  await Promise.all(promises)
  res.send('Hello from B')
})

function callDatabase () {
  const iterations = Math.floor(100_000_000 * Math.random())
  for (let i = 0; i < iterations; i++) {
    // do nothing
  }
}

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`)
})

In the Jaeger interface (http://localhost:16686), you can see the complete trace with the spans for each service and operation.

5.7 Demo — Tracing and monitoring with OpenTelemetry

In this demo, we instrument Node.js microservices with OpenTelemetry, which allows using multiple exporters (Zipkin, Jaeger, Console, Prometheus) depending on the configuration.

package.json dependencies (OpenTelemetry)

{
  "scripts": {
    "start:A": "node --import ./tracing.js service-A.js",
    "start:A:zipkin": "EXPORTER=zipkin node --import ./tracing.js service-A.js",
    "start:A:jaeger": "EXPORTER=jaeger node --import ./tracing.js service-A.js",
    "start:B": "node --import ./tracing.js service-B.js",
    "start:B:zipkin": "EXPORTER=zipkin node --import ./tracing.js service-B.js",
    "start:B:jaeger": "EXPORTER=jaeger node --import ./tracing.js service-B.js"
  },
  "dependencies": {
    "@opentelemetry/api": "...",
    "@opentelemetry/auto-instrumentations-node": "...",
    "@opentelemetry/exporter-prometheus": "...",
    "@opentelemetry/exporter-trace-otlp-http": "...",
    "@opentelemetry/exporter-zipkin": "...",
    "@opentelemetry/instrumentation-winston": "...",
    "@opentelemetry/sdk-node": "...",
    "@opentelemetry/sdk-trace-base": "...",
    "@opentelemetry/resources": "...",
    "@opentelemetry/semantic-conventions": "...",
    "express": "...",
    "morgan": "...",
    "winston": "..."
  }
}

Tracing configuration file (tracing.js)

// module-5/03-open-telemetry/tracing.js
import { NodeSDK } from '@opentelemetry/sdk-node'
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node'
import { ConsoleSpanExporter } from '@opentelemetry/sdk-trace-base'
import { Resource } from '@opentelemetry/resources'
import {
  SEMRESATTRS_SERVICE_NAME
} from '@opentelemetry/semantic-conventions'
import { ZipkinExporter } from '@opentelemetry/exporter-zipkin'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { PrometheusExporter } from '@opentelemetry/exporter-prometheus'
import { WinstonInstrumentation } from '@opentelemetry/instrumentation-winston'

const serviceName = process.env.SERVICE_NAME ?? 'open-telemetry-A'

// Configurer les trois exporteurs possibles
const consoleExporter = new ConsoleSpanExporter()
const zipkinExporter = new ZipkinExporter({
  url: 'http://localhost:9411/api/v2/spans',
  serviceName
})
const otlpExporter = new OTLPTraceExporter({
  url: 'http://localhost:4318/v1/traces',
  headers: {}
})

const exporters = {
  console: consoleExporter,
  zipkin: zipkinExporter,
  jaeger: otlpExporter
}

// Sélectionner l'exporteur dynamiquement via la variable d'environnement
const traceExporter = exporters[process.env.EXPORTER] ?? consoleExporter

const sdk = new NodeSDK({
  resource: new Resource({
    [SEMRESATTRS_SERVICE_NAME]: serviceName
  }),
  traceExporter,
  instrumentations: [
    new WinstonInstrumentation(), // Ajoute trace_id aux logs Winston
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': {
        enabled: false
      }
    })
  ],
  metricReader: new PrometheusExporter({
    port: serviceName.endsWith('A') ? 9464 : 9465,
    host: '0.0.0.0'
  })
})

// Initialiser le SDK
sdk.start()

// Arrêt gracieux
process.on('SIGTERM', () => {
  sdk.shutdown()
    .then(() => console.log('Tracing terminated'))
    .catch((error) => console.log('Error terminating tracing', error))
    .finally(() => process.exit(0))
})

Important points:

• WinstonInstrumentation: automatically adds trace_id to Winston logs, allowing logs and traces to be correlated
• getNodeAutoInstrumentations(): automatically instrument HTTP, Express, etc.
• PrometheusExporter: exposes OpenTelemetry metrics in Prometheus format
• The exporter is dynamically selected via the EXPORT environment variable

Common server.js code (OpenTelemetry)

// module-5/03-open-telemetry/server.js
import http from 'http'
import { trace, metrics } from '@opentelemetry/api'
import express from 'express'
import morgan from 'morgan'
import winston from 'winston'

function makeRequest (url) {
  return new Promise((resolve, reject) => {
    http.get(url, (response) => {
      let data = ''
      response.on('data', (chunk) => {
        data += chunk
      })
      response.on('end', () => {
        resolve(data)
      })
    })
  })
}

export default function getApp (serverName) {
  const logger = winston.createLogger({
    level: 'info',
    format: winston.format.prettyPrint(),
    defaultMeta: { service: serverName },
    transports: [
      new winston.transports.Console()
    ]
  })

  const app = express()
  app.use(morgan('combined'))
  app.use((req, res, next) => {
    logger.info(req.headers)
    next()
  })

  // Obtenir le tracer et meter depuis l'API OpenTelemetry
  const tracer = trace.getTracer(serverName)
  const meter = metrics.getMeter(serverName)

  return { app, tracer, meter, makeRequest, logger }
}

Note: With OpenTelemetry, we use native http.get() rather than fetch because OpenTelemetry automatically instruments the Node.js http module to propagate the context.

Service A (OpenTelemetry)

// module-5/03-open-telemetry/service-A.js
import opentelemetry from '@opentelemetry/api'

import getApp from './server.js'

const PORT = process.env.PORT || 8000
const CLIENT_PORT = process.env.CLIENT_PORT || 8001

const { app, tracer, meter, makeRequest } = getApp('open-telemetry-A')

app.get('/a', async (req, res) => {
  const response = await makeRequest(`http://localhost:${CLIENT_PORT}/b`)
  
  // Ajouter un événement au span actif
  opentelemetry.trace.getActiveSpan().addEvent('Response received', { 
    message: 'Received response from B', response 
  })

  // Créer des spans enfants avec l'API fluide
  const processOutput = async (span) => {
    const output = doSomething()
    span.addEvent('Output processed', { message: `Processed output: ${output}` })
    span.end()
  }

  await tracer.startActiveSpan('process-output-1', processOutput)
  await tracer.startActiveSpan('process-output-2', processOutput)

  res.send('Hello from A')
})

// Créer des métriques avec le meter OpenTelemetry
const eventsCounter = meter.createCounter('service-A.event.count', {
  description: 'Number of events processed'
})

const histogram = meter.createHistogram('service-A.event.duration', {
  description: 'The duration of event processing',
  boundaries: [0.01, 0.05, 0.1, 0.5, 1, 2, 5]
})

function doSomething () {
  eventsCounter.add(1)
  const startTime = new Date().getTime()
  const iterations = Math.floor(100_000_000 * Math.random())
  for (let i = 0; i < iterations; i++) {
    // do nothing
  }
  histogram.record(new Date().getTime() - startTime)
  return iterations
}

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`)
})

Service B (OpenTelemetry)

// module-5/03-open-telemetry/service-B.js
import getApp from './server.js'

const PORT = process.env.PORT || 8001

const { app, tracer, meter } = getApp('open-telemetry-B')

app.get('/b', async (req, res) => {
  const promises = []
  for (let i = 0; i < 3; i++) {
    promises.push(new Promise((resolve) => {
      const childSpan = tracer.startSpan(`call-database-${i}`)
      setImmediate(() => {
        callDatabase()
        resolve()
        childSpan.end()
      })
    }))
  }
  await Promise.all(promises)
  res.send('Hello from B')
})

const databaseCounter = meter.createCounter('service-B.db.client.call.count', {
  description: 'Number of database calls'
})

function callDatabase () {
  databaseCounter.add(1)
  const iterations = Math.floor(100_000_000 * Math.random())
  for (let i = 0; i < iterations; i++) {
    // do nothing
  }
}

app.listen(PORT, () => {
  console.log(`Server listening on port ${PORT}`)
})

Advantages of OpenTelemetry over native Zipkin/Jaeger

5.8 Course Summary

We’ve covered a lot of content in this course. Here is a summary of the most important topics:

What we learned

Observability in Node.js microservices: We explored what observability means in the context of Node.js microservices and why it is crucial to have mechanisms in place to be able to observe your systems. This can save a significant amount of time and money and ensure that your production environments run smoothly.

Logging with Winston and Pino: We saw how to configure and use common libraries like Winston and Pino for logging in Node.js, and explored some best practices to keep in mind when logging, such as using logging levels, structured formats, and correlation IDs.

ELK Stack: We got our hands dirty and launched a local ELK cluster. With this, we successfully aggregated the logs from our microservices, and this allowed us to query and visualize the log data in Kibana. We saw a demo of how this could help us solve problems in a microservices environment.

Prometheus and Grafana: We used Prometheus and Grafana to monitor our microservices. We instrumented them with both the Prometheus client library and OpenTelemetry, and used Prometheus to scrape the metrics. We built a dashboard in Grafana that helped us quickly understand how our service was performing in real time, and we set up an alert with Alertmanager.

Distributed tracing: Finally, we explored the world of distributed tracing using both Jaeger and Zipkin to visualize traces that span multiple services and operations and understand where the bottlenecks are in our application.

GloboTicket as a case study: The best part is that we used GloboTicket as a case study, exploring all these tools hands-on with Docker Compose.

Recommendations

I highly recommend spending some time playing with the code samples and really getting to grips with these tools, as they can really enrich your experience when working with microservices.

Next steps

From here you should have all the knowledge required to learn about:

• Advanced features of the Elastic Stack
• Docker and Kubernetes
• Prometheus and Grafana (advanced features)
• Jaeger, Zipkin and OpenTelemetry (advanced features)

6. Code Files — Full Reference

Structure of practical exercises

materials/
├── package.json
├── .nvmrc
├── .gitignore
├── module-2/
│   ├── README.md
│   ├── 01-logging-with-pino/
│   │   ├── index.js           # Exemple Pino avec tous les niveaux
│   │   └── package.json
│   ├── 02-logging-with-winston/
│   │   ├── index.js           # Winston avec couleurs et format personnalisé
│   │   └── package.json
│   ├── 03-selective-logging/
│   │   ├── index.js           # Exemple PII - SafeUser vs UnsafeUser
│   │   └── package.json
│   └── 04-express-logging/
│       ├── index.js           # Application Express complète
│       ├── logger.js          # Configuration Winston
│       ├── morganMiddleware.js # Middleware Morgan
│       └── package.json
├── module-3/
│   ├── README.md
│   ├── 01-elasticsearch-transport/
│   │   ├── index.js           # Transport direct vers Elasticsearch
│   │   └── package.json
│   ├── 02-logstash-transport/
│   │   ├── index.js           # Transport via Logstash
│   │   └── package.json
│   ├── 03-filebeat-example/
│   │   ├── index.js           # Écriture dans fichier pour Filebeat
│   │   └── package.json
│   ├── 04-debugging-exercise/
│   │   ├── index.js           # Simulation du trafic GloboTicket
│   │   └── package.json
│   └── elk-stack/
│       ├── docker-compose.yml  # Stack ELK complet
│       ├── README.md
│       ├── filebeat/
│       │   └── filebeat.yml    # Configuration Filebeat
│       ├── logstash/
│       │   ├── logstash.conf   # Pipeline Logstash
│       │   └── logstash.yml    # Configuration Logstash
│       └── microservice-logs/  # Répertoire des fichiers de logs
├── module-4/
│   ├── README.md
│   ├── 01-metrics/
│   │   ├── index.js           # Microservice instrumenté avec prom-client
│   │   └── package.json
│   └── prometheus-grafana/
│       ├── docker-compose.yml  # Prometheus, Grafana, Alertmanager, Webhook
│       ├── datasource.yml      # Source de données Grafana
│       ├── alertmanager.yml    # Configuration Alertmanager
│       ├── README.md
│       ├── dashboards/
│       │   └── dashboard.json  # Dashboard Grafana pré-configuré
│       └── prometheus/
│           ├── prometheus.yml  # Configuration Prometheus
│           └── alerting_rules.yml # Règles d'alerte
└── module-5/
    ├── README.md
    ├── 01-zipkin-example/
    │   ├── server.js          # Code commun (tracer Zipkin)
    │   ├── service-A.js       # Service A
    │   ├── service-B.js       # Service B
    │   └── package.json
    ├── 02-jaeger-example/
    │   ├── server.js          # Code commun
    │   ├── jaegerTracer.js    # Créateur de tracer Jaeger
    │   ├── jaegerMiddleware.js # Middleware Express Jaeger
    │   ├── service-A.js       # Service A
    │   ├── service-B.js       # Service B
    │   └── package.json
    ├── 03-open-telemetry/
    │   ├── tracing.js         # Configuration OpenTelemetry SDK
    │   ├── server.js          # Code commun
    │   ├── service-A.js       # Service A avec métriques
    │   ├── service-B.js       # Service B avec métriques
    │   └── package.json
    └── jaeger-zipkin/
        ├── docker-compose.yml  # Zipkin, Jaeger, Prometheus
        ├── prometheus.yml      # Configuration Prometheus (ports OTel)
        └── README.md

Tool summary table

Useful commands

# Module 2 - Logging avec Pino
cd materials/module-2/01-logging-with-pino
npm install
LOG_LEVEL=debug node index.js

# Module 2 - Logging avec Winston
cd materials/module-2/02-logging-with-winston
npm install
LOG_LEVEL=info node index.js

# Module 2 - Express avec Winston et Morgan
cd materials/module-2/04-express-logging
npm install
node index.js

# Module 3 - Démarrer le stack ELK
cd materials/module-3/elk-stack
docker compose up -d
# Attendre que tous les services soient démarrés (~30 secondes)
# Kibana: http://localhost:5601

# Module 3 - Transport Elasticsearch
cd materials/module-3/01-elasticsearch-transport
npm install
node index.js

# Module 3 - Exercise de débogage
cd materials/module-3/04-debugging-exercise
npm install
node index.js
# Les logs sont dans materials/module-3/elk-stack/microservice-logs/debugging.log

# Module 4 - Démarrer Prometheus + Grafana
cd materials/module-4/prometheus-grafana
docker compose up -d
# Prometheus: http://localhost:9090
# Grafana: http://localhost:3000 (admin/admin)
# Alertmanager: http://localhost:9093

# Module 4 - Démarrer le microservice instrumenté
cd materials/module-4/01-metrics
npm install
node index.js
# Métriques: http://localhost:8000/metrics

# Module 5 - Démarrer Zipkin + Jaeger
cd materials/module-5/jaeger-zipkin
docker compose up -d
# Zipkin: http://localhost:9411
# Jaeger: http://localhost:16686

# Module 5 - Exemple Zipkin
cd materials/module-5/01-zipkin-example
npm install
npm run start:B  # Terminal 1
npm run start:A  # Terminal 2
curl http://localhost:8000/a

# Module 5 - Exemple Jaeger
cd materials/module-5/02-jaeger-example
npm install
npm run start:B  # Terminal 1
npm run start:A  # Terminal 2
curl http://localhost:8000/a

# Module 5 - Exemple OpenTelemetry (avec Zipkin)
cd materials/module-5/03-open-telemetry
npm install
npm run start:B:zipkin  # Terminal 1
npm run start:A:zipkin  # Terminal 2
curl http://localhost:8000/a

# Module 5 - Exemple OpenTelemetry (avec Jaeger)
EXPORTER=jaeger npm run start:B  # Terminal 1
EXPORTER=jaeger npm run start:A  # Terminal 2
curl http://localhost:8000/a

Documentation generated on 2026-06-20

Search Terms

node.js · microservices · monitoring · logging · apis · backend · full-stack · web · jaeger · grafana · configuration · docker · elk · opentelemetry · prometheus · service · tracing · compose · kibana · logstash · stack · zipkin · elasticsearch · express