Node.js Microservices Resilience and Fault Tolerance

Table of Contents

1. Course Overview
2. Introduction to Resilience and Fault Tolerance

• 2.1 Resilience vs. fault tolerance
• 2.2 Importance of resilience
• 2.3 Importance of Fault Tolerance
• 2.4 Historical examples of system failures
• 2.5 Demo — GloboTicket Application

3. Resilience and fault tolerance — Deepening

• 3.1 Causes of system failures
• 3.2 Extrinsic factors
• 3.3 Intrinsic factors
• 3.4 Impact of system failures
• 3.5 Prevent system failures
• 3.6 Identify and Mitigate Single Points of Failure (SPOF)
• 3.7 Redundancy vs. high availability
• 3.8 Role of load balancing
• 3.9 Demo — Load Balancing with Kubernetes

4. Retry, Timeout and Circuit Breakers

• 4.1 Transient Failures
• 4.2 Timeouts
• 4.3 Simple Retries
• 4.4 Exponential Backoff
• 4.5 Circuit Breaker
• 4.6 Demo — Timeouts
• 4.7 Demo — Retries and Exponential Backoff
• 4.8 Demo — Circuit Breaker

5. Partial failure management

• 5.1 Graceful Degradation
• 5.2 Fallback Pattern
• 5.3 Caching
• 5.4 Demo — Cache, Fallback and Graceful Degradation

6. Advanced techniques

• 6.1 Fixed Window Rate Limiting
• 6.2 Sliding Window Rate Limiting
• 6.3 Leaky Bucket
• 6.4 Bulkhead Pattern
• 6.5 Load Shedding
• 6.6 Demo — Load Shedding

7. Case studies and best practices

• 7.1 Netflix — Chaos Engineering
• 7.2 Google Cloud — High-performance private network
• 7.3 Amazon Web Services — Resilient Services

8. GloboTicket application file reference

1. Course Overview

“The best way to avoid system failures is not to build software.”

In a hyper-connected digital world where services exist in healthcare, finance and other industries, failures are inevitable. Outages not handled properly can cripple an entire service — like the recent outage that left several African countries without internet connectivity for a week.

This course covers the following key topics:

• Timeouts — limit the time to wait for a response
• Circuit Breakers — interrupt calls to a faulty service
• Rate Limiting — control query throughput
• Caching — cache data to make it available even in the event of an outage
• Graceful Degradation — gracefully degrade service when components fail

Prerequisites

This course assumes that you are already familiar with:

• Node.js — server-side JavaScript runtime
• Docker — application containerization
• Kubernetes — container orchestration

If you are new to any of these tools, it is recommended that you first complete the introductory courses available on Pluralsight.

2. Introduction to Resilience and Fault Tolerance

This module explores the importance of resilience and fault tolerance in microservices. It presents why these concepts are essential to maintaining the stability and reliability of a system, especially in a distributed environment like that of microservices.

2.1 Resilience vs. fault tolerance

Resilience and fault tolerance are two terms often used together, but they refer to distinct aspects of system design.

Resilience ZXQSEG0001: Resilience refers to the ability of a system to recover quickly after a failure. Resilient systems are proactive: they continually monitor potential failures and have mechanisms to respond quickly when a problem arises. A resilient system can, for example, automatically switch to a backup service if the primary service fails. : La résilience fait référence à la capacité d’un système à récupérer rapidement après une défaillance. Les systèmes résilients sont proactifs : ils surveillent en permanence les défaillances potentielles et disposent de mécanismes pour réagir rapidement lorsqu’un problème survient. Un système résilient peut, par exemple, basculer automatiquement vers un service de secours si le service principal tombe en panne.

Fault Tolerance ZXQSEG0001: Fault tolerance refers to the ability of a system to continue to function correctly, even when one or more components fail. Fault-tolerant systems are reactive: they are designed to maintain continuity of service, often through redundancy or backup mechanisms. For example, multiple servers running the same service allow a failed server to be replaced without interruption. : La tolérance aux pannes désigne la capacité d’un système à continuer de fonctionner correctement, même lorsqu’un ou plusieurs composants sont défaillants. Les systèmes tolérants aux pannes sont réactifs : ils sont conçus pour maintenir la continuité du service, souvent grâce à la redondance ou à des mécanismes de sauvegarde. Par exemple, plusieurs serveurs exécutant le même service permettent qu’un serveur défaillant soit remplacé sans interruption.

Key Differences

Practical example: A resilient system automatically reroutes traffic away from failing components to maintain system operation. A fault-tolerant system uses redundant servers running the same service, so that if one server goes down, the others continue to provide the service without any downtime.

2.2 Importance of resilience

Fault Isolation

Fault isolation is crucial in microservices to prevent the failure of one microservice from cascading through the entire system. When a failure occurs in a microservice, a resilient architecture ensures that the impact is contained within that microservice and that other services continue to function. By isolating faults, the reliability of the entire system is improved.

Adaptability to changes

Microservices architectures often involve frequent updates, deployments, and changes. Resilience ensures that the system can adapt to these changes seamlessly, without causing disruption or downtime. Changes can be made without taking down the entire system, minimizing disruption to users.

Improved user experience

Resilience directly impacts user experience by minimizing the impact of failures and providing fallback mechanisms during service interruptions. A resilient system that handles failures elegantly builds user confidence and satisfaction.

2.3 Importance of Fault Tolerance

Continuous Availability

Fault tolerance is essential to ensure that the system remains available and operational in the presence of outages or failures. Continuous availability is especially crucial for applications that require 24/7 accessibility.

Reduced downtime

Downtime can have significant financial and reputational consequences for organizations. Fault tolerance aims to minimize downtime by quickly detecting and mitigating failures, ensuring that systems remain operational.

Improved scalability

Fault-tolerant systems are often designed with scalability in mind. The ability to handle increased load or demand without succumbing to failure improves overall system scalability.

2.4 Historical examples of system failures

The Y2K bug (Year 2000 Problem)

The Y2K bug, also known as the Millennium Bug or Y2K problem, was a computer problem related to the formatting and storage of dates. Many computer systems and applications represented years using only the last two digits (example: 99 for 1999), creating the potential for confusion when changing to the year 2000. This problem could have led to errors, miscalculations, and system failures in various industries.

The resilience of systems depended on how well they could adapt to necessary changes. Organizations that proactively addressed the problem by updating their software, replacing outdated systems, and conducting extensive testing demonstrated resilience in the face of a potential crisis.

The Chernobyl nuclear disaster

The Chernobyl nuclear disaster, which occurred on April 25, 1986 at a nuclear power plant in Ukraine, provides an eloquent context for understanding the importance of resilience and fault tolerance in complex systems.

This disaster was caused by a combination of design flaws, operator errors, and a lack of fault tolerance inherent in the reactor design. The lack of sufficient security redundancies and fail-safe mechanisms played a significant role in the severity of the incident.

2.5 Demo — GloboTicket App

In this course, the demo application used is GloboTicket, owned by a fictitious company called Globomantics. This application illustrates how microservices handle various aspects of event management.

Architecture of GloboTicket

┌─────────────────────────────────────────────────────────────────────┐
│                           GloboTicket                               │
│                                                                     │
│  ┌──────────┐    ┌───────────────┐    ┌──────────────────────────┐  │
│  │  Web UI  │───▶│  API Gateway  │───▶│  Bookings Service        │  │
│  └──────────┘    └───────────────┘    │  (gestion des réservations)│  │
│                        │              └──────────────────────────┘  │
│                        │              ┌──────────────────────────┐  │
│                        └─────────────▶│  Catalog Service          │  │
│                                       │  (catalogue des événements)│  │
│                                       └──────────────────────────┘  │
│                                       ┌──────────────────────────┐  │
│                                       │  User Service             │  │
│                                       │  (authentification)       │  │
│                                       └──────────────────────────┘  │
└─────────────────────────────────────────────────────────────────────┘

• Web UI — Next.js interface that transmits requests to an API gateway
• Bookings Service — NestJS service managing all event bookings
• Catalog Service (Events) — NestJS service managing the events catalog
• User Service (Auth) — NestJS service managing user authentication

3. Resilience and fault tolerance — Deepening

This module delves deeper into resilience and fault tolerance, exploring system failures, single points of failure (SPOF), and load balancing.

3.1 Causes of system failures

Why do systems crash? The causes can be grouped into two categories:

• External (extrinsic) causes — elements originating outside the system, generally beyond the control of the organization
• Internal (intrinsic) causes — elements originating from within the system, directly influenced by the organization or individuals who manage it

3.2 Extrinsic factors

External causes of system failures include:

Power Outages

Power outages, whether short or prolonged, can cause system failures if sufficient backup power is not available. Critical systems may experience interruptions causing data loss and impacting ongoing operations. This highlights the importance of having robust backup power systems. Regular testing and maintenance of emergency power systems is crucial to ensure their reliability.

Hardware Failures

Hardware failures such as damaged hard drives or faulty RAM can lead to system crashes and data loss. Regular monitoring of hardware health, timely replacements, and redundancy strategies like RAID are essential to mitigate the impact of hardware failures.

Security Breaches

Security breaches, whether due to software vulnerabilities or unauthorized access, pose a significant threat. Cyberattacks, data breaches or unauthorized manipulation of system components can compromise sensitive information, disrupt services and harm an organization’s reputation. Physical security is also important: unauthorized access to data centers can lead to theft, vandalism or tampering.

Natural Disasters

Natural disasters like earthquakes or floods can cause physical damage to data centers, servers, and other critical infrastructure. This can lead to extended downtime and data loss. Implementing geographically dispersed data centers or cloud solutions with redundancy can help mitigate the impact of localized natural disasters. This is one of the reasons why cloud providers like AWS have data centers in many regions around the world.

3.3 Intrinsic factors

Internal causes of system failures include:

Software bugs and misconfigurations

Software systems are complex. Despite rigorous testing, bugs can still be found in the final version of software. These bugs can lead to unexpected errors and system crashes. Additionally, misconfigurations — where settings are set incorrectly — can also disrupt the intended operation of the software.

Insufficient resource allocation

Inadequate resource allocation is a common source of system failures. For example, attempting to run resource-intensive calculations (such as running a sophisticated language model) on hardware with limited processing power will result in slow performance, crashes, or even complete system failure.

Insufficient tests

The importance of thorough testing cannot be overstated. Incomplete or inadequate testing can miss critical scenarios, allowing bugs to go undetected. This can lead to unexpected behavior when software is deployed in real-world conditions.

Human Error

Human factors contribute significantly to system failures. Negligence or carelessness — such as not adhering to established procedures or skipping crucial steps in the development and deployment process — can introduce errors. These can range from simple oversights to more serious errors that compromise the stability and security of a system.

3.4 Impact of system failures

System failures have significant negative impacts on employees, business owners and organizations:

Downtime and loss of productivity

System failures often result in significant downtime, disrupting normal workflow and hindering productivity. When critical features or services are unavailable due to an outage, teams are forced to divert their attention from regular tasks to resolve the issue. This can lead to missed deadlines and increased stress for employees.

Reputation damage

In industries where reliability and adherence to service level agreements (SLAs) are paramount, system failures can seriously damage an organization’s reputation. Particularly in sectors like fintech, security or infrastructure, customers and stakeholders expect a high level of reliability. Prolonged or frequent outages can erode trust and cause lasting damage to the organization’s image.

Loss of customers and revenue

System failures can lead to loss of customers who, faced with frequent interruptions, may seek more reliable alternatives. This loss of customers translates directly into a drop in revenue. For businesses that rely heavily on online platforms or digital services, the financial impact can be substantial with long-term consequences.

Risk of data loss

Some system failures, particularly those related to database corruption, can pose significant challenges. Recovery from such incidents may require specialist expertise and may not always result in complete data restoration. The time gap between the last backup and the occurrence of the failure introduces a risk of data loss.

3.5 Prevent system failures

Disaster Recovery Plans

Develop a comprehensive disaster recovery plan that outlines step-by-step procedures for bringing systems back online after a disaster. This includes identifying critical systems, prioritizing their restoration, and assigning responsibilities to relevant teams or individuals.

Redundancy and failover mechanisms

As the popular adage goes, don’t put all your eggs in one basket. Implement redundancy and failover mechanisms to eliminate single points of failure — which may involve having backup services, using load balancing, and employing distributed architectures to ensure that if one component fails, another can seamlessly take over.

Regular employee training

When the people who run an organization aren’t aware of how to manage its systems, things can quickly get out of hand. Conducting regular training sessions to keep employees informed of the organization’s disaster recovery systems, protocols and procedures will ensure that they are well prepared to troubleshoot common problems and respond effectively to system failures.

System audits and regular updates

New updates are released for libraries, tools and software every day. Some include critical security fixes that, if ignored, can have cascading effects. Perform routine system audits to identify potential points of weakness, security gaps, or outdated components that could pose a risk.

3.6 Identify and mitigate single points of failure (SPOF)

A Single Point of Failure (SPOF) is a part of a system whose failure or malfunction can disrupt the operations of all associated systems in a microservice. The goal is to minimize or eliminate SPOFs to ensure that the failure of one component does not bring down the entire system.

Three-step process

Step 1 — Discovery and Analysis

It is essential to deeply understand the architecture and dependencies of microservices:

• Service Dependency Mapping — identify all services and map their dependencies, both internal and external
• Critical Path Analysis — determine sequences and dependencies that, if interrupted, could result in a system failure
• Failure Mode Analysis — identify potential failure scenarios at the infrastructure and application levels
• Load Testing — evaluate system behavior under different load levels and identify performance bottlenecks

Step 2 — Planning Mitigation Actions

Develop strategies to address identified SPOFs:

• redundancy and resilience strategies: replication, clustering, failover mechanisms and load balancing for critical components
• Isolation and containment techniques: microservice isolation and fault domain isolation to limit the impact of failures
• Circuit breakers and retry strategies to gracefully handle transient failures
• Graceful degradation to ensure that critical functions continue to work, even in a degraded manner

Step 3 — Implementation and testing

Deploy planned solutions and conduct rigorous testing to validate their effectiveness.

3.7 Redundancy vs. high availability

Redundancy

Redundancy involves the duplication of critical components or resources within a system. This duplication provides backups or alternatives that can be used in the event of a failure.

Analogy: Redundancy is like having a spare tire in your car. If one tire goes flat, you have others to keep you rolling. Similarly, in a system with redundancy, if a component fails, there are redundant copies or instances available to support its functions.

High Availability

High availability refers to the ability of a system to remain operational and accessible a high percentage of the time, typically measured as a percentage of availability over a given period of time.

Analogy: High availability is like having a car that rarely breaks down and is always ready to go. Just as a reliable car ensures that you can reach your destination without unexpected delays, a system with high availability ensures that users access its services with minimal downtime.

Main Difference

In summary: redundancy is a mechanism to mitigate the impact of failures, while high availability is a goal to ensure uninterrupted service.

3.8 Role of load balancing

Load balancing plays a crucial role in resilience by distributing incoming traffic across multiple servers or resources.

Contributions of load balancing to resilience

1. Fault tolerance — Load balancers distribute incoming requests across multiple servers. If a server goes down or becomes unavailable, the load balancer can redirect traffic to healthy servers. This improves fault tolerance by reducing the impact of individual server failures.

2. Horizontal scalability — Load balancers facilitate horizontal scalability by evenly distributing incoming traffic across multiple servers. As demand increases, additional servers can be added to the pool, and the load balancer dynamically adjusts traffic distribution.

3. Resource Usage Optimization — Load balancers help optimize resource usage by evenly distributing incoming requests across available servers. This prevents any individual server from being overwhelmed with requests while others remain underutilized.

4. Improved performance and availability — Load balancers can improve performance and availability by directing requests to the server with the lowest latency or highest availability, minimizing response times.

5. Traffic management and control — Load balancers provide traffic management and control capabilities, allowing administrators to configure routing policies, prioritize certain types of traffic, and implement traffic shaping rules.

3.9 Demo — Load Balancing with Kubernetes

GloboTicket project structure (Module 2)

globoticket-microservice-module-two-timeouts/
├── apps/
│   ├── auth/          # Service d'authentification (NestJS)
│   ├── bookings/      # Service de réservations (NestJS)
│   ├── events/        # Service d'événements/catalogue (NestJS)
│   └── web/           # Interface web (Next.js)
├── k8s/               # Configurations Kubernetes
│   ├── bookings.yml
│   ├── config.yml
│   ├── events.yml
│   └── web.yml
├── docker-compose.yaml
├── package.json
└── script.sh

Deployment script

#!/bin/bash  
echo " [ - ] Building docker images with docker compose"
docker-compose build

echo " [ - ] Pushing docker images to remote repository"
docker-compose push

echo " [ - ] Applying kubernetes configuration"
kubectl apply -f k8s

Docker Compose

version: "2.4"

services:
  bookings:
    image: me-central1-docker.pkg.dev/globomantics-globoticket/globoticket/bookings
    build:
      context: ./apps/bookings
      dockerfile: Dockerfile

  event-catalog:
    image: me-central1-docker.pkg.dev/globomantics-globoticket/globoticket/events
    build:
      context: ./apps/events
      dockerfile: Dockerfile

  web:
    image: me-central1-docker.pkg.dev/globomantics-globoticket/globoticket/web
    build:
      context: ./apps/web
      dockerfile: Dockerfile

Kubernetes — ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-map
data:
  bookings-url: http://bookings-service.default.svc.cluster.local:3000
  events-url: http://event-catalog-service.default.svc.cluster.local:3000
  web-url: http://web-service.default.svc.cluster.local:3000

Kubernetes — Deployment events.yml (with Load Balancer)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: event-catalog-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: event-catalog
  template:
    metadata:
      labels:
        app: event-catalog
    spec:
      containers:
        - name: event-catalog
          image: me-central1-docker.pkg.dev/globomantics-globoticket/globoticket/events
          ports:
            - containerPort: 3000
              name: events-port
---
apiVersion: v1
kind: Service
metadata:
  name: event-catalog-service
spec:
  selector:
    app: event-catalog
  ports:
    - protocol: TCP
      port: 3000
      targetPort: events-port
  type: LoadBalancer

Note: The key point is that the service is of type LoadBalancer. Kubernetes greatly simplifies setting up a load balancer — just specify type: LoadBalancer in the service configuration.

Kubernetes — Deployment web.yml (with environment variables from ConfigMap)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: me-central1-docker.pkg.dev/globomantics-globoticket/globoticket/web
          ports:
            - containerPort: 3000
              name: web-port
          env:
            - name: BOOKINGS_SERVICE
              valueFrom:
                configMapKeyRef:
                  name: config-map
                  key: bookings-url
            - name: EVENTS_SERVICE
              valueFrom:
                configMapKeyRef:
                  name: config-map
                  key: events-url
---
apiVersion: v1
kind: Service
metadata:
  name: web-service
spec:
  selector:
    app: web
  ports:
    - protocol: TCP
      port: 3000
      targetPort: web-port
  type: LoadBalancer

Kubernetes — Deployment bookings.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bookings-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: bookings
  template:
    metadata:
      labels:
        app: bookings
    spec:
      containers:
        - name: bookings
          image: me-central1-docker.pkg.dev/globomantics-globoticket/globoticket/bookings
          ports:
            - containerPort: 3000
              name: bookings-port
---
apiVersion: v1
kind: Service
metadata:
  name: bookings-service
spec:
  selector:
    app: bookings
  ports:
    - protocol: TCP
      port: 3000
      targetPort: bookings-port
  type: LoadBalancer

Kubernetes commands used in the demo

# Appliquer les configurations Kubernetes
kubectl apply -f k8s

# Inspecter les services avec surveillance continue
kubectl get services --watch

# Vérifier que les load balancers sont actifs
kubectl get services

# Vérifier si les load balancers fonctionnent
kubectl get endpoints

4. Retry, Timeout and Circuit Breakers

In this module, we learn about timeouts, transient failures, circuit breakers and some useful retry patterns to implement in our applications.

4.1 Transient Failures

Transient faults can be described as a brief disruption in a system, such as a momentary loss of network connection, temporary service unavailability, or timeouts caused by a busy service.

In a cloud environment where multiple resources are running on a single instance or competing for memory and storage, transient failures are inevitable. This may be due to procedures put in place by a cloud service provider or network provider.

Important: Transient failures do not only occur in cloud environments. They can also occur on-premises or even on a server running locally on your personal computer.

4.2 Timeouts

The problem

Consider two services: Service A which sends a request to Service B. Normally, we should expect a response from Service B within milliseconds — at worst, within seconds.

What happens if this response never reaches Service A because Service B experiences a transient failure and Service A remains waiting?

If no limit is set on the waiting time, the resources available on Service A will be busy waiting for a response from Service B. If all requests remain waiting, we will end up crashing both servers.

The solution: Timeouts

A timeout is simply: wait for a given period of time, and if you do not receive a response from the person you are waiting for, ignore and stop waiting.

In our case, just configure Service A to wait for Service B for 10 seconds. And if Service B has nothing to give us, we ignore and never wait for the response.

Golden rule: Avoid excessively long timeouts. This can have a cascading effect — like in the example where all of A’s resources were simply idle waiting for a response from B.

Example of configuring a timeout with Axios in Next.js

// Service HTTP avec timeout configuré via Axios
class HttpService {
  async makeRequest(url: string) {
    try {
      const response = await axios.get(url, {
        timeout: 5000, // 5000 ms = 5 secondes
      });
      return response.data;
    } catch (error) {
      // Capture les erreurs incluant les timeouts
      // et les remonte pour un traitement ultérieur
      throw error;
    }
  }
}

In this code:

• makeRequest method makes an HTTP GET request using the Axios library
• Configuration timeout: 5000 (5000 ms = 5 seconds) handles potential delays
• On successful response, method returns data
• Otherwise, it captures errors including timeouts and reports them for further processing

Demonstration in the GloboTicket project

In the events service of module 2, an artificial delay of 10 seconds was added to simulate a non-responsive service:

// apps/events/src/events/events.controller.ts
@Controller('events')
export class EventsController {
  constructor(private eventsService: EventsService) {}

  @Get()
  async getAllEvents() {
    // Délai artificiel pour simuler un service non réactif
    await new Promise((resolve) => setTimeout(resolve, 10000));
    return await this.eventsService.getAllEvents();
  }
  
  // ...
}

On the web side (Next.js), the timeout of 5 seconds is configured in Axios:

// apps/web/src/lib/index.ts
import axios, { AxiosError, AxiosResponse } from "axios"

const fetcher = async <T>(url: string): Promise<T> => {
  try {
    const response: AxiosResponse<ApiResponse> = await axios.get(url, {
      timeout: 5000, // 5 secondes
    })
    return response.data as T
  } catch (error) {
    if (axios.isAxiosError(error)) {
      const axiosError = error as AxiosError<ApiResponse>
      console.error("Axios error:", axiosError.message)
      throw axiosError
    } else {
      console.error("Non-Axios error:", error)
      throw error
    }
  }
}

export default fetcher

Since the endpoint takes 10 seconds to respond but our timeout is configured at 5 seconds, the timeout triggers after 5 seconds of waiting.

4.3 Simple Retries

Given that Service B’s outage was only a transient failure and was likely unavailable for a short period of time, should we simply cause a timeout for each request?

We can continue to do this, but it could potentially affect our user experience. What we can do is retry our requests.

Simple retry implementation

With the simple retry pattern, after our initial request expires, we try again a few times until we obtain positive feedback from Service B.

Problem with simple retry

However, if every other service keeps trying again, this could overwhelm Service B and make the situation worse. Returning queries can cause increased load on the service AND any downstream services it depends on — like its database for example. Overwhelming Service B can have a domino effect on many other services it depends on.

Conclusion: In certain situations, simple retries work perfectly, but we don’t want our services to be too aggressive. There is a better approach.

4.4 Exponential Backoff

The problem of simple retries revisited

With simple retries, if multiple services bombarded Service B with simultaneous retry requests, it could crash our service.

The solution: Exponential Backoff with Jitter

Unlike simple retries, with exponential backoff we can let Service A increase the period between retries so that Service B has enough time to recover between our requests.

In this example, Service A retries at exponential intervals. To add randomness and avoid synchronization of retries across multiple instances, a randomization element called jitter is introduced. Jitter introduces a slight variation in wait times between retries, helping to avoid situations where multiple instances of a service might retry simultaneously.

There is always a maximum backoff duration to avoid excessively long wait times. After reaching this maximum, subsequent retries occur at the maximum interval.

Exponential Backoff Formula

$$\text{nextRetryInterval} = \text{baseInterval} \times 2^{\text{retryCount}}$$

• baseInterval — the initial retry interval set
• retryCount — the number of attempts made

Implementation with axios-retry

The axios-retry library makes it very easy to implement retries and exponential backoff.

Installation

npm install axios-retry
# ou
yarn add axios-retry

Configuring simple retries

import axios from 'axios'
import axiosRetry from 'axios-retry'

axiosRetry(axios, {
  retries: 10, // Nombre maximum de retries
  shouldResetTimeout: true, // Réinitialise le timeout entre les retries
  retryCondition: (error) => {
    // Logique personnalisée pour déterminer si la requête doit être relancée
    console.log(`Retry attempt: ${error.config['axios-retry'].retryCount}`)
    return true // Toujours réessayer dans cet exemple
  }
})

Added exponential backoff (one line)

axiosRetry(axios, {
  retries: 10,
  shouldResetTimeout: true,
  retryDelay: axiosRetry.exponentialDelay, // Backoff exponentiel intégré
  retryCondition: (error) => {
    console.log(`Retry attempt: ${error.config['axios-retry'].retryCount}`)
    return true
  }
})

With the retryDelay: axiosRetry.exponentialDelay, retries are performed exponentially, thus avoiding the “retry storm” which could overwhelm the service.

4.5 Circuit Breaker

Difference with Retries

With the retry pattern, we assume that our service is simply experiencing a temporary downtime and would respond positively after a few retries.

The circuit breaker is different: we anticipate that the chances of a request failing are high and that our service will not recover as quickly. The failure is non-transient.

Operation

Circuit breakers in microservices act as a safety mechanism to prevent the propagation of failures by acting as a middleware or proxy for operations that might fail.

Just like circuit breakers at home, circuit breakers in microservices have three states:

The 3 states of the Circuit Breaker

         ┌─────────────────────────────────────────────────────────────┐
         │                                                             │
         │  FERMÉ (Closed)                                             │
         │  ┌──────────────────────────────────────────────────────┐   │
         │  │ Toutes les requêtes passent.                         │   │
         │  │ Service B fonctionne normalement.                    │   │
         │  └──────────────────────────────────────────────────────┘   │
         │          │                                                   │
         │          │ Seuil d'erreurs atteint                           │
         │          ▼                                                   │
         │  OUVERT (Open)                                               │
         │  ┌──────────────────────────────────────────────────────┐   │
         │  │ Les requêtes sont bloquées.                          │   │
         │  │ Service B est défaillant / surchargé.                │   │
         │  │ On protège Service B de la surcharge.                │   │
         │  └──────────────────────────────────────────────────────┘   │
         │          │                                                   │
         │          │ Après un délai (resetTimeout)                     │
         │          ▼                                                   │
         │  SEMI-OUVERT (Half-Open)                                     │
         │  ┌──────────────────────────────────────────────────────┐   │
         │  │ Quelques requêtes passent pour tester Service B.     │   │
         │  │ Service B est en cours de récupération.              │   │
         │  │ Si réussite → retour à FERMÉ                         │   │
         │  │ Si échec → retour à OUVERT                           │   │
         │  └──────────────────────────────────────────────────────┘   │
         └─────────────────────────────────────────────────────────────┘

• Closed State — The circuit breaker lets all requests from Service A pass to Service B. Service B is functioning correctly.

• Open State — When the state of Service B is degraded, the circuit breaker prevents requests from A to B to prevent Service B from being overwhelmed.

• Half-Open State — After a while, Service B recovers but is not fully ready. We don’t want to immediately overwhelm Service B with traffic. The circuit breaker, based on certain parameters, determines which requests can pass through to ensure that Service B is fully operational before overwhelming it with a full stream of requests.

Analogy: Just as we don’t jump out of bed but wake up gradually, the same goes for Service B — we give him time to fully recover.

4.6 Demo — Timeouts

In the module 4 demo (module-4-timeout branch of the GitHub repository), an artificial delay is added to the events service to simulate a non-responsive service:

// apps/events/src/events/events.controller.ts
@Get()
async getAllEvents() {
  // Simulation d'un service non réactif avec un délai de 10 secondes
  await new Promise((resolve) => setTimeout(resolve, 10000));
  return await this.eventsService.getAllEvents();
}

On the web side, the 5 second timeout has become an option for Axios:

// Appel avec timeout de 5 secondes
const response = await axios.get(url, { timeout: 5000 });

Result: Since the endpoint takes 10 seconds to respond but our timeout is 5 seconds, the timeout triggers after 5 seconds of waiting.

4.7 Demo — Retries and Exponential Backoff

In this demo (module-4-retries branch), the 10 second delay is removed and the retries are configured with axios-retry:

import axiosRetry from 'axios-retry'

// Configuration des retries simples
axiosRetry(axios, {
  retries: 10,
  shouldResetTimeout: true,
  retryCondition: (error) => {
    // Logique métier ici pour décider si on réessaie
    console.log(`Retry attempt: ${error.config['axios-retry'].retryCount}`)
    return true
  }
})

Observation: With simple retries, we see that the server can be bombarded with simultaneous retries.

Solution: Exponential Backoff

axiosRetry(axios, {
  retries: 10,
  shouldResetTimeout: true,
  retryDelay: axiosRetry.exponentialDelay, // Une seule ligne pour l'exponential backoff
  retryCondition: (error) => {
    console.log(`Retry attempt: ${error.config['axios-retry'].retryCount}`)
    return true
  }
})

Result: The server is no longer bombarded with simultaneous retries but the retries are carried out exponentially.

4.8 Demo — Circuit Breaker

The circuit breaker implementation uses the Opossum library in NestJS.

Circuit Breaker Interceptor (NestJS)

import {
  Injectable,
  NestInterceptor,
  ExecutionContext,
  CallHandler,
  BadGatewayException,
  Logger,
} from '@nestjs/common';
import { Observable } from 'rxjs';
import * as CircuitBreaker from 'opossum';

@Injectable()
export class CircuitBreakerInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    // Récupère les options du circuit breaker depuis les métadonnées du handler
    const circuitBreakerOptions = Reflect.getMetadata(
      'circuitBreakerOptions',
      context.getHandler(),
    );

    // Si pas d'options configurées, on exécute normalement
    if (!circuitBreakerOptions) {
      return next.handle();
    }

    // Créer un nouveau circuit Opossum
    const circuit = new CircuitBreaker(
      () => next.handle().toPromise(),
      circuitBreakerOptions,
    );

    return new Observable((observer) => {
      // Si le circuit est ouvert (service indisponible)
      circuit.on('open', () => {
        Logger.error('Circuit breaker is open - downstream service unavailable');
        observer.error(new BadGatewayException());
      });

      // Si le circuit se ferme (service disponible à nouveau)
      circuit.on('close', () => {
        Logger.log('Circuit breaker is closed - downstream service available');
        observer.complete();
      });

      // Déclencher le circuit
      circuit
        .fire()
        .then((result) => observer.next(result))
        .catch(() => observer.error(new BadGatewayException()));

      // Fermer le circuit lors du désabonnement
      return () => circuit.close();
    });
  }
}

Circuit Breaker Decorator

import { SetMetadata } from '@nestjs/common';

export const CircuitBreakerOptions = (options: {
  timeout?: number;
  errorThresholdPercentage?: number;
  resetTimeout?: number;
}) => SetMetadata('circuitBreakerOptions', options);

Use in a Controller

@Controller('events')
export class EventsController {
  @Get()
  @UseInterceptors(CircuitBreakerInterceptor)
  @CircuitBreakerOptions({
    timeout: 5000,                  // Déclenche une erreur si > 5 secondes
    errorThresholdPercentage: 50,   // Ouvre le circuit si 50% des requêtes échouent
    resetTimeout: 30000,            // Essaie à nouveau après 30 secondes
  })
  async getAllEvents() {
    return await this.eventsService.getAllEvents();
  }
}

Explanation of options:

5. Management of partial failures

This module explores other ways of handling partial failures: graceful degradation, fallback pattern and caching.

5.1 Graceful Degradation

Definition

A microservice gracefully degrades when it is functioning correctly even when one of its dependencies experiences downtime.

Graceful Degradation vs. Fault Tolerance

You could say it’s similar to what we said about fault tolerant applications. Here is the difference:

• Fault Tolerance — If we have a cluster of two services, in fault tolerance, our cluster would have an additional service on standby ready to take over in the event of a problem.

• Graceful Degradation — Our cluster would be reduced to a single service, but would continue to act as if two services were running.

Concrete example

Scenario: Alice scans an NFC chip with her phone at an event to get an exhibitor’s details. Our mobile client makes a request to our service to get these details, but our service’s database is completely unavailable.

Instead of failing and telling Alice “We can’t get the details for this exponent”, we can degrade gracefully by catching the error and quickly fetching the cached version from our cache, without harming the user experience.

async getExhibitorDetails(id: string) {
  try {
    // Tenter de récupérer depuis la base de données
    const details = await this.database.getExhibitor(id);
    // Mettre en cache pour utilisation ultérieure
    await this.cache.set(`exhibitor:${id}`, details);
    return details;
  } catch (error) {
    // Dégradation gracieuse : récupérer depuis le cache
    Logger.log('Database unavailable - falling back to cache');
    const cachedDetails = await this.cache.get(`exhibitor:${id}`);
    if (cachedDetails) {
      return cachedDetails;
    }
    throw new ServiceUnavailableException('Unable to retrieve exhibitor details');
  }
}

5.2 Fallback Pattern

“The only way to avoid software system failure is to avoid software system development.”

The fallback pattern ensures that a service continues to provide value and maintain functionality in the face of partial system failure, by providing an alternative action when the primary operation or service is unavailable or fails.

The beauty of the fallback pattern lies in its simplicity and effectiveness in ensuring uninterrupted service delivery, despite unforeseen failures.

Concrete example

Scenario: Our web interface makes a request to a recommendation service that uses a machine learning model to recommend events. This service is unfortunately unavailable.

Instead of returning an error, the fallback pattern allows you to identify the situation and fall back on the return of the highest classified events.

@Injectable()
export class RecommendationService {
  constructor(
    private readonly mlRecommendationEngine: MLRecommendationEngine,
    private readonly popularEventsService: PopularEventsService,
  ) {}

  async getRecommendationsForUser(userId: string) {
    try {
      // Opération principale : recommandations personnalisées via ML
      const recommendations = 
        await this.mlRecommendationEngine.getPersonalizedRecommendations(userId);
      return recommendations;
    } catch (error) {
      // Fallback : retourner les événements populaires si le moteur ML est indisponible
      Logger.warn(
        'ML recommendation engine unavailable - falling back to popular events',
      );
      return await this.popularEventsService.getTopRankedEvents();
    }
  }
}

Advantages of the Fallback Pattern:

• User still receives relevant results, even if primary service is unavailable
• Maintaining functional value without interruption visible to the user
• Reduced impact of partial outages on user experience

5.3 Caching

Caching is a technique commonly used in software and computer systems to store frequently accessed data in a temporary location where other services can consume it. This temporary space can be in memory or on CDNs.

Advantages of cache

• Performance improvement — everything seems faster
• Availability — access to certain data whether the main database is operational or not
• Reducing database pressure — for read-intensive applications when users increase

The 6 most popular caching methods

1. Database Cache

Database cache is implemented in conjunction with databases to facilitate rapid data retrieval, thereby avoiding repeating complex queries. We write the responses to these expensive queries in memory so that subsequent responses are faster.

2. Read-Through Cache

In a read-through cache implementation, we first look for the requested data in the cache:

• If found → cache hit: the cache returns the data
• If not found → cache miss: retrieve the data from the database, store it in the cache for future queries, then return it to the client

Client → Cache → (hit) → Client
            ↓ (miss)
         Database → Cache → Client

3. Write-Through Cache

Write-through cache is the opposite of read-through: instead of writing data first to the database and then to the cache, the data is written directly to the cache and the database is updated at the same time.

Client → Cache → Database (synchrone)

4. Write-Back Cache

Write-back is almost identical to write-through: you write to the cache first and update the database afterwards. But instead of writing to the database immediately, the update is deferred (asynchronous), which allows for faster write performance but introduces a risk of data loss in the event of a cache miss before persistence.

Client → Cache → (async) → Database

5. Write-Around Cache

With write-around, data is written directly to the database, bypassing the cache. This avoids polluting the cache with data that may not be reread frequently.

Client → Database (directement, bypass cache)

6. Cache-Aside (Cache-side)

Hide-aside (also called lazy loading) is the most common pattern. The application is responsible for reading and writing from the database, and the cache does not communicate directly with the database.

Client → Cache → (hit) → Client
            ↓ (miss)
Client → Database → Client (puis mise à jour du cache)

Important considerations for caching

• Cache Invalidation — How and when to invalidate cached data is one of the most difficult problems in computing
• TTL (Time-To-Live) — Set a lifetime for cache entries
• Eviction policies — LRU (Least Recently Used), LFU (Least Frequently Used), etc.
• Cache Stampede — Situation where many requests arrive simultaneously for expired data

5.4 Demo — Cache, Fallback and Graceful Degradation

In this demo (Module 5), we simultaneously implement caching, the fallback pattern and graceful degradation using the NestJS Cache Manager.

Configuring the module with Cache Manager

// apps/events/src/app.module.ts
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { AppService } from './app.service';
import { EventsModule } from './events/events.module';
import { OrganizersModule } from './organizers/organizers.module';
import { AttendeesModule } from './attendees/attendees.module';
import { CacheModule } from '@nestjs/cache-manager';

@Module({
  imports: [
    EventsModule,
    OrganizersModule,
    AttendeesModule,
    CacheModule.register(), // Enregistrement du module de cache
  ],
  controllers: [AppController],
  providers: [AppService],
})
export class AppModule {}

Service Events with Cache Manager, Fallback and Graceful Degradation

// apps/events/src/events/events.service.ts
import {
  BadRequestException,
  Inject,
  Injectable,
  Logger,
} from '@nestjs/common';
import { PrismaService } from '../prisma.service';
import { Event } from '.prisma/client';
import { CACHE_MANAGER, Cache } from '@nestjs/cache-manager';

@Injectable()
export class EventsService {
  constructor(
    private prisma: PrismaService,
    @Inject(CACHE_MANAGER) private cacheManager: Cache, // Injection du Cache Manager
  ) {}

  async getAllEvents() {
    try {
      // Tentative de récupération depuis la base de données
      const events = await this.prisma.event.findMany();
      
      // Mise en cache des résultats (Database Cache)
      if (events) await this.cacheManager.set('events', events);
      
      // Pour démontrer le fallback, on force une erreur :
      throw new Error('Failed to fetch from database');
      
    } catch (error) {
      // Graceful degradation : fallback vers le cache
      Logger.log('Falling back to cache');
      
      const events = await this.cacheManager.get('events');
      
      if (events) {
        return events; // Retourne les données en cache (potentiellement stale)
      } else {
        throw new BadRequestException(
          'Failed to fetch events and cached events',
        );
      }
    }
  }

  async createEvent(data: Omit<Event, 'id' | 'createdAt' | 'updatedAt'>) {
    return this.prisma.event.create({ data });
  }

  async getEventById(id: string) {
    return await this.prisma.event.findUnique({ where: { id } });
  }

  async updateEvent(id: string, data: Partial<Event>) {
    return await this.prisma.event.update({ where: { id }, data });
  }

  async deleteEvent(id: string) {
    return await this.prisma.event.delete({ where: { id } });
  }
}

Installed dependencies (events service package.json)

{
  "dependencies": {
    "@nestjs/cache-manager": "^2.2.2",
    "@nestjs/common": "^10.0.0",
    "@nestjs/core": "^10.0.0",
    "cache-manager": "^5.4.0",
    "@prisma/client": "5.8.1"
  }
}

Demo test:

1. First call to endpoint → events are retrieved from database and cached
2. Second call → forced error is triggered, code gracefully degrades and returns cache data
3. The logs confirm: "Falling back to cache" followed by the return of data

In this demo, we were able to implement a cache, use the fallback pattern and degrade gracefully. Each of these strategies helps ensure that services remain functional and continue to deliver value even in the face of unexpected failures.

6. Advanced techniques

This module explores rate limiting techniques (fixed window, sliding window, leaky bucket), bulkhead pattern and load shedding.

6.1 Fixed Window Rate Limiting

Setting Rate Limiting

rate limiting is a policy implemented by software systems to control the rates at which users or services can access a resource. When the request rate exceeds the threshold defined by the rate limiter, requests are throttled or blocked.

Advantages of Rate Limiting

• Distributed Denial of Service (DDoS) Attack Prevention — In some cases, sophisticated DDoS attacks distribute requests among many different sources, sometimes millions of IP addresses. Distributing the attack allows each source to avoid exceeding rate limits.
• Service Overload Protection
• Guarantee of fair use of resources

Fixed Window — Principle

The fixed window technique is very simple. In this rate limiting method, we define a ceiling or a limit for the maximum number of requests that we want to receive for a given period. Any request that exceeds this limit after the end of the time window is refused, usually with an HTTP error (HTTP 429 Too Many Requests).

Example: We assign a window of 1 minute for a ceiling of 3 requests. Requests 1, 2 and 3 fall within the 1 minute window and the defined cap. The following requests (4 and 5) are refused.

Cookie jar analogy

Imagine you have a cookie jar and you want to share cookies with your friends, but you don’t want them to take too many at once. So you decide that every 5 minutes you will fill the cookie jar with a fixed number of cookies, say 10. This means that no matter how many cookies your friends take during those 5 minutes, once the time is up the jar is filled again with exactly 10 cookies. Friends can only have cookies until the jar is empty, and they must wait until the 5 minutes are up for it to be refilled.

Fixed Window problem

The fixed window problem is that at the boundary between two windows, a user can send many requests and double them (for example, 10 requests at the end of window 1 + 10 requests at the start of window 2 = 20 effective requests in a short time).

6.2 Sliding Window Rate Limiting

Principle

With the sliding window, instead of waiting a fixed period like 5 minutes before filling the cookie jar, we decide that your friends can only take a certain number of cookies in a specific period of time.

Example: The limit is 10 cookies every 5 minutes. Imagine you have a timer that starts as soon as someone takes a cookie. As long as the timer has not reached 5 minutes AND the total number of cookies taken does not exceed 10, your friends can continue taking cookies without problem.

The restriction: If someone takes too many cookies and the timer has not yet been reset, no more cookies can be taken until enough time has passed for the timer to slide forward and allow more cookies to be taken.

Advantage over the fixed window: The sliding window solves the problem of “bursts” at the boundary between two windows, because the window moves continuously over time.

Fenêtre fixe :     [──────── 1 min ────────][──────── 1 min ────────]
Fenêtre glissante: ─────────[──────── 1 min ────────]─────────────────
                         (déplace en continu avec le dernier événement)

6.3 Leaky Bucket

Principle

The leaky bucket controls flow by allowing requests to be processed at a constant rate, regardless of peak demand.

Analogy: We have a special bucket for cookies called the leaky bucket. We decide to only allow a certain number of cookies to be taken from the bucket in a given period of time.

Here’s how it works:

• Cookies continually fall into the bucket at a fixed rate, say 5 cookies per minute
• The bucket has a limit on how many cookies it can hold at a time, say 10 cookies
• If the bucket is full and someone tries to add more cookies, they overflow — excess requests are rejected or handled differently
• Anyone want to take some cookies from the bucket? He can take them at a constant rate, but only as many as the bucket currently holds

Requêtes entrantes → [     Seau (max 10)    ] → Traitement (taux constant: 5/min)
                       (débordement si plein)

Features:

• Smooth out traffic spikes by serving requests at a constant rate
• Excess requests are either queued or rejected
• Ensures predictable processing throughput

6.4 Bulkhead Pattern

The bulkhead pattern (insulation bulkhead) is inspired by the watertight bulkheads of ships which divide the hull into separate compartments. If one compartment floods, the others remain intact, preventing the ship from sinking.

In microservices, the bulkhead pattern isolates different components or functionalities into separate resource pools. If a component experiences problems (like thread exhaustion), the impact is contained within its pool and does not propagate to others.

// Exemple conceptuel de bulkhead avec des pools de connexions séparés
const criticalOperationsPool = new ConnectionPool({ maxConnections: 50 });
const nonCriticalOperationsPool = new ConnectionPool({ maxConnections: 20 });

// Les opérations critiques ont leur propre pool isolé
async function criticalOperation() {
  const conn = await criticalOperationsPool.acquire();
  // ...
}

// Les opérations non critiques utilisent un pool séparé
async function nonCriticalOperation() {
  const conn = await nonCriticalOperationsPool.acquire();
  // ...
}

6.5 Load Shedding

Load shedding is a technique that consists of deliberately rejecting requests when the system is under too much load, in order to protect the main service.

Unlike rate limiting which controls the flow over a period, load shedding is more dynamic and is triggered according to the current state of system load.

6.6 Demo — Load Shedding

In this demo, we implement a load shedder which dynamically offloads the load based on the number of requests in a given interval.

LoadSheddingMiddleware

import {
  Injectable,
  NestMiddleware,
  Logger,
  ServiceUnavailableException,
} from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';

@Injectable()
export class LoadSheddingMiddleware implements NestMiddleware {
  private readonly logger = new Logger(LoadSheddingMiddleware.name);
  
  // Nombre maximum de requêtes autorisées par intervalle
  private readonly MAX_REQUEST_PER_INTERVAL = 2;
  
  // Compteur de requêtes depuis la dernière réinitialisation
  private requestCount = 0;
  
  // Timestamp de la dernière réinitialisation
  private lastResetTime = Date.now();

  use(req: Request, res: Response, next: NextFunction) {
    const now = Date.now();

    // Vérifier s'il faut réinitialiser le compteur (toutes les 60 secondes)
    if (now - this.lastResetTime > 60000) {
      this.requestCount = 0;
      this.lastResetTime = now;
    }

    // Incrémenter le compteur de requêtes
    this.requestCount++;
    this.logger.log(`Current request count: ${this.requestCount}`);

    // Appliquer le load shedding si le seuil est dépassé
    if (this.requestCount > this.MAX_REQUEST_PER_INTERVAL) {
      this.logger.warn(
        `Load shedding applied - request count: ${this.requestCount}`,
      );
      res
        .status(503)
        .json({ message: 'Service temporarily unavailable - load shedding applied' });
      return;
    }

    // Continuer normalement si sous le seuil
    next();
  }
}

Middleware explanation:

• MAX_REQUEST_PER_INTERVAL — constant which defines the maximum number of requests authorized in a given period (here 2, but configurable)
• requestCount — tracks the total number of requests received since the last reset
• lastResetTime — timestamp of last reset, used to determine when to reset the request counter
• Middleware logic is invoked for each incoming request
• If more than 1 minute has passed since the last reset, the counter is reset to 0
• If load shedding is triggered, a 503 Service Unavailable response is sent to the client

Test with loadtest

# Installation de loadtest
npm install -g loadtest

# Test de charge : 3 requêtes concurrentes sur notre service
loadtest -c 3 http://localhost:3000/events

Result: Load shedding is triggered because our load shedder is configured to manage 2 requests per interval but our test sends 3 requests simultaneously.

Load shedding can be configured to meet the needs of your application. This is just an example to demonstrate how load shedding helps build resilient and fault-tolerant systems.

7. Case studies and best practices

Resilience and fault tolerance are endless topics, with thousands of ways to manage them. This module discusses how some of the world’s largest technology organizations—Netflix, Google, and Amazon Web Services—manage resilience and fault tolerance.

7.1 Netflix — Chaos Engineering

Netflix Philosophy

Netflix’s commitment to resilience is demonstrated by the adoption of chaos engineering practices, a methodology aimed at proactively identifying weaknesses in the system before they lead to widespread outages or service interruptions.

What is Chaos Engineering?

Chaos engineering involves intentionally injecting failures — such as network latency, server crashes, or database errors — into a production environment to observe how the system reacts.

Chaos Monkey

Netflix’s most famous tool is Chaos Monkey, developed by Netflix and part of the Simian Army sequel. Chaos Monkey automates this process by:

• Randomly terminating instances or processes
• Inducing failures in various system components
• Simulating entire Availability Zone failures

Benefits of Chaos Engineering at Netflix

1. Vulnerability Discovery — Through chaos engineering, Netflix can discover vulnerabilities and dependencies in its infrastructure that would not be apparent under normal operating conditions.

2. Validation of failover mechanisms — By simulating real failures in a controlled environment, engineers gain valuable insights into system behavior under stress and can identify opportunities to improve resilience and fault tolerance.

3. Redundancy Validation — Chaos engineering allows Netflix to validate the effectiveness of its redundancy and failover mechanisms, ensuring that critical services remain available even in the event of unexpected disruptions.

By embracing chaos and uncertainty, Netflix remains at the forefront of innovation in the digital entertainment industry, delivering seamless streaming experiences to millions of users around the world.

7.2 Google Cloud — High Performance Private Network

Google network infrastructure

Google operates one of the world’s largest and most advanced private networks, consisting of a global network backbone that interconnects its data centers. This private backbone network is built using dedicated fiber optic cables and specialized networking hardware, allowing Google to maintain full control over network traffic and ensure maximum performance and reliability.

Latency reduction

One of the key benefits of Google’s private network backbone is its ability to minimize latency — the delay experienced by data as it travels between devices. By leveraging a global network of strategically located data centers and optimizing routing algorithms, Google can reduce the distance data must travel, thereby minimizing latency and improving the responsiveness of its services.

Minimizing packet loss

In addition to minimizing latency, Google’s private network backbone is designed to minimize packet loss — which occurs when data packets are dropped or corrupted during transmission. Google achieves this by using advanced network protocols and techniques such as:

• Redundant paths — multiple routes for data packets
• Error detection — identifying and correcting transmission errors
• Congestion control algorithms — network overload prevention

These measures help ensure that data packets reach their intended destinations reliably and without errors.

7.3 Amazon Web Services — Resilient Services

As a leading cloud service provider, AWS has a broad range of services where resilience and fault tolerance are essential. AWS powers popular products and services we use today, and downtime can have a massive domino effect on economies and our daily lives.

Amazon S3 (Simple Storage Service)

Amazon S3 is a highly scalable object storage service designed to store and retrieve any amount of data at any time.

Resilience mechanisms:

• S3 automatically replicates data across multiple Availability Zones within a region to ensure high durability and availability
• When storing objects in S3, users can specify storage class and replication configuration
• In the event of an outage in one Availability Zone, S3 transparently redirects requests to replicas stored in other Availability Zones, ensuring continued access to stored data

Durability: Amazon S3 provides 99.999999999% (11 nines) object durability over one year.

Amazon EC2 (Elastic Compute Cloud)

Amazon EC2 provides resizable compute capacity in the cloud, allowing users to launch virtual servers (instances) on demand.

Resilience mechanisms:

• EC2 instances can be deployed across multiple Availability Zones within a region to achieve fault tolerance and high availability
• Users can create EC2 Auto Scaling groups that distribute instances across multiple Availability Zones to ensure workload availability and scalability
• AWS Elastic Load Balancing seamlessly integrates with EC2 to distribute incoming traffic across multiple instances in different Availability Zones, optimizing performance and fault tolerance

Amazon RDS (Relational Database Service)

Amazon RDS is a managed relational database service that simplifies the setup, operation, and scaling of relational databases in the cloud. RDS supports several database engines: MySQL, PostgreSQL, Oracle and SQL Server.

Resilience mechanisms:

• When provisioning an RDS instance, users can choose the Multi-AZ deployment option, which automatically creates synchronous standby replicas in different Availability Zones
• These replicas serve as failover targets in the event of a primary database failure, ensuring high availability and data durability
• RDS manages data failure and replication across Availability Zones transparently to users, minimizing downtime and data loss

AWS Multi-AZ Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                          AWS Region (ex: us-east-1)                      │
│                                                                          │
│  ┌────────────────┐   ┌────────────────┐   ┌────────────────┐           │
│  │ Availability   │   │ Availability   │   │ Availability   │           │
│  │ Zone A         │   │ Zone B         │   │ Zone C         │           │
│  │                │   │                │   │                │           │
│  │ ┌──────────┐   │   │ ┌──────────┐   │   │ ┌──────────┐   │           │
│  │ │ EC2      │   │   │ │ EC2      │   │   │ │ EC2      │   │           │
│  │ │ Instance │   │   │ │ Instance │   │   │ │ Instance │   │           │
│  │ └──────────┘   │   │ └──────────┘   │   │ └──────────┘   │           │
│  │ ┌──────────┐   │   │ ┌──────────┐   │   │ ┌──────────┐   │           │
│  │ │ RDS      │   │   │ │ RDS      │   │   │ │ RDS      │   │           │
│  │ │ Primary  │◀──┼───┼▶│ Standby  │   │   │ │ Standby  │   │           │
│  │ └──────────┘   │   │ └──────────┘   │   │ └──────────┘   │           │
│  │ ┌──────────┐   │   │ ┌──────────┐   │   │ ┌──────────┐   │           │
│  │ │ S3       │◀──┼───┼▶│ S3       │◀──┼───┼▶│ S3       │   │           │
│  │ │ Replica  │   │   │ │ Replica  │   │   │ │ Replica  │   │           │
│  │ └──────────┘   │   │ └──────────┘   │   │ └──────────┘   │           │
│  └────────────────┘   └────────────────┘   └────────────────┘           │
│                                                                          │
│                    ┌──────────────────────────┐                          │
│                    │   Elastic Load Balancer  │                          │
│                    └──────────────────────────┘                          │
└─────────────────────────────────────────────────────────────────────────┘

AWS services like S3, RDS, and EC2 are designed with built-in capabilities to automatically distribute workloads and data across multiple Availability Zones within a region. This distributed architecture improves fault tolerance, resiliency, and high availability by mitigating the impact of failures in individual availability zones and ensuring the continued operation of cloud applications and services.

8. GloboTicket application files reference

Project architecture

globoticket-microservice-module-two-timeouts/
├── apps/
│   ├── auth/                          # Service d'authentification
│   │   └── src/
│   │       ├── app.module.ts          # Module principal
│   │       ├── app.controller.ts      # Contrôleur principal
│   │       ├── app.service.ts         # Service principal
│   │       └── main.ts                # Point d'entrée
│   │
│   ├── bookings/                      # Service de réservations
│   │   ├── prisma/                    # Schéma Prisma ORM
│   │   └── src/
│   │       ├── bookings/
│   │       │   ├── bookings.controller.ts
│   │       │   ├── bookings.service.ts
│   │       │   ├── bookings.module.ts
│   │       │   └── dto/
│   │       ├── app.module.ts
│   │       ├── prisma.service.ts
│   │       └── main.ts
│   │
│   ├── events/                        # Service d'événements (catalogue)
│   │   ├── prisma/
│   │   └── src/
│   │       ├── events/
│   │       │   ├── events.controller.ts
│   │       │   ├── events.service.ts
│   │       │   ├── events.module.ts
│   │       │   └── dto/
│   │       ├── app.module.ts
│   │       ├── prisma.service.ts
│   │       └── main.ts
│   │
│   └── web/                           # Interface utilisateur (Next.js)
│       └── src/
│           ├── lib/
│           │   ├── index.ts           # Fetcher Axios + endpoints
│           │   └── types.ts
│           ├── pages/
│           │   ├── index.tsx
│           │   └── api/
│           ├── components/
│           └── styles/
│
├── k8s/                               # Configurations Kubernetes
│   ├── bookings.yml                   # Deployment + Service (LoadBalancer)
│   ├── config.yml                     # ConfigMap (URLs des services)
│   ├── events.yml                     # Deployment + Service (LoadBalancer)
│   └── web.yml                        # Deployment + Service (LoadBalancer)
│
├── docker-compose.yaml                # Configuration Docker Compose
├── package.json                       # Workspace Yarn
└── script.sh                          # Script de build/push/deploy

Bookings Controller

// apps/bookings/src/bookings/bookings.controller.ts
import { Body, Controller, Post } from '@nestjs/common';
import { BookingsService } from './bookings.service';
import { CreateBookingDto } from './dto/booking.dto';

@Controller('booking')
export class BookingsController {
  constructor(private bookingService: BookingsService) {}

  @Post()
  async createBooking(@Body() data: CreateBookingDto) {
    return await this.bookingService.createBooking(data);
  }
}

Bookings Service

// apps/bookings/src/bookings/bookings.service.ts
import { Injectable } from '@nestjs/common';
import { PrismaService } from 'src/prisma.service';
import { Booking } from '.prisma/client';

@Injectable()
export class BookingsService {
  constructor(private prisma: PrismaService) {}

  async createBooking(
    data: Omit<Booking, 'id' | 'createdAt' | 'quantity' | 'updatedAt'>,
  ) {
    return this.prisma.booking.create({ data });
  }

  async getBookings() {
    return this.prisma.booking.findMany();
  }
}

Bookings Main (NestJS bootstrap)

// apps/bookings/src/main.ts
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { ValidationPipe } from '@nestjs/common';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Validation avec class-validator
  app.useGlobalPipes(
    new ValidationPipe({
      whitelist: true,
    }),
  );

  app.enableCors({
    origin: '*',
    methods: '*',
    allowedHeaders: '*',
  });

  await app.listen(process.env.PORT);
}
bootstrap();

Events Controller (Module 2 — with artificial delay)

// apps/events/src/events/events.controller.ts (Module 2 - timeout demo)
import {
  Body,
  Controller,
  Get,
  Param,
  ParseUUIDPipe,
  Patch,
  Post,
} from '@nestjs/common';
import { EventsService } from './events.service';
import { CreateEventDto, EditEventDto } from './dto/event.dto';

@Controller('events')
export class EventsController {
  constructor(private eventsService: EventsService) {}

  @Get()
  async getAllEvents() {
    // Délai artificiel de 10 secondes pour simuler un service non réactif
    await new Promise((resolve) => setTimeout(resolve, 10000));
    return await this.eventsService.getAllEvents();
  }

  @Get(':id')
  async getEventById(@Param('id', ParseUUIDPipe) id: string) {
    return await this.eventsService.getEventById(id);
  }

  @Post()
  async createEvent(@Body() data: CreateEventDto) {
    return await this.eventsService.createEvent(data);
  }

  @Patch(':id')
  async updateEvent(id: string, @Body() data: EditEventDto) {
    return await this.eventsService.updateEvent(id, data);
  }
}

Events Service (Module 2)

// apps/events/src/events/events.service.ts (Module 2)
import { Injectable } from '@nestjs/common';
import { PrismaService } from '../prisma.service';
import { Event } from '.prisma/client';

@Injectable()
export class EventsService {
  constructor(private prisma: PrismaService) {}

  async createEvent(data: Omit<Event, 'id' | 'createdAt' | 'updatedAt'>) {
    return this.prisma.event.create({ data });
  }

  async getAllEvents() {
    return await this.prisma.event.findMany();
  }

  async getEventById(id: string) {
    return await this.prisma.event.findUnique({ where: { id } });
  }

  async updateEvent(id: string, data: Partial<Event>) {
    return await this.prisma.event.update({ where: { id }, data });
  }

  async deleteEvent(id: string) {
    return await this.prisma.event.delete({ where: { id } });
  }
}

Web Fetcher with Axios (lib/index.ts)

// apps/web/src/lib/index.ts
import axios, { AxiosError, AxiosResponse } from "axios"

interface ApiResponse {
  data: any
}

const fetcher = async <T>(url: string): Promise<T> => {
  try {
    const response: AxiosResponse<ApiResponse> = await axios.get(url)
    return response.data as T
  } catch (error) {
    if (axios.isAxiosError(error)) {
      const axiosError = error as AxiosError<ApiResponse>
      console.error("Axios error:", axiosError.message)
      throw axiosError
    } else {
      console.error("Non-Axios error:", error)
      throw error
    }
  }
}

export default fetcher

type Services = "events" | "bookings"

// Endpoints des services depuis les variables d'environnement
// (injectées depuis le ConfigMap Kubernetes)
export const serviceEndpoints: Record<Services, string | undefined> = {
  events: process.env.EVENTS_SERVICE,
  bookings: process.env.BOOKINGS_SERVICE,
}

Local boot commands

# Démarrer tous les services en mode développement
yarn dev

# Ou démarrer chaque service individuellement
yarn auth      # Service d'authentification
yarn bookings  # Service de réservations
yarn events    # Service d'événements
yarn web       # Interface web Next.js

Main dependencies

Search Terms

node.js · microservices · resilience · fault · tolerance · apis · backend · full-stack · web · cache · failures · circuit · kubernetes · load · breaker · globoticket · service · system · amazon · backoff · chaos · degradation · deployment · exponential