Introduction to Deep Learning

Original language: English (notes in French)

Table of Contents

• Introduction
• Introducing Deep Learning
• 1.1 What is deep learning?
• 1.2 Networks and layers
• 1.3 Where deep learning shines
• 1.4 Why is deep learning so powerful?
• Universal function approximation
• Automatic feature learning
• Scalability and ethical considerations
• 1.5 Generalization, overfitting and underfitting
• Overfitting
• Underfitting
• 1.6 Where deep learning fits in the machine learning landscape
• 1.7 Neural networks — Detailed architecture
• The artificial neuron
• Network layers
• Weights and biases
• Activation functions
• 1.8 Demo — TensorFlow Playground
• 1.9 Network architectures
• Fully Connected Perceptron (MLP)
• Convolutional Neural Network (CNN)
• Recurrent Neural Network (RNN)
• Transformers
• Other methods (Kernel Methods, SVM)
• The black box problem and Explainable AI
• Training a Network
• 2.1 The loss function
• Mean Squared Error (MSE)
• Concrete example of MSE calculation
• 2.2 Gradient descent
• Basic intuition (1 weight)
• Multi-dimensional extension
• Blind Mountain Analogy
• 2.3 Local vs. global minimum
• 2.4 Preparing the data
• Split train / test
• Split train / validation / test
• 2.5 Cross-validation
• 2.6 Preprocessing your data
• Normalization and standardization
• Logarithmic transformation
• Feature selection and feature engineering
• 2.7 The backpropagation algorithm
• Forward pass
• Backward pass and chain rule
• Microgrid step-by-step example
• 2.8 Loss and weights — Calculation of gradients
• Gradient for w2 (output layer)
• Gradient for w1 (hidden layer)
• Weights update
• The vanishing gradient problem
• Backpropagation efficiency
• 2.9 Model performance
• Loss curves
• Early stopping
• Classification Metrics
• Confusion matrix
• 2.10 Regularization
• L2 Regularization (Weight Decay)
• Dropout
• Other regularization techniques
• 2.11 Optimizing backpropagation
• Learning rate — most important hyperparameter
• Current hyperparameters
• Batch size and gradient descent variants
• Advanced optimizers
• Hyperparameter search
• Summary and key concepts
• Glossary

1. Introduction

Welcome to this training on deep learning. This technology has completely transformed the field of artificial intelligence over the past decade. If you’ve used a voice assistant, seen a self-driving car, or chatted with ChatGPT, you’ve already experienced deep learning in action.

This training covers how these systems work, what makes them so powerful, and how they are trained to accomplish these tasks. There’s a lot of math behind deep learning, but this training keeps things accessible and conceptual. Formulas and diagrams will be presented, but the goal is to make it all as intuitive as possible.

Training structure:

• Module 1: What is deep learning and why has it become so popular
• Module 2: How neural networks learn via the backpropagation process

2. Introducing Deep Learning

1.1 What is deep learning?

Deep learning is a branch of machine learning that uses neural networks, computer systems loosely inspired by the human brain. These networks are made up of layers of connected nodes, called neurons, which process information.

In popular media it is said that neural networks learn just like the human brain, but in reality the similarity is quite superficial. A neural network is basically a huge matrix, a big bag of numbers. Information flows between parts of the network just as neurons in the brain transmit signals, but in a neural network it is simply numbers multiplied, added, and transformed in various ways.

The “neural” part is basically a metaphor. It’s not like the network really thinks or understands things the same way a human brain does, at least as far as we know.

graph LR
    A[🧠 Cerveau humain<br/>Neurones biologiques<br/>Cellules vivantes<br/>Signaux électrochimiques] 
    B[💻 Neural Network<br/>Neurones artificiels<br/>Nombres / matrices<br/>Multiplications et sommes]
    A -.->|Inspiration superficielle| B
    style A fill:#e8f4f8,stroke:#2196F3
    style B fill:#f3e8f8,stroke:#9C27B0

1.2 Networks and layers

A neural network is structured in layers. Think of it as matrices stacked on top of each other, big lists of numbers. Each layer contains a set of neurons (numbers) and these neurons are connected to neurons in the previous and next layers. Information flows from one layer to another.

The “deep” part in deep learning means that we are talking about neural networks with many layers. This allows the network to learn very complex patterns in the data. The more layers you have, the deeper the network and the more powerful it is, but also harder to train.

graph LR
    subgraph INPUT["Input Layer"]
        I1((x₁))
        I2((x₂))
        I3((x₃))
    end
    subgraph H1["Hidden Layer 1"]
        H11((h₁))
        H12((h₂))
        H13((h₃))
        H14((h₄))
    end
    subgraph H2["Hidden Layer 2"]
        H21((h₅))
        H22((h₆))
        H23((h₇))
    end
    subgraph OUTPUT["Output Layer"]
        O1((ŷ₁))
        O2((ŷ₂))
    end

    I1 --> H11 & H12 & H13 & H14
    I2 --> H11 & H12 & H13 & H14
    I3 --> H11 & H12 & H13 & H14
    H11 & H12 & H13 & H14 --> H21 & H22 & H23
    H21 & H22 & H23 --> O1 & O2

    style INPUT fill:#e8f4f8,stroke:#2196F3
    style H1 fill:#fff3e0,stroke:#FF9800
    style H2 fill:#fff3e0,stroke:#FF9800
    style OUTPUT fill:#e8f5e9,stroke:#4CAF50

1.3 Where deep learning shines

Here are some areas where deep learning has really made a difference:

Computer vision

Deep learning absolutely dominates computer vision: image recognition, object detection, and everything visual.

• Your phone’s face unlock is powered by a deep network
• The technology that allows self-driving cars to understand their surroundings and make decisions in real time
• Filters and effects in apps like Instagram and Snapchat

Natural language processing (NLP)

Understanding and generating text is another strong point of deep neural networks.

• Automatic translation – Chatbots like ChatGPT that can hold conversations, write essays and even poetry

Audio and speech recognition

Deep learning has enabled huge improvements in speech recognition:

• Technology that allows you to dictate text or control your devices with your voice
• Voice cloning: training a neural network with your own voice to generate new speech in your voice — it’s a little worrying, but also pretty incredible
• Background noise processing: a deep neural network can detect which part is voice and which part is noise, then remove the noise while keeping the voice intact

Music and artistic creation

Neural networks are used to create new music and process audio.

Games and reinforcement learning

Deep learning combined with reinforcement learning has created systems that can beat world champions at chess, Go and other complex games.

1.4 Why is deep learning so powerful?

Universal function approximation

Neural networks are universal function approximators, which means that they can model any relationship between inputs and outputs. This makes them incredibly versatile for a wide range of tasks. In other words, neural networks can theoretically learn anything given enough data and computing power.

“Neural networks are the second best solution for everything.” — University professor cited in the training

This means that in general you can get better or more efficient results with a more specialized algorithm, but you will have to design that algorithm, which takes time and expertise. The beauty of deep learning is that it can be applied to so many different problems without needing to be redesigned from scratch.

Automatic feature learning

In traditional machine learning, you often have to manually design your features. A feature is a specific property of data that you believe is important to the task at hand.

Examples of manual feature engineering:

• To predict the price of a house: number of bedrooms, total area
• To recognize cats: pointed ears, presence of whiskers, shape of the eyes

With most other machine learning techniques, you have to decide for yourself which features to look for, and then the algorithm learns to use those features to make predictions. But with deep learning, the network discovers these features by itself. Instead of having to decide what’s important, you just give it raw data and the network will determine which parts of the data are important.

If you train a deep learning model to recognize cats in images, you simply give it photos of cats and non-cats (dogs, for example), and the network will automatically learn how to identify relevant features — the shape of ears, the presence of whiskers, the shape of eyes — without you having to explicitly tell it to look for those features.

Note: In practice, you often still need to do feature engineering to obtain the best results. But the key point is that deep learning can learn complex features without needing to be explicitly programmed to search for them.

Scalability and ethical considerations

The more data and computing power you give to a deep learning model, the better it tends to perform. With traditional machine learning algorithms, there is often a point of diminishing returns. Adding more data stops helping after reaching a certain threshold. But deep learning networks continue to improve with more data and calculation.

This is why companies like OpenAI and Google are investing billions in both collecting massive datasets and building powerful computing infrastructures. Larger networks trained on more data simply perform better.

⚠️ Ethical and environmental note: The energy consumption to drive these massive models is enormous. Additionally, it has been shown that this scaling also has diminishing returns — LLMs need to scale harder and harder for smaller and smaller improvements. Before training a huge network, it is worth thinking about whether it is really necessary, and whether a more efficient and less wasteful solution does not exist.

1.5 Generalization, overfitting and underfitting

Here’s something important to understand about machine learning in general and deep learning in particular. The goal isn’t just to learn the training data, it’s to generalize to new, unseen data.

Analogy: Imagine you are studying for an exam. If you memorize every practice question word for word, you could get a great score on a test that uses exactly those questions. But if the exam has different questions testing the same concepts, you would be in trouble. This is essentially the difference between memorization and true understanding.

The whole point of a neural network is that it generalizes. It learns the underlying patterns in the data, not just the specific examples it was trained on. And this is what allows it to perform on new data that it has never seen before.

Concrete example: If we show the network a million images of cats during training, we want it to be able to recognize a new cat that it has never seen before. If it can only recognize these specific one million images it was trained on, the network does not generalize.

Overfitting

Overfitting occurs when your model learns the training data too well. This seems weird — how can you learn something too well?

What happens is that because the network sort of zooms in too close to the cats in the training data, it starts picking up noise and irrelevant details that are specific to that data. When this happens, you will see that the network performs poorly on new data because it has essentially memorized the training examples rather than learning the general patterns that would allow it to recognize new cats.

Performance sur données d'entraînement : EXCELLENTE ✓
Performance sur nouvelles données       : MAUVAISE   ✗
→ Le modèle a mémorisé, pas appris

Deep networks with their millions of parameters are particularly prone to this problem. They have enough capacity to store entire datasets.

Underfitting

The opposite problem is underfitting, when your model is too simple to capture patterns in the data.

Performance sur données d'entraînement : MAUVAISE ✗
Performance sur nouvelles données       : MAUVAISE ✗
→ Le modèle n'a pas appris les patterns de base

The art of training a good model is finding the balance between these two extremes. We want a model complex enough to capture important patterns, but not so complex that it memorizes noise and irrelevant details. This is why we always test models on data that they have not seen during training.

graph TD
    A[Capacité du modèle] --> B{Trop simple ?}
    A --> C{Juste bien ?}
    A --> D{Trop complexe ?}
    B --> E[UNDERFITTING<br/>Mauvais sur train ET test]
    C --> F[BONNE GÉNÉRALISATION<br/>Bon sur train ET test]
    D --> G[OVERFITTING<br/>Excellent sur train<br/>Mauvais sur test]
    style E fill:#ffcccc,stroke:#f44336
    style F fill:#ccffcc,stroke:#4CAF50
    style G fill:#ffcccc,stroke:#f44336

1.6 Where deep learning fits in the machine learning landscape

graph TD
    ML[Machine Learning] --> SL[Supervised Learning<br/><i>On connaît la bonne réponse</i>]
    ML --> UL[Unsupervised Learning<br/><i>On ne connaît pas la bonne réponse</i>]

    SL --> CLS[Classification<br/><i>Chat ou chien ?</i>]
    SL --> REG[Regression<br/><i>Prédire un nombre<br/>ex: chance de pluie</i>]
    UL --> CLU[Clustering<br/><i>Grouper des points similaires</i>]

    CLS --> NN1[Neural Networks ✓]
    REG --> NN2[Neural Networks ✓]
    CLU --> NN3[Neural Networks ✓]

    style ML fill:#e3f2fd,stroke:#1565C0
    style SL fill:#e8f5e9,stroke:#2E7D32
    style UL fill:#fff3e0,stroke:#E65100
    style NN1 fill:#f3e5f5,stroke:#6A1B9A
    style NN2 fill:#f3e5f5,stroke:#6A1B9A
    style NN3 fill:#f3e5f5,stroke:#6A1B9A

Where does deep learning fit into the broader field of machine learning?

We have two types of machine learning: supervised and unsupervised. It basically comes down to the question: do you know the right answer to the problem you’re trying to solve? If so, you are using supervised learning. Otherwise, it’s unsupervised.

The main point is that neural networks appear in each of these categories. And deep learning simply means that we use neural networks with many layers. The term “deep learning” doesn’t tell you anything about the type of problem the model solves.

1.7 Neural networks — Detailed architecture

The artificial neuron

A neural network is made up of small computing units called neurons. These neurons are nothing like the neurons in your brain, which are real living cells that grow and respond to all kinds of signals and chemicals. A neuron in a neural network is really just a small mathematical function. It takes numbers as input, does a calculation, and produces a number as output. That’s all.

graph LR
    X1((x₁)) -->|w₁| SUM
    X2((x₂)) -->|w₂| SUM
    X3((x₃)) -->|w₃| SUM
    BIAS((b)) -->|+1| SUM
    SUM[Σ weighted sum<br/>z = w₁x₁ + w₂x₂ + w₃x₃ + b] --> ACT[f Activation<br/>function]
    ACT --> OUT((output a))
    style SUM fill:#fff9c4,stroke:#F9A825
    style ACT fill:#e8f5e9,stroke:#388E3C
    style OUT fill:#e3f2fd,stroke:#1565C0

Internal calculation of a neuron:

$$z = \sum_{i=1}^{n} w_i x_i + b = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n + b$$

$$a = f(z)$$

Where:

• $x_i$ = inputs
• $w_i$ = weights — represent the importance of each entry
• $b$ = bias — reference value
• $z$ = value before function activation
• $f$ = activation function
• $a$ = neuron output

Network layers

• Input layer: where you introduce your raw data — pixel values ​​from an image, or features from a dataset like the area or number of floors of a house for a network that learns real estate prices. These are simply numbers that are given to the input layer for processing.

• Hidden layers: Each takes the outputs of the previous layer and processes them further. In deep learning, there can be hundreds or even thousands. LLMs like ChatGPT can have hundreds of layers with thousands of neurons in each layer, leading to billions of connections.

• Output layer: Gives your final results — a predicted price for a house or a rating label for an image.

What each layer learns (example: cat classification):

Weights and biases

For each input, the neuron also has a separate number called weight which represents the importance of that input. Think of each weight as a volume knob:

• A high weight: “raise this signal, it is important”
• Low weight: “lower this signal, ignore it”

The neuron takes all of these incoming signals, multiplies each by its weight, and adds them all together. And there is one more thing in a neuron: bias. It is a value that is always added to the sum. It is like a base value or default setting that allows the neuron to shift its output up or down independently of all inputs. Without bias, if all inputs are zero, the output will also always be zero.

The process of training a network consists of adjusting the weights and biases. We can say that the weights represent the knowledge of the network, and learning means changing the value of each weight and bias so that the knowledge improves.

Activation functions

After calculating the weighted sum, the neuron also applies an activation function to the sum. The result determines the neuron’s output value.

Why activation functions are essential:

If we only had the weighted sum without activation function, the network could only learn linear relationships between inputs and outputs, which would be very limiting. Many real-world problems are nonlinear — the relationship between inputs and outputs is not just a straight line. Activation functions allow the network to learn these complex patterns.

Sigmoid function:

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

The sigmoid function transforms any number to a value between 0 and 1.

ReLU (Rectified Linear Unit) function:

$$f(z) = \max(0, z)$$

ReLU is often preferred in modern networks because it does not crush gradients in the same way as sigmoid, which helps solve the vanishing gradient problem.

graph LR
    subgraph "Sigmoid σ(z)"
        direction LR
        S1["z très négatif → 0"]
        S2["z = 0 → 0.5"]
        S3["z très positif → 1"]
    end
    subgraph "ReLU f(z)"
        direction LR
        R1["z < 0 → 0"]
        R2["z ≥ 0 → z"]
    end

1.8 Demo — TensorFlow Playground

The TensorFlow Playground (playground.tensorflow.org) is an excellent interactive tool that allows you to build and train simple neural networks directly in your browser.

What you can do in the Playground:

• Building networks with an input layer and an output layer
• Add and remove neurons for each layer
• Add additional layers
• See that all neurons in a layer are automatically connected to all nodes in the previous and next layers
• Hover over a connection to see the corresponding weight
• Select different activation functions

1.9 Network architectures

Fully Connected Perceptron (MLP)

What we have shown so far is called a fully connected perceptron, a fancy name for the network design where each neuron is connected to each neuron in the next layer. This is what “fully connected” means. Data flows from left to right.

This is the best known and simplest network architecture, and it works well for many problems, but it is not the only way to structure a neural network.

Convolutional Neural Network (CNN)

When working with images, we often use a Convolutional Neural Network (CNN). These networks have a connectivity pattern different from fully connected which is reminiscent of the functioning of the visual part of a biological brain.

• Hidden layers may include special types of layers such as convolutional layers which represent a mathematical operation called convolution, important in image processing
• The neurons do not receive input from the entire image at once, but instead there is a small sliding window called a kernel which scans the image for specific patterns
• Much more efficient than fully connected layers
• Works great for image processing because images have a spatial structure that CNNs are built to exploit

graph LR
    IMG[Image d'entrée<br/>ex: 28×28 pixels] --> CONV1[Convolutional Layer 1<br/>Kernel glissant<br/>Détection d'edges]
    CONV1 --> POOL1[Pooling Layer<br/>Sous-échantillonnage]
    POOL1 --> CONV2[Convolutional Layer 2<br/>Détection de features<br/>plus abstraites]
    CONV2 --> POOL2[Pooling Layer]
    POOL2 --> FC[Fully Connected Layer<br/>Classification]
    FC --> OUT[Output<br/>Chat / Chien]
    style IMG fill:#e3f2fd,stroke:#1565C0
    style CONV1 fill:#fff3e0,stroke:#E65100
    style CONV2 fill:#fff3e0,stroke:#E65100
    style POOL1 fill:#fce4ec,stroke:#880E4F
    style POOL2 fill:#fce4ec,stroke:#880E4F
    style FC fill:#e8f5e9,stroke:#1B5E20
    style OUT fill:#f3e5f5,stroke:#4A148C

Recurrent Neural Network (RNN)

For data sequences such as text or time series, Recurrent Neural Networks (RNNs) are used. These networks have connections that loop back on themselves, which means that the network has some sort of memory.

• It can process one element of a sequence and then use that result when processing the next element
• Information no longer flows strictly from left to right
• Perfect for tasks like language translation or predicting the next word in a sequence

graph LR
    X1["x₁ (mot 1)"] --> RNN1[RNN Cell]
    RNN1 -->|h₁ état caché| RNN2[RNN Cell]
    X2["x₂ (mot 2)"] --> RNN2
    RNN2 -->|h₂ état caché| RNN3[RNN Cell]
    X3["x₃ (mot 3)"] --> RNN3
    RNN3 --> OUT[Output<br/>Prédiction]
    RNN1 -.->|Mémoire / contexte| RNN2
    RNN2 -.->|Mémoire / contexte| RNN3
    style RNN1 fill:#e8f5e9,stroke:#2E7D32
    style RNN2 fill:#e8f5e9,stroke:#2E7D32
    style RNN3 fill:#e8f5e9,stroke:#2E7D32

Transformers

More recently, a new architecture called Transformer has revolutionized the field. This is what powers ChatGPT and many other cutting-edge language models.

• Specialized to manage data sequences very efficiently
• Proven incredibly effective for language tasks
• Based on an attention mechanism that allows the model to focus on different parts of the sequence

graph TD
    INPUT[Séquence d'entrée<br/>Tokens] --> EMB[Embedding Layer]
    EMB --> PE[Positional Encoding]
    PE --> ATTN[Self-Attention<br/>Multi-Head Attention]
    ATTN --> FFN[Feed-Forward Network]
    FFN --> NORM[Layer Normalization]
    NORM -->|N fois| ATTN
    NORM --> OUTPUT[Séquence de sortie]
    style ATTN fill:#f3e5f5,stroke:#6A1B9A
    style FFN fill:#e8f5e9,stroke:#1B5E20

Other methods (Kernel Methods, SVM)

Neural networks are not the only players at play. There are many other machine learning techniques that can be very effective for certain problems. A well-known family of methods is called kernel methods, with Support Vector Machines (SVM) being the most famous example.

These methods work in a very different way from neural networks, and for some smaller datasets or well-structured problems, kernel methods can outperform neural networks. They also tend to be quicker to train. However, they do not scale as well to very large and complex tasks like image recognition or language generation.

The black box problem and Explainable AI

You may have heard that it’s unclear how AI works, and that’s true. Once you train your network to do something — that is, we adjust the weights for each connection — we don’t know how it works. It’s a black box.

We know that data goes in and predictions go out, but what happens in between we really don’t know. At this point, you just have a bunch of numbers — all these weights, maybe millions or billions — and it would be impossible for a human to look at them and understand how any of these numbers correspond to something we can understand like a cat’s whiskers.

When is this a problem?

For many practical applications this does not matter. If the network gives you an accurate prediction and you can test it thoroughly, you don’t need to understand exactly how it arrives at those predictions.

But in other cases, like if the network is used to make a medical diagnosis or bank loan decisions, we would really like to know why the network made a particular decision.

This is an active area of ​​research called Explainable AI (XAI), and it is something worth considering when designing and deploying these systems.

Key Point: Neural networks are powerful, but they come with a tradeoff. You get great performance in exchange for losing some interpretability.

3. Training a Network

2.1 The loss function

Welcome to this module on training neural networks. Recall that a neural network is essentially a large collection of weights and biases. When you first create a network, these weights are usually initialized to small random values, and at this point the network knows nothing. When you give it an input, you get a random output.

Training is the process of adjusting weights and biases so that the network’s predictions gradually improve. But how do you know in which direction to adjust them and by how much?

When data is fed into a neural network, it produces a prediction, an output. In supervised learning, we already know the right answer. For each element in our training data, we can compare the network’s prediction to the actual response, and measure how incorrect it is.

This measure of network error is called the loss, and the mathematical formula we use to calculate it is called the loss function.

Mean Squared Error (MSE)

There are many different loss functions. Here is a common example, the Mean Squared Error (MSE):

$$\mathcal{L} = \frac{1}{n} \sum_{i=1}^{n} (\hat{y}_i - y_i)^2$$

Where:

• $n$ = number of examples in the dataset
• $\hat{y}_i$ = value predicted by the network
• $y_i$ = real value (correct answer)

What this formula says: For each example in our dataset, take the difference between what the network predicts and the actual value, square it to ensure it is always positive, then average these squared differences across all examples.

Concrete example of MSE calculation

Let’s imagine that our cat recognition network is trained on a training set of 3 images, and the network output is 1 if it is a cat, 0 otherwise.

$$\text{Total} = 0 + 1 + 0.04 = 1.04$$ $$\text{MSE} = \frac{1.04}{3} \approx 0.347$$

Key Insight: A small loss means our predictions are close to the actual values, and a large loss means they are far away. The goal during training is to minimize loss function — push it as low as possible.

2.2 Gradient descent

Basic Intuition (1 weight)

Let’s take a simplified example. Let’s imagine a network with a single weight. Our loss function could look like a curve. It represents the average error over a training set, as a function of the weight in our network.

Initially, the weight is at a random initial value. The value of the loss function tells us how well we are doing. During training, we change the weight to find the point where the loss is lowest.

To do this, we look at the slope of the function to find the direction going down. This can be done in several small steps.

Loss
  │
  │    ╮              
  │      ╲            Objectif :
  │       ╲    ╭─     trouver le minimum
  │        ╲  ╱       (plus petite loss)
  │         ╲╱        
  └──────────────── Weight
           ↑
       Minimum (optimal)

It is possible to exceed the target with a step too large. But the slope of the function then points in the opposite direction, and we start to come back. Generally, one will take smaller and smaller steps during training in the hope of finding the optimal value.

Extension to multiple dimensions

With a network with two weights, the loss function is in three dimensions. The loss now looks like a landscape. The objective is to find the point on this landscape where the loss is the smallest, and this point corresponds to the best set of weights for our network.

We always follow the slope of the function downward, although in many dimensions this is called the gradient. What we do is follow the gradient down to find the lowest point.

$$w_{new} = w_{old} - \alpha \cdot \nabla_w \mathcal{L}$$

Where:

• $\alpha$ = learning rate
• $\nabla_w \mathcal{L}$ = gradient of the loss compared to the weight

Blind Mountain Analogy

Imagine that you are standing in a hilly landscape, blindfolded. Your goal is to find the lowest point, the deepest valley floor. You can’t see it, but you can feel the slope of the ground beneath your feet. What would you do?

You would probably sense which way is going down and take a step in that direction. Then you would feel the slope again and take another step, and you would continue to do this until you reached a point where the ground seemed flat, meaning you had reached the bottom.

This is essentially what gradient descent does, except instead of a physical landscape, we’re talking about a mathematical landscape. And instead of your position, we adjust the weights of our neural network. The landscape represents how incorrect our network is, and we want to find the lowest point where the network is least incorrect.

In reality, this landscape has many more than two or three dimensions. A network with a million weights would have a million-dimensional landscape, impossible to visualize, but mathematically it makes no difference. The principle is the same.

2.3 Local vs. global minimum

There is a problem. When your learning problem becomes a little more interesting than the simplest case, the landscape of the loss will be quite rugged with lots of valleys and hills.

Loss
  │
  │   ╮         ╭╮
  │    ╲    ╭╮  ╱ ╲
  │     ╲  ╱  ╲╱   ╲   ╭─
  │      ╲╱              ╲╱
  │  Local↑          ↑Global
  │  minimum          minimum
  └──────────────────────── Weight

It is quite likely that by using gradient descent you will find a valley that is low, but not the lowest valley. This is called a local minimum as opposed to the global minimum, which is the actual lowest point.

This is one of the main problems to manage when using gradient descent: finding yourself stuck in a minimum room. This usually means that we get a pretty good result, and it seems like the best result we can get from our current position, but in reality, we can do better.

Unfortunately, for real problems there is usually no way to know whether you have found the global optimum or not.

Good news: In practice for deep neural networks, this problem turns out to be less serious than one might think. The landscape has so many dimensions that what looks like a valley in one direction might still have a path down in another direction. Researchers have found that for very large networks, most local minima tend to be quite close in quality to the global minimum.

2.4 Preparing the data

Before starting to train a neural network, we need to talk about data. It doesn’t matter how ingenious your network architecture is or how sophisticated your training algorithm — if your data isn’t prepared properly, your network is going to struggle.

There is a well-known saying in the machine learning world: “garbage in, garbage out”. If you give the network messy and poorly structured data, it will learn messy and poorly structured patterns.

Split train/test

How to test if our network has generalized well? Before training even begins, we take some of our data and put it aside. We hide it from the network during training and only use it afterwards to see how well the network performs on data it has never seen.

This gives us two groups:

• Training set: what the network learns from
• Test set: what we use to evaluate performance

A common split is something like 80% training / 20% test, although the exact ratio may vary depending on the amount of data available.

Why not use everything for training? If you use all your data for training, you have no way of knowing whether the network is truly generalizing or just memorizing. You need the separate test set as an independent verification.

Split train / validation / test

Many people go even further and divide the data into three parts:

graph LR
    DATA[Dataset complet] --> TRAIN[Training Set ~70%<br/>Le réseau apprend dessus]
    DATA --> VAL[Validation Set ~15%<br/>Surveillance pendant l'entraînement<br/>Décisions d'ajustement]
    DATA --> TEST[Test Set ~15%<br/>Évaluation finale<br/>Ne toucher qu'à la fin]
    style TRAIN fill:#e8f5e9,stroke:#2E7D32
    style VAL fill:#fff3e0,stroke:#E65100
    style TEST fill:#fce4ec,stroke:#880E4F

• Validation set: used during training to monitor network performance on data it has not been trained on, without touching the test set. Helps make decisions about when to stop training or when to adjust the approach.
• Test set: only used at the very end as a final unbiased evaluation.

2.5 Cross-validation

cross-validation is a technique that addresses a problem: during a single split train/test, we can be lucky or unlucky on which data ends up in which group. By chance, all the difficult examples could end up in the test set, making the model worse than it really is.

K-Fold Cross-Validation (example with K=5):

graph TD
    DATA[Dataset complet] --> F1[Fold 1]
    DATA --> F2[Fold 2]
    DATA --> F3[Fold 3]
    DATA --> F4[Fold 4]
    DATA --> F5[Fold 5]
    
    subgraph "Itération 1"
        T1[Train: Folds 2,3,4,5] --> M1[Modèle 1]
        M1 --> V1[Test: Fold 1 → Score 1]
    end
    subgraph "Itération 2"
        T2[Train: Folds 1,3,4,5] --> M2[Modèle 2]
        M2 --> V2[Test: Fold 2 → Score 2]
    end
    subgraph "..."
        T3[...] --> M3[...]
        M3 --> V3[... → Score N]
    end
    
    V1 & V2 & V3 --> AVG[Score moyen<br/>Estimation plus fiable]

Cross-validation divides the data into, say, five equal parts called folds. You then train the model five times, each time using a different fold as the test set and the remaining four folds as the training data. You end up with five different performance scores, and you can average them to get a more reliable estimate.

This gives a much more accurate picture of the model’s performance, but it takes longer since you are training the model multiple times. For deep learning, where training can take hours or even days, full cross-validation isn’t always practical, but it’s still a valuable concept to know.

2.6 Preprocessing your data

Real-world data is rarely in a neural network-ready format. They may have missing values, features at very different scales, or contain irrelevant information. Preprocessing is the step where we clean and transform the data to make it suitable for training.

Normalization and standardization

One of the most important preprocessing steps is normalization, which means scaling your features so that they are all in a similar range.

Problem without normalization: Let’s imagine two features:

• Feature A: 0 to 1 (a percentage)
• Feature B: 0 to 1,000,000 (a house price in dollars)

If given to a neural network without normalization, the network will struggle because the gradients for the large scale feature (the price) will be much larger than the gradients for the smaller numbers. The price gradient will dominate and the small-scale feature will barely have a say in the results.

Standardization (Z-score normalization):

$$x_{standardized} = \frac{x - \mu}{\sigma}$$

Where $\mu$ is the mean and $\sigma$ is the standard deviation. This results in a feature with a mean of 0 and a standard deviation of 1.

Min-Max Scaling:

$$x_{scaled} = \frac{x - x_{min}}{x_{max} - x_{min}}$$

This scales everything to the range [0, 1].

Logarithmic transformation

The logarithmic transformation is particularly useful when you have data that spans several orders of magnitude such as prices ranging from $10 to $10 million.

$$x_{log} = \log(x)$$

When you log such data:

• You compress these large ranges into a much more manageable scale
• The log transformation also tends to make skewed distributions more normal, which neural networks often prefer
• Particularly common in fields like finance, biology and physics where data naturally covers very wide ranges

Feature selection and feature engineering

Feature selection: Decide which features to include. Not all information in your dataset is useful for the task at hand. Including irrelevant features can actually hurt performance because the network could waste capacity trying to find patterns in this noisy data.

Feature engineering: The art of creating new features from your raw data. Sometimes combining existing features or transforming them in domain-specific ways can make the learning problem easier for a network. Although deep learning can learn some of it automatically, good feature engineering is still considered one of the most important skills in machine learning.

In summary — Pre-workout checklist:

✓ Diviser en training/validation/test sets
✓ Normaliser/standardiser toutes les features
✓ Gérer les valeurs manquantes
✓ Sélectionner les features pertinentes
✓ Appliquer les transformations nécessaires (log, etc.)
✓ Feature engineering si applicable

2.7 The backpropagation algorithm

How do you actually train a neural network? There is a specific approach to this called the backpropagation algorithm. We already know that training a network means calculating the gradient of the loss function and using it to adjust the weights. But how to calculate this gradient for a network with millions of weights distributed over many layers?

This is where backpropagation comes in. This is how we actually do gradient descent with neural networks. It gives us an efficient way to determine how much each individual weight contributes to the overall error, and allows us to adjust each weight in the right direction to minimize loss.

Forward pass

When data is fed into a network, the information flows forward through the layers from input to output. It’s called the forward pass. This is the normal way to use a network. We give it an input and we obtain an output, a prediction that we can compare to the expected response to calculate the loss.

graph LR
    IN[Input x] -->|w₁, b₁| H["Hidden Neuron<br/>z_h = w₁·x + b₁<br/>a_h = σ(z_h)"]
    H -->|w₂, b₂| OUT["Output Neuron<br/>z_o = w₂·a_h + b₂<br/>a_o = σ(z_o)"]
    OUT --> LOSS["Loss<br/>L = (a_o - y)²"]
    style IN fill:#e3f2fd,stroke:#1565C0
    style H fill:#fff3e0,stroke:#E65100
    style OUT fill:#e8f5e9,stroke:#2E7D32
    style LOSS fill:#fce4ec,stroke:#880E4F

Backward pass and chain rule

Backpropagation then works backwards. Starting with the loss at the output, it goes back through the network layer by layer, calculating how much each weight contributed to the error. This is the backward pass.

To do this, we use a mathematical operation called the partial derivative, which tells us how much each weight contributes to the loss for the current input. We can calculate this partial derivative by applying the chain rule.

$$\frac{\partial \mathcal{L}}{\partial w} = \frac{\partial \mathcal{L}}{\partial a_o} \cdot \frac{\partial a_o}{\partial z_o} \cdot \frac{\partial z_o}{\partial w}$$

This equation essentially says: to know how the weight $w$ affects the loss, we follow the chain of dependencies going back from the loss.

Step-by-step example on a microgrid

Here is an example of backpropagation on a microgrid with:

• 1 input
• 1 hidden neuron
• 1 neuron output

graph LR
    X((x = 0.5)) -->|w₁| HN["Hidden Neuron<br/>z_h = w₁·x + b₁<br/>a_h = σ(z_h)"]
    HN -->|w₂| ON["Output Neuron<br/>z_o = w₂·a_h + b₂<br/>a_o = σ(z_o)"]
    ON --> TARGET((Target y = 1))
    style X fill:#e3f2fd,stroke:#1565C0
    style HN fill:#fff3e0,stroke:#E65100
    style ON fill:#e8f5e9,stroke:#2E7D32
    style TARGET fill:#f3e5f5,stroke:#4A148C

Initial settings (example):

• Input $x = 0.5$
• Expected output $y = 1$
• Activation function: Sigmoid $\sigma(z) = \frac{1}{1+e^{-z}}$

Forward Pass:

1. Hidden neuron: $$z_h = w_1 \cdot x + b_1$$ $$a_h = \sigma(z_h)$$

2. Neuron output: $$z_o = w_2 \cdot a_h + b_2$$ $$a_o = \sigma(z_o) = 0.693 \text{ (example)}$$

3. Calculation of the loss (MSE with 1 example): $$\mathcal{L} = (a_o - y)^2 = (0.693 - 1)^2 = 0.13$$

The network predicts 0.693, but the correct answer is 1. The loss is 0.13.

2.8 Loss and weights — Calculation of gradients

The question now is: how to adjust the weights $w_1$ and $w_2$ to reduce this loss? This is the gradient descent part — trying to go down the slope to reach the point where the loss is minimal.

To do this, we must determine how much the loss is affected by each of the weights. This is where backpropagation comes in. We will calculate how much the loss changes when we make a small change at each weight. In calculus, this is called a partial derivative, and it tells us exactly how much to blame each weight for the error.

Gradient for w2 (output layer)

The chain rule tells us:

$$\frac{\partial \mathcal{L}}{\partial w_2} = \frac{\partial \mathcal{L}}{\partial a_o} \cdot \frac{\partial a_o}{\partial z_o} \cdot \frac{\partial z_o}{\partial w_2}$$

Each of these subparts has relatively simple calculations. By plugging in the calculated values, we obtain:

$$\frac{\partial \mathcal{L}}{\partial w_2} \approx -0.11$$

This result tells us: if we increase $w_2$ by a certain amount, the loss will decrease by approximately a tenth of this amount. So to reduce the loss, we must slightly increase this weight.

Gradient for w1 (hidden layer)

To go one level further back, we replace the last term with terms which follow the flow of information going back:

$$\frac{\partial \mathcal{L}}{\partial w_1} = \frac{\partial \mathcal{L}}{\partial a_o} \cdot \frac{\partial a_o}{\partial z_o} \cdot \frac{\partial z_o}{\partial a_h} \cdot \frac{\partial a_h}{\partial z_h} \cdot \frac{\partial z_h}{\partial w_1}$$

By calculating all these values:

$$\frac{\partial \mathcal{L}}{\partial w_1} \approx -0.012$$

This weight has a smaller gradient than that for the output neuron — only 0.012, which makes sense because it is further from the output, so its effect on the loss will be more indirect.

Update weights

Now that we have these gradients, we can use them to update the weights. We subtract a fraction of the gradient from each weight, and this fraction is called the learning rate $\alpha$.

$$w_{new} = w_{old} - \alpha \cdot \frac{\partial \mathcal{L}}{\partial w}$$

With $\alpha = 0.5$ (example):

$$w_2^{new} = w_2^{old} - 0.5 \times (-0.11) = w_2^{old} + 0.055$$ $$w_1^{new} = w_1^{old} - 0.5 \times (-0.012) = w_1^{old} + 0.006$$

Both gradients were negative, which means that the loss decreases as we increase the weights, so both weights increased a little to bring the loss down. If we now execute the forward pass with these new weights, we would obtain a prediction a little closer to 1 and a slightly lower loss.

Important note: The biases are also updated the same way, it’s just an extra step.

The vanishing gradient problem

Notice the values we have just calculated:

• Gradient for $w_2$: -0.11 (one tenth)
• Gradient for $w_1$: -0.012 (one hundredth)

It’s almost 10 times smaller, and this happens because of all those multiplications in the chain rule. Each time we go up a layer, we multiply by another set of terms, and many of these terms are small numbers, making the results even smaller.

Now imagine a network of not 2, but 50 or 100 layers. Each layer multiplies the gradient by another small number. After enough layers, these gradients become infinitesimally small, essentially zero. And when the gradient is zero, the weight is not updated at all, which means that the first layers of the network stop learning.

This is called the vanishing gradient problem, and it was one of the biggest obstacles to training deep networks for a long time.

Solutions:

• Use ReLU activation function instead of Sigmoid (does not compress gradients the same way)
• Special architectures like residual connections (skip connections) in ResNet networks
• Batch normalization

graph LR
    subgraph "Vanishing Gradient"
        G5["Gradient couche 5: 0.5"]
        G4["Gradient couche 4: 0.05"]
        G3["Gradient couche 3: 0.005"]
        G2["Gradient couche 2: 0.0005"]
        G1["Gradient couche 1: ≈ 0 ❌"]
        G5 --> G4 --> G3 --> G2 --> G1
    end

Backpropagation efficiency

You may have noticed that the calculation for $w_1$ included all the same terms as the calculation for $w_2$ plus a few extra. This is not a coincidence. This is exactly what makes backpropagation effective.

When we calculate the gradients from the output to the input, each layer reuses the results already calculated for the next layer, so we do not have to start from scratch for each weight. This makes backpropagation much faster than calculating each gradient independently, especially for networks with many layers.

In summary — The training cycle:

graph TD
    FP[Forward Pass<br/>Calcul de la prédiction] --> LOSS["Calcul de la Loss<br/>L = f(ŷ, y)"]
    LOSS --> BP[Backward Pass<br/>Backpropagation<br/>Chain Rule]
    BP --> UPD[Mise à jour des weights<br/>w = w - α·∇L]
    UPD --> FP
    style FP fill:#e3f2fd,stroke:#1565C0
    style LOSS fill:#fce4ec,stroke:#880E4F
    style BP fill:#fff3e0,stroke:#E65100
    style UPD fill:#e8f5e9,stroke:#2E7D32

2.9 Model performance

You have trained a network and you have seen the loss go down, but is the model really good? How do you know if it will perform on real data?

Loss curves

The most fundamental tool for monitoring training is the loss curve. This is simply a plot of loss over time, where the x-axis shows the number of training rounds (epochs) and the y-axis shows the loss value.

Loss
  │
  │╲
  │ ╲  Training Loss      ← Descend toujours
  │  ╲
  │   ╲__________
  │    ╲─────────────────  Validation Loss
  │
  └──────────────── Epochs

In a healthy workout, both curves go down together. The training loss might be slightly lower than the validation loss, and that’s okay — the network is directly optimized on the training data after all.

Signs of overfitting:

Loss
  │
  │╲          ╱ ← Validation loss remonte = OVERFITTING
  │ ╲        ╱
  │  ╲      ╱  ─── Validation Loss
  │   ╲────╱
  │    ╲──────────── Training Loss (continue de descendre)
  └──────────────── Epochs
           ↑
        Meilleur point (early stopping)

Signs of underfitting:

Loss
  │──────────────────── Validation Loss (haute et plateau)
  │═══════════════════ Training Loss (haute et plateau)
  └──────────────── Epochs

Early stopping

One way to handle overfitting is called early stopping. The idea is simple:

1. Monitor validation loss during training
2. Stop when it starts to rise
3. Use the weights of the point where the validation loss was lowest
4. This prevents the network from learning too much

Classification Metrics

Loss is useful for training, but when we want to communicate the performance of a model, we often use other metrics that are easier to interpret.

Accuracy:

$$\text{Accuracy} = \frac{\text{Number of correct predictions}}{\text{Total number of predictions}}$$

But accuracy can be misleading. Imagine a model to detect a rare disease that affects 1% of the population. A model that simply predicted “no disease” for everyone would be 99% accurate. It’s a terrible model, but the accuracy seems excellent.

This is where precision and recall come in:

Where:

• $TP$ = True Positives
• $FP$ = False Positives
• $FN$ = False Negatives

These two metrics are often in opposition:

• A very conservative model for predicting disease will have high accuracy, but low recall
• A model that signals everyone will have high recall, but low precision

F1 Score:

The F1 score combines precision and recall into a single number by taking their harmonic mean:

$$F1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

It is useful when you want a single metric that balances both.

Confusion matrix

The confusion matrix gives an even more detailed picture. It shows, for each class, how many examples were correctly classified and how many were confused with other classes.

Example of a confusion matrix for animal recognition:

On the diagonal, we see the correct classifications. All other numbers represent bad predictions. In this example, we can clearly see that the network performs poorly on rabbits because it classifies many of them as cats or dogs.

2.10 Regularization

When we detect overfitting, we want ways to prevent it. We’ve already mentioned early stopping, but there are other techniques collectively called regularization that help prevent the network from learning too much.

L2 Regularization (Weight Decay)

L2 regularization, also known as weight decay, adds a factor to the loss function that discourages the network from having very large weights.

$$\mathcal{L}{regularized} = \mathcal{L}{original} + \lambda \sum_i w_i^2$$

Where $\lambda$ is a hyperparameter that controls the strength of the regularization.

Why are large weights a problem? Large weights tend to make the network more sensitive to small changes in the input, which is a sign of overfitting. By adding a small penalty for large weights, we encourage the network to find a simpler solution.

Dropout

During training, the dropout randomly turns off a percentage of neurons in each layer by setting their output to zero.

This may seem counterintuitive — why would we deliberately break parts of our network? But the idea is that by randomly disabling neurons, we prevent the network from depending too much on a single neuron or a specific combination of neurons, and this forces it to learn more robust and general features.

Réseau normal :        Réseau avec Dropout (taux 50%) :
○─────○─────○          ○─────○─────○
│╲   │╲   │           │╲   ✗     │
│ ╲  │ ╲  │           │ ╲       │
○─────○─────○          ○─────✗─────○
│╲   │╲   │           │╲         │
○─────○─────○          ○─────○─────✗
                        (Neurons ✗ = désactivés aléatoirement)

Important: Dropout is used only during training. When using the network for real predictions, all neurons will be active.

Other regularization techniques

The right choice depends on your specific problem and your specific network architecture.

2.11 Optimizing backpropagation

We have already covered the core training loop: forward pass, calculate the loss function, backpropagation, then update the weights. But until now, some of these details have been treated as black boxes. In this section, let’s open up those boxes and talk about the settings and choices that can make the difference between a model that trains well and one that doesn’t.

Learning rate — most important hyperparameter

The learning rate is undoubtedly the most important parameter in training a neural network. Remember that during the gradient descent, we adjust the weights by subtracting a small fraction of the gradient. The learning rate is this fraction. It controls the size of the step we take each time we update the weights.

Trop petit learning rate :
Loss ╲
     ╲
      ╲         → Apprentissage, mais TRÈS lentement
       ──────────────────────
               
Trop grand learning rate :
Loss ╲    ╱╲   ╱╲
     ╲╱    ╲╱   ╲   → Oscille, overshoots, n'converge pas
     
Learning rate optimal :
Loss ╲
      ╲
       ╲─────
            ╲────── → Descend efficacement

If the learning rate is too small: The network will learn, but incredibly slowly. Each step only moves the weights a tiny amount, so it can require an enormous number of iterations to reach a good solution.

If the learning rate is too large: The steps are too large. Instead of descending gracefully into the low loss valley, the optimizer overshoots and bounces back. It does not converge.

Common hyperparameters

The learning rate is an example of what is called a hyperparameter. As opposed to regular parameters (weights and biases that the network learns), hyperparameters are parameters that you choose before training begins. The network does not adjust its own learning rate during training.

Batch size and gradient descent variants

There are several ways to determine when to update the weights:

The mini-batch gradient descent is the method most used in practice. It divides the training data into small batches and calculates the gradient on one batch at a time. This gives a good balance because the updates are quite stable (averaged over a batch) and the training is reasonably fast.

Advanced optimizers

Over the years, researchers have proposed many variations of basic backpropagation that can train faster and more reliably.

Momentum:

The idea is inspired by physics. Imagine a ball rolling downhill. She builds up speed as she goes, and with that speed she can roll through small bumps without getting stuck.

Standard gradient descent updates weights only on the current gradient, but with momentum, the update also takes into account the direction of previous updates. So if the weights have been moving in a certain direction for a while (rolling downhill), they speed up by taking larger update steps.

$$v_t = \beta v_{t-1} + \alpha \nabla_w \mathcal{L}$$ $$w_{new} = w_{old} - v_t$$

RMSprop (Root Mean Square Propagation):

Takes a different approach: adapt the learning rate for each weight individually. Weights that received large and consistent gradients obtain a smaller learning rate, while weights with small or inconsistent gradients obtain a larger learning rate.

Rprop (Resilient Backpropagation):

Completely ignores the magnitude of the gradient and only looks at the sign (positive or negative). If the gradient continues to point in the same direction, Rprop takes larger steps; if the gradient changes direction, it takes smaller steps.

Adam (Adaptive Momentum Estimation):

Adam is the default optimizer in deep learning. It essentially combines the ideas of momentum and RMSprop:

• Follows the direction of previous gradients (like momentum)
• Tracks the magnitude of these gradients (like RMSprop)

$$m_t = \beta_1 m_{t-1} + (1-\beta_1) \nabla_w \mathcal{L}$$ $$v_t = \beta_2 v_{t-1} + (1-\beta_2) (\nabla_w \mathcal{L})^2$$ $$w_{new} = w_{old} - \alpha \cdot \frac{m_t}{\sqrt{v_t} + \epsilon}$$

Where $\beta_1 \approx 0.9$, $\beta_2 \approx 0.999$, $\epsilon$ small to avoid division by zero.

Adam has become a sort of default choice in deep learning because it works well on a very wide range of problems with minimal tuning. If you’re not sure which optimizer to use, Adam is a solid place to start.

graph TD
    OPT[Optimiseurs de Neural Networks]
    OPT --> SGD[SGD de base<br/>Simple mais lent]
    OPT --> MOM[SGD + Momentum<br/>Accélération par inertie]
    OPT --> RMS[RMSprop<br/>LR adaptatif par weight]
    OPT --> ADAM[Adam ⭐<br/>Momentum + RMSprop<br/>Choix par défaut]
    OPT --> RPROP[Rprop<br/>Seulement le signe du gradient]
    style ADAM fill:#e8f5e9,stroke:#2E7D32,stroke-width:3px

Searching for hyperparameters

We have all these hyperparameters to define: the learning rate, the batch size, the network architecture, the choice of the optimizer, and more. How do we find the right combination for our problem?

Manual approach: The easiest is to try different values by hand and see what works. We start with common default values, train the model, check the results, then adjust the parameters. This method is quite effective for many problems, especially once one develops an intuition about which parameters are most important.

Grid Search: We define a set of values to try for each hyperparameter, then we train a model for each combination.

# Exemple de Grid Search conceptuel
learning_rates = [0.001, 0.01, 0.1]
batch_sizes    = [32, 64, 128, 256]
# 3 × 4 = 12 modèles à entraîner

This can become very computationally expensive if you have a lot of combinations to try.

Random Search: Often more efficient than grid search. Instead of trying each combination on the grid, we randomly sample combinations from the space of possible hyperparameter values. Research has shown that random search tends to find good hyperparameters more efficiently because it explores the space more uniformly.

graph LR
    subgraph "Grid Search"
        direction TB
        G1[LR=0.001, BS=32]
        G2[LR=0.001, BS=64]
        G3[LR=0.01, BS=32]
        G4[LR=0.01, BS=64]
        G5[...]
    end
    subgraph "Random Search"
        direction TB
        R1[LR=0.003, BS=48]
        R2[LR=0.07, BS=100]
        R3[LR=0.02, BS=256]
        R4[...]
    end
    style G1 fill:#fce4ec
    style G2 fill:#fce4ec
    style G3 fill:#fce4ec
    style G4 fill:#fce4ec
    style R1 fill:#e8f5e9
    style R2 fill:#e8f5e9
    style R3 fill:#e8f5e9

4. Summary and key concepts

Module 1 — Summary

Module 2 — Summary

5. Glossary

Notes generated from the full training transcript

Search Terms

deep · neural · networks · machine · data · science · gradient · network · backpropagation · loss · regularization · weights · calculation · descent · feature · function · hyperparameters · layer · layers · mse · overfitting · pass · split · test