Recurrent Neural Networks (RNNs)

Table of Contents

• General Introduction
• Fundamentals of RNNs for time series forecasting
• 1.1 Sequential data and RNN concepts for forecasting
• Time series data
• Limitations of classic feedforward networks
• The solution: the RNN and its hidden state
• Unrolling
• RNN Key Properties Summary
• 1.2 Implementation of a simple RNN model and comparison with a baseline
• General notebook configuration
• Multi-series data preparation
• RNNForecaster architecture
• Training loop
• Naive baseline: naive_last_value
• Evaluation metrics: MAE and RMSE
• Results and interpretation
• LSTM and GRU models for long time series
• 2.1 Behavior of the RNN on long series and gradient problems
• The problem of vanishing and exploding gradient
• Exploding gradient (W > 1)
• Vanishing gradient (W < 1)
• Experimental demonstration with short vs long contexts
• Introduction to LSTMs as a solution
• 2.2 Comparison of LSTM, GRU and RNN on electric load forecasting
• LSTMForecaster Architecture
• GRUForecaster architecture
• Hyperparameter tuning (split-grid)
• LSTM vs GRU vs RNN comparison results
• Trade-off analysis
• Model selection guide
• From RNNs to Attention and Transformers
• 3.1 Seq2Seq, Teacher Forcing and attention mechanisms
• Switch to multi-step Seq2Seq problem
• Truncated Backpropagation Through Time (TBPTT)
• Encoding and Decoding
• Teacher Forcing
• TinyTransformerForecaster Architecture
• Sinusoidal Positional Encoding
• Self-Attention and Encoder Transformer
• 3.2 Multi-step forecasting with Transformers
• Instantiation and architecture of the Transformer
• Rollout vs Direct Multi-step Output
• Final comparative results
• Drive dynamics analysis
• Conclusion and summary of model evolution
• Appendices
• Complete Math Equations
• Architecture diagrams
• Summary of hyperparameters used

1. General Introduction

Welcome to this training on Recurrent Neural Networks (RNNs). The goal is to build a strong intuition about how recurrent models handle sequential data, so that you can make good choices when deploying sequence models on real-world forecasting problems.

Throughout this training, the common thread is a concrete use case: you are part of a team responsible for building a forecast model for electricity demand. You have a dataset of energy consumption measured over time, and your mission is to predict what the next load will be so that the operational department can plan in advance.

The training gradually covers:

1. The fundamentals of RNNs and their application to time series forecasting
2. LSTMs and GRUs as solutions to the limitations of vanilla RNNs
3. The evolution towards Transformers and mechanisms of attention

2. Fundamentals of RNNs for time series forecasting

1.1 Sequential Data and RNN Concepts for Forecasting

Time series data

To understand why RNNs are useful, one must first understand the structure of time series data. In the context of electrical load forecasting:

• The X axis represents time instances (hours, days, weeks depending on the context)
• The Y axis represents the electrical charge (Low / Medium / High in a simplified version; actual numerical values in the dataset)

This type of data is called a time series because the values ​​are ordered in time, and this ordering is the essence of the problem: what happened before helps explain what will happen next.

We find this same pattern in other problems:

• Stock Price Data
• Server traffic
• Temperature readings

Example with two clients:

• Client 1 (purple line): consumption rises for a few time steps, falls, then rises again.
• Client 2 (pink line): data only available from a given time. Maybe this customer is new, or the collection of clean data has only started halfway.

Key point: We may have more history for one customer than for another. The neural network used must be able to work with variable amounts of sequential data.

To predict the consumption of the purple client at time step 6, we want to use the previous 5 time steps. For the pink client, we only have 3 time steps available. The model must therefore be flexible in the amount of sequential data used to make a prediction.

Limitations of classic feedforward networks

A typical feedforward neural network has an Input layer, a Hidden layer and an Output layer. It’s a powerful tool, but it has fundamental limitations for sequential data:

1. Information only flows forward. Once an input passes through the network, there is no built-in mechanism for that calculation to propagate to the next input in the sequence.

2. No time state. Each entry is treated as a new start. The network cannot naturally build a cumulative understanding of what it has seen over time.

3. Fixed size context. If we want the model to know the past, we must manually include this past in the current input, which quickly becomes complex and forces the use of a fixed size context.

The solution: the RNN and its hidden state

The key question is: how do we get a neural network to be able to use previous information to affect subsequent decisions?

The answer: add a loop in the neural network which allows previous information to be passed forward. This is exactly what a Recurrent Neural Network (RNN) does.

An RNN has a loop mechanism which acts as a path allowing information to flow from one time step to the next. This is called the hidden state, which is a representation of all previous inputs.

flowchart LR
    X["x_t\n(entrée courante)"] --> RNN["Cellule RNN"]
    H_prev["h_{t-1}\n(hidden state précédent)"] --> RNN
    RNN --> H_next["h_t\n(nouveau hidden state)"]
    RNN --> Y["ŷ_t\n(sortie/prédiction)"]
    H_next -->|"boucle récurrente"| H_prev2["h_t → h_{t+1}"]

The hidden state update is defined by:

$$h_t = \tanh(W_{hh} \cdot h_{t-1} + W_{xh} \cdot x_t + b_h)$$

And the model output:

$$\hat{y}t = W{hy} \cdot h_t + b_y$$

Where:

• $x_t$ is the input at time step $t$
• $h_{t-1}$ is the hidden state of the previous time step
• $W_{hh}$ is the hidden-to-hidden weight matrix
• $W_{xh}$ is the input-to-hidden weight matrix
• $W_{hy}$ is the hidden-to-output weight matrix
• $b_h$, $b_y$ are the biases

Unrolling

To understand how the RNN works, we can unroll it over time. Let’s take the example with 5 time steps (like the purple client):

flowchart LR
    X1["x₁\n(t=1)"] --> RNN1["RNN\nh₀→h₁"]
    X2["x₂\n(t=2)"] --> RNN2["RNN\nh₁→h₂"]
    X3["x₃\n(t=3)"] --> RNN3["RNN\nh₂→h₃"]
    X4["x₄\n(t=4)"] --> RNN4["RNN\nh₃→h₄"]
    X5["x₅\n(t=5)"] --> RNN5["RNN\nh₄→h₅"]
    RNN1 -->|"h₁"| RNN2
    RNN2 -->|"h₂"| RNN3
    RNN3 -->|"h₃"| RNN4
    RNN4 -->|"h₄"| RNN5
    RNN5 --> Y["ŷ₆\n(prédiction t=6)"]
    style Y fill:#f9f,stroke:#333

RNNs work sequentially: they are provided one input at a time.

1. Time step 1: energy is supplied at time step 1, the network produces an output
2. Time step 2: we supply the energy at time step 2 and the hidden state of the previous step. At this point, the RNN has information on steps 1 and 2
3. Repetition until the final step, where it becomes clear that the RNN has accumulated information from all previous steps
4. At this point, we obtain the prediction of the electric charge at time step 6

Important note: The same problem can be formulated as classification — we simply pass the last hidden state into a feedforward network trained to predict a class label rather than a numeric value.

Key Benefit — Weight Sharing:

Weights are shared across time. $W_1$ and $W_2$ (input-to-hidden and hidden-to-hidden weights) are reused at each time step. We apply the same transformations learned over and over again as we move forward in the sequence. This allows the model to:

• Working naturally with sequences of variable length
• Maintain a controlled number of parameters
• Reduce the risk of overfitting (fewer parameters = less risk of memorizing noise)
• Better generalize on new unseen sequences

Summary of key RNN properties

Conceptually, an RNN is a feedforward network with memory. It processes data step by step, updates its internal state, and uses this evolving state to make predictions about what comes next.

1. Hidden temporal state

• The hidden state allows the model to transport information from previous time steps to the current step
• The network maintains an evolving internal representation of what it saw as it moves forward in the sequence

2. Natural management of variable length sequences

• Whether you have 5 time steps for one client or 3 for another, the same architecture can handle both without redrawing the model

3. Weight sharing over time

• The network applies an identical set of learned transformations moving forward in the sequence
• This keeps the number of parameters consistent over time and allows the model to generalize efficiently across sequences of different lengths

1.2 Implementation of a simple RNN model and comparison with a baseline

General notebook configuration

The notebook is organized with separate modules for:

• Data management and preparation (data)
• The model (models)
• The baselines (baselines)
• Metrics
• Training

Here is the basic configuration:

import torch
import numpy as np

# Reproductibilité
SEED = 42
torch.manual_seed(SEED)
np.random.seed(SEED)

# Visualisation : un seul client pour les plots
CUSTOMER_ID = "MT_001"

# ----- Configuration de la fenêtre glissante (Window setup) -----
window_length = 24   # Regarder les 24 dernières valeurs horaires (1 journée complète de contexte)
horizon = 1          # Prédire 1 seul pas en avant (la prochaine heure)
stride = 4           # Avancer la fenêtre de 4h à chaque échantillon d'entraînement

# ----- Configuration du modèle -----
batch_size = 64
hidden_size = 64     # Taille du hidden state (espace de représentation interne)

# ----- Configuration du split -----
frequency = "h"            # Fréquence horaire
prediction_length = 24     # 1 jour réservé pour la validation
rolling_evaluation = 7     # 1 semaine de couverture de test par rolling

# ----- Configuration de l'entraînement -----
n_epochs = 10
learning_rate = 0.001
weight_decay = 0.0    # Pas de regularisation pour ce run de base

# ----- Chemin des données -----
data_path = "./data/electricity"

Note on the stride:

• A stride of 1 gives more windows and denser coverage, but can increase the training size and slow things down
• A stride of 4 is a practical compromise: we advance by 4 hours each time

Note on hidden_size:

• This is the size of the hidden state, the representation that carries the information passed through the recurrent calculation

Multi-series data preparation

The central data preparation function is prepare_multi_series_data. It uses public electricity consumption data (adapted from the Hugging Face processing flow for this dataset).

# Aperçu simplifié de prepare_multi_series_data
def prepare_multi_series_data(data_path, window_length, horizon, stride, 
                               frequency, prediction_length, rolling_evaluation,
                               batch_size):
    
    # 1. Chargement des données et calcul des frontières train/val/test
    frame, train_end, val_end = build_frame_and_boundaries(
        data_path, frequency, prediction_length, rolling_evaluation
    )
    
    # 2. Construction des fenêtres par client et fusion
    #    Chaque série commence à son premier timestamp non-nul
    #    → différents clients peuvent avoir des historiques de longueurs différentes
    #    → ce n'est PAS un problème pour le RNN
    
    # 3. Mise à l'échelle (scaling) en utilisant UNIQUEMENT la tranche d'entraînement
    #    pour chaque client → évite la fuite (leakage) de val/test dans le preprocessing
    
    # 4. Conversion en fenêtres supervisées + split par timestamp
    
    # 5. Chargement paresseux (lazy loading) pour l'entraînement via MultiSeriesTrainDataset
    #    → pas de matérialisation de toutes les fenêtres en mémoire
    #    → génération à la demande → gestion mémoire efficace même avec des millions de fenêtres
    
    return train_loader, val_loader, test_loader

Important points:

• 370 customers in full dataset
• Over 2.6 million training windows
• Each series starts at its first non-zero timestamp: exactly the situation described in the slides — different clients can be active at different times and therefore have histories of different lengths. The RNN handles this naturally.
• Scaling uses only the training slice for each client → avoids any leakage of validation/test data

RNNForecaster architecture

import torch
import torch.nn as nn

class RNNForecaster(nn.Module):
    """
    RNN vanilla pour la prévision de séries temporelles.
    
    Architecture :
    - 1 couche récurrente (nn.RNN)
    - 1 tête linéaire (Linear head) pour la prédiction
    """
    def __init__(self, input_size: int, hidden_size: int, 
                 horizon: int, num_layers: int = 1):
        super().__init__()
        self.rnn = nn.RNN(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True
        )
        self.head = nn.Linear(hidden_size, horizon)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x : (batch, seq_len, input_size)
        out, _ = self.rnn(x)
        # Prendre la dernière sortie cachée à travers le temps
        last_hidden = out[:, -1, :]   # (batch, hidden_size)
        return self.head(last_hidden)  # (batch, horizon)

Checking the number of trainable parameters: 4,353

Decomposition:

• Input-to-hidden matrix: input_size × hidden_size = 1 × 64 = 64
• Hidden-to-hidden matrix: hidden_size × hidden_size = 64 × 64 = 4096
• Recurrent layer bias: 2 × hidden_size = 128 (input bias + hidden bias)
• Linear head (hidden → horizon): hidden_size × horizon + horizon = 64 × 1 + 1 = 65
• Total: 64 + 4096 + 128 + 65 = 4,353 ✓

Note: The recurrent loop itself is handled inside nn.RNN, where a hidden state is propagated step by step through the sequence.

Training loop

import torch.optim as optim

def train_model(model, train_loader, val_loader, n_epochs, 
                learning_rate, weight_decay=0.0):
    """
    Boucle d'entraînement standard pour un forecaster récurrent.
    
    Pour chaque batch :
    1. Forward pass
    2. Calcul de la MSE loss
    3. Backward pass (Backpropagation Through Time - BPTT)
    4. Mise à jour avec Adam
    
    Tracking : train_loss, val_loss, gradient_norm par epoch
    """
    optimizer = optim.Adam(model.parameters(), 
                           lr=learning_rate, 
                           weight_decay=weight_decay)
    criterion = nn.MSELoss()
    
    history = {
        "train_loss": [],
        "val_loss": [],
        "grad_norm": []
    }
    
    for epoch in range(n_epochs):
        model.train()
        train_losses = []
        grad_norms = []
        
        for batch_x, batch_y in train_loader:
            optimizer.zero_grad()
            
            # 1. Forward pass
            output = model(batch_x)
            
            # 2. MSE loss
            loss = criterion(output, batch_y)
            
            # 3. Backward pass (BPTT se passe ici)
            loss.backward()
            
            # Tracking du gradient norm
            grad_norm = sum(
                p.grad.norm().item() ** 2
                for p in model.parameters()
                if p.grad is not None
            ) ** 0.5
            
            # 4. Adam optimizer step
            optimizer.step()
            
            train_losses.append(loss.item())
            grad_norms.append(grad_norm)
        
        # Évaluation validation (sans gradients)
        model.eval()
        val_losses = []
        with torch.no_grad():
            for batch_x, batch_y in val_loader:
                output = model(batch_x)
                loss = criterion(output, batch_y)
                val_losses.append(loss.item())
        
        history["train_loss"].append(np.mean(train_losses))
        history["val_loss"].append(np.mean(val_losses))
        history["grad_norm"].append(np.mean(grad_norms))
        
        print(f"Epoch {epoch+1}/{n_epochs} | "
              f"Train Loss: {history['train_loss'][-1]:.4f} | "
              f"Val Loss: {history['val_loss'][-1]:.4f} | "
              f"Grad Norm: {history['grad_norm'][-1]:.4f}")
    
    return history

Base run results (10 epochs):

• The train loss goes down from 0.14 to 0.07
• Validation loss remains in the same general band with some fluctuations
• A noticeable jump towards the middle of the workout, then drops back down — stable curve, no divergence or collapse

Warning: Depending on stride settings and other configurations, training may take several hours. Take this into account during your experiments.

Naive baseline: naive_last_value

To assess whether the model learns something useful, we compare it to a simple persistence baseline:

def naive_last_value(windows: torch.Tensor, horizon: int) -> torch.Tensor:
    """
    Baseline de persistance : prédit que la prochaine valeur sera égale
    à la dernière valeur observée dans la fenêtre.
    
    C'est une baseline simple mais très utile en pratique.
    """
    last_val = windows[:, -1:]           # dernière valeur de chaque fenêtre
    return last_val.expand(-1, horizon)  # répéter sur tout l'horizon

We also use the inverse_transform_targets_by_item function:

def inverse_transform_targets_by_item(scaled_preds, scaled_targets, 
                                       scalers_by_item, item_ids):
    """
    Convertit les cibles et prédictions mis à l'échelle vers les unités de charge 
    originales, en utilisant le scaler correct pour chaque client.
    
    Ainsi, les métriques sont dans les unités réelles et directement interprétables.
    """
    real_preds = {}
    real_targets = {}
    for item_id in item_ids:
        scaler = scalers_by_item[item_id]
        real_preds[item_id] = scaler.inverse_transform(scaled_preds[item_id])
        real_targets[item_id] = scaler.inverse_transform(scaled_targets[item_id])
    return real_preds, real_targets

Evaluation Metrics: MAE and RMSE

Two metrics are used to evaluate performance:

$$MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

$$RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$$

Why use both together?

Using the two together is useful in load forecasting, because the MAE gives a clear idea of ​​the typical deviation, while the RMSE makes it easier to see if the model is still producing large point errors.

Results and interpretation

Results table (base run):

The recurring model obtains a lower MAE and a lower RMSE than the baseline naive_last_value. This means two things simultaneously:

1. On average, predictions are closer to true load values
2. The model produces fewer large errors

Even with a single recurrent layer and a short input window, the model already extracts useful temporal structure that a simple persistence rule cannot capture.

Curve analysis:

• Training loss curve: descends steadily through epochs → optimization works as expected
• Validation loss curve: stays higher and shows noise, including a noticeable jump around mid-training, then levels off — stable, no divergence

Analysis of predictions vs reality (plot by customer):

• Around abrupt changes and peaks: the recurring model tends to follow the direction of the series more closely, while the naive baseline visibly lags behind
• In flatter regions: both may appear similar
• There are also times when the recurring pattern overshoots

Conclusion: This is the type of result we want to see here. The recurrent model improves compared to a simple baseline, but it is still far from perfect. It captures short-term changes more effectively, while still making noticeable errors — which is expected for such a small and simple setup.

3. LSTM and GRU models for long time series

2.1 Behavior of the RNN on long series and gradient problems

We saw how to use the feedback loop in a vanilla RNN to unfold a network that works well with different amounts of sequential data. Now let’s discuss one reason why vanilla RNNs are not widely used in practice.

The problem of vanishing and exploding gradient

The fundamental problem: The more you unroll an RNN, the harder it is to train. This is called the Vanishing and Exploding Gradient Problem. This is linked to the weights of the loop which are reused each time it is unrolled.

For ease of understanding, let’s ignore other weights and biases and focus only on W (the hidden-to-hidden weight matrix).

Exploding gradient (W > 1)

If $W > 1$, for example $W = 2$:

After $T$ no running time, the signal is multiplied by:

$$\text{signal} \propto W^T = 2^T$$

This number gets huge very quickly. This is why we call it the explosive gradient problem.

During backpropagation optimization, derivatives (gradients) are calculated and used in the gradient descent algorithm to update the parameters. When this huge number ends up in some gradients, the updates become too large to take small, measured steps toward the right weights and biases.

The more the network is unrolled, the worse it is, because the gradients must travel backwards through more steps.

Vanishing gradient (W < 1)

If $W < 1$, for example $W = 0.5$:

$$\text{signal} \propto W^T = 0.5^T$$

Instead of taking steps too big, we end up taking steps too small. The gradients evaporate, and the model can no longer learn long-term dependencies.

graph LR
    subgraph "Exploding Gradient (W=2)"
        direction LR
        E1["W=2\nstep 1\n×2"] --> E2["step 2\n×4"] --> E3["step 3\n×8"] --> E4["step T\n×2^T 💥"]
    end
    subgraph "Vanishing Gradient (W=0.5)"
        direction LR
        V1["W=0.5\nstep 1\n×0.5"] --> V2["step 2\n×0.25"] --> V3["step 3\n×0.125"] --> V4["step T\n×0.5^T ≈0"]
    end

Experimental demonstration with short vs long contexts

To make this phenomenon concrete, we compare the behavior of the RNN with:

• Short context: 24 hour window
• Long context: full week window (24 × 7 = 168 hours)

Everything else remains stable so that context length is the only variable tested.

# Configuration du test
# Contexte court : 24h
short_window_config = {
    "window_length": 24,
    "horizon": 1,
    "stride": 4
}

# Contexte long : 1 semaine
long_window_config = {
    "window_length": 24 * 7,   # 168 heures
    "horizon": 1,
    "stride": 4
}

# Note : item_ids utilise un sous-ensemble fixe pour garder les temps d'itération pratiques
# Si item_ids=None, le même pipeline tourne sur tous les clients disponibles

Results after 3 epochs:

Plot Validation Loss:

• Short context curve: continues to descend slowly over the epochs
• Long context curve: does the opposite — continues to rise over epochs. With more history, the model does not gain confidence; it becomes harder to optimize correctly, and validation behavior drifts in the wrong direction.

Plot Gradient Norm:

• Short context: stays in a tight band
• Long context: rises sharply, ending very high. This is the pressure of the long context window showing up — updates become much larger as the window gets longer.

It wasn’t an explosive gradient crash for this run, but it’s a very different regime than the short window — and we can see the trend we talked about in the slides manifesting.

Where the backpropagation happens in the code:

# Dans train_model, c'est cette seule ligne qui déclenche la BPTT :
loss.backward()
# ^ Cette ligne unique est là où se passe la Backpropagation Through Time.
# Le RNN a été déroulé sur toute la fenêtre, et le passage backward doit 
# pousser les gradients en arrière à travers chaque étape de cette fenêtre.

Think of it as a chain of multiplications. When we multiply a factor through many additional steps:

• Small shrink effects shrink more
• Small growth effects grow bigger

A longer window gives more steps, so gradients have more opportunities to explode or evaporate.

Introduction to LSTMs as a solution

The solution to this problem is Long Short-Term Memory (LSTM), an extension of vanilla RNN designed to avoid this problem.

The basic idea of ​​LSTM:

Instead of using the same feedback loop connection for events that happened a long time ago or just one step behind to make a prediction about the next time step, LSTMs use separate paths to carry the information forward:

• A path for long-term memories (cell state)
• A path for short-term memories (hidden state)

And the important part: LSTMs use gates to control what flows through these paths. Gates decide what to keep and what to forget, so that useful information can move forward without being multiplied again and again with each step. This gives the gradients a much more stable path through time.

2.2 Comparison of LSTM, GRU and RNN on electrical load forecasting

Now that we have reached a practical limit with vanilla RNNs, let’s look at the next two recurrent models that we actually use in practice: LSTMs and Gated Recurrent Units (GRUs).

LSTMForecaster architecture

class LSTMForecaster(nn.Module):
    """
    LSTM Forecaster pour la prévision de séries temporelles.
    
    Le LSTM utilise deux états :
    - cell state (C_t) : mémoire à long terme, chemin de gradient stable
    - hidden state (h_t) : représentation à court terme exposée en sortie
    
    Quatre gates contrôlent le flux d'information :
    - Forget gate (f_t) : quoi oublier du cell state
    - Input gate (i_t) : quoi écrire dans le cell state
    - Cell candidate (C̃_t) : nouvelles informations potentielles
    - Output gate (o_t) : quoi exposer comme hidden state
    """
    def __init__(self, input_size: int, hidden_size: int, 
                 horizon: int, num_layers: int = 1):
        super().__init__()
        self.lstm = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True
        )
        self.head = nn.Linear(hidden_size, horizon)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x : (batch, seq_len, input_size)
        out, (h_n, c_n) = self.lstm(x)
        # h_n : dernier hidden state, c_n : dernier cell state
        last_hidden = out[:, -1, :]    # (batch, hidden_size)
        return self.head(last_hidden)  # (batch, horizon)

LSTM equations:

$$f_t = \sigma(W_f \cdot [h_{t-1},, x_t] + b_f) \quad \text{(forget gate)}$$

$$i_t = \sigma(W_i \cdot [h_{t-1},, x_t] + b_i) \quad \text{(input gate)}$$

$$\tilde{C}t = \tanh(W_C \cdot [h{t-1},, x_t] + b_C) \quad \text{(candidate cell)}$$

$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \quad \text{(cell state update)}$$

$$o_t = \sigma(W_o \cdot [h_{t-1},, x_t] + b_o) \quad \text{(output gate)}$$

$$h_t = o_t \odot \tanh(C_t) \quad \text{(hidden state update)}$$

Where $\sigma$ is the sigmoid function and $\odot$ is the element-by-element product (Hadamard product).

flowchart TD
    subgraph LSTM["Cellule LSTM"]
        direction TB
        CT_prev["C_{t-1}\n(cell state long terme)"] --> FG_mult["× f_t"]
        FG_mult --> CT_new["C_t"]
        
        IG_mult["× i_t"] --> CT_new
        
        X["x_t"] --> FG["Forget Gate\nσ(W_f·[h,x])"]
        H_prev["h_{t-1}"] --> FG
        FG --> FG_mult

        X --> IG["Input Gate\nσ(W_i·[h,x])"]
        H_prev --> IG
        IG --> IG_mult

        X --> CC["Cell Candidate\ntanh(W_C·[h,x])"]
        H_prev --> CC
        CC --> IG_mult

        CT_new --> tanh_block["tanh(C_t)"]
        X --> OG["Output Gate\nσ(W_o·[h,x])"]
        H_prev --> OG
        OG --> OG_mult["× o_t"]
        tanh_block --> OG_mult
        OG_mult --> HT["h_t\n(hidden state)"]
    end

    style CT_prev fill:#ffd,stroke:#888
    style CT_new fill:#ffd,stroke:#888
    style HT fill:#dfd,stroke:#888

Key advantage of LSTM: The cell state provides a clean path for information to persist across many time steps, which helps with long-term dependencies and stabilizes the gradient flow compared to an RNN with simple hyperbolic tanh.

GRUForecaster architecture

class GRUForecaster(nn.Module):
    """
    GRU Forecaster - version simplifiée et allégée du LSTM.
    
    Le GRU utilise deux gates :
    - Reset gate (r_t) : contrôle comment l'état passé influence l'état candidat
    - Update gate (z_t) : contrôle la balance entre ancien et nouvel état
    
    Pas de cell state séparé : le comportement mémoire est intégré dans le hidden state.
    → Moins de paramètres pour la même taille de hidden state
    → Entraînement souvent plus simple
    """
    def __init__(self, input_size: int, hidden_size: int, 
                 horizon: int, num_layers: int = 1):
        super().__init__()
        self.gru = nn.GRU(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            batch_first=True
        )
        self.head = nn.Linear(hidden_size, horizon)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        out, _ = self.gru(x)
        last_hidden = out[:, -1, :]
        return self.head(last_hidden)

GRU equations:

$$r_t = \sigma(W_r \cdot [h_{t-1},, x_t]) \quad \text{(reset gate)}$$

$$z_t = \sigma(W_z \cdot [h_{t-1},, x_t]) \quad \text{(update gate)}$$

$$\tilde{h}t = \tanh(W \cdot [r_t \odot h{t-1},, x_t]) \quad \text{(hidden state candidate)}$$

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \quad \text{(hidden state update)}$$

Differences between LSTM and GRU:

GRU was introduced as a lighter alternative to LSTM. Instead of having a separate cell state plus three gates, the GRU uses two main gates and integrates the memory behavior into the hidden state itself.

Hyperparameter tuning (split grid)

The key discipline when comparing models is to keep a research budget aligned — same set of candidate configurations, same number of trials — so as not to accidentally give a model an advantage just because it received more tuning.

# Grille d'hyperparamètres partagée pour les 3 modèles
param_grid = {
    "hidden_size": [64, 96, 128],
    "num_layers": [1, 2],
}

# Fixé pour tous les modèles (pas de knobs à tourner)
fixed_params = {
    "learning_rate": 0.001,
    "weight_decay": 0.0,
}

# Seulement 2 epochs pendant le tuning (signal directionnel, pas un run complet)
tuning_epochs = 2

# Boucle de tuning pour les 3 modèles
models_to_tune = {
    "RNN": RNNForecaster,
    "LSTM": LSTMForecaster,
    "GRU": GRUForecaster,
}

tuning_results = []

for model_name, ModelClass in models_to_tune.items():
    for hidden_size in param_grid["hidden_size"]:
        for num_layers in param_grid["num_layers"]:
            config = {
                "hidden_size": hidden_size,
                "num_layers": num_layers,
                **fixed_params
            }
            
            model = ModelClass(
                input_size=1,
                hidden_size=hidden_size,
                horizon=horizon,
                num_layers=num_layers
            )
            
            start_time = time.time()
            history = train_model(
                model, train_loader, val_loader,
                n_epochs=tuning_epochs,
                learning_rate=fixed_params["learning_rate"]
            )
            elapsed = time.time() - start_time
            
            tuning_results.append({
                "model": model_name,
                "hidden_size": hidden_size,
                "num_layers": num_layers,
                "val_loss": history["val_loss"][-1],
                "grad_norm": history["grad_norm"][-1],
                "training_time_s": elapsed
            })

Tuning duration: Approximately half an hour for 6 hyperparameter combinations × 3 models = 18 runs in total.

Important note: We don’t dwell too much on the tuning validation numbers because we only trained 2 epochs. The real point is to choose a reasonable setup for each model and then make the real comparison with a longer workout.

Best configurations found by tuning:

LSTM vs GRU vs RNN comparison results

After instantiating each model with its best hyperparameters, we train for 6 epochs (complete training):

# Instanciation avec les meilleurs hyperparamètres du tuning
best_rnn = RNNForecaster(input_size=1, hidden_size=128, horizon=1, num_layers=1)
best_lstm = LSTMForecaster(input_size=1, hidden_size=96, horizon=1, num_layers=2)
best_gru = GRUForecaster(input_size=1, hidden_size=96, horizon=1, num_layers=2)

# Entraînement complet
for model, name in [(best_rnn, "RNN"), (best_lstm, "LSTM"), (best_gru, "GRU")]:
    history = train_model(
        model, train_loader, val_loader,
        n_epochs=6,
        learning_rate=0.001
    )

Results on test set:

Analysis of results:

• LSTM obtains the strongest RMSE and MAE, indicating that it better captures the magnitude and timing of demand variations more consistently
• GRU: its MAE is better than the vanilla RNN, but its RMSE is slightly worse
• RNN vanilla is still surprisingly competitive in RMSE, despite its known fragility in terms of training dynamics

Why can GRU beat RNN on MAE but lose on RMSE?

A useful way to think about it:

• MAE weights all errors linearly
• RMSE penalizes large errors much more heavily

If the GRU is generally more stable and reduces the typical error most of the time, then the MAE can improve. But if it occasionally misses a sharp spike a little more than the RNN on a handful of points, then the RMSE may look worse, even if the overall daily error is better.

Analysis of trade-offs

Detailed model comparison:

Analysis by component:

1. Number of parameters: Gating adds additional weight matrices — that’s the cost you pay. The separation is clear: RNN has by far the fewest parameters, then GRU as expected, and LSTM has the most.

2. Training time: The RNN is much faster, which makes sense given its small size. Between GRU and LSTM, the times are in the same ballpark. In this run, the GRU is actually slower, even though it has fewer parameters. Training time depends on more than just the number of parameters.

3. Standard gradient: The norm gradient of the vanilla RNN is clearly higher — consistent with what we saw. It’s just more prone to unstable gradients when backpropping through long unwinds. The GRU has the lowest gradient norm and the LSTM is in the middle.

Cautionary interpretation: A lower norm gradient in itself is not necessarily better, but in this context it is a decent signal that the optimization is behaving smoother and that one is less likely to see the kind of explosive behavior that makes the training fragile.

The GRU — a good example of the compromise:

• Fewer parameters → smaller hypothetical space → can help reduce the risk of overfitting
• But → can also limit how finely the model separates overlapping temporal patterns

Model Selection Guide

Important note: Don’t treat this as a model being universally better. It really depends on what matters to you when deploying.

4. From RNNs to Attention and Transformers

3.1 Seq2Seq, Teacher Forcing and attention mechanisms

We now continue with the evolution of these sequence models. This module introduces a slightly different configuration and the model you’ve probably heard of: the Transformer.

Switch to multi-step Seq2Seq problem

Key change in configuration:

# Configuration Module 3 — Horizon devient 24 (au lieu de 1)
window_length = 24 * 7   # Fenêtre historique : 1 semaine complète
horizon = 24             # Prédire TOUTE une journée en une fois (multi-step)
stride = 4

# Dataset réduit : 40 clients (au lieu de 370 ou plus)
# → Garder les temps d'itération sous contrôle lors de la comparaison des stratégies
# → On peut scaler back-up plus tard pour voir où l'entraînement devient un goulot
n_customers = 40

Why this change transforms the task:

Instead of predicting just the next value, we ask the model to produce an entire day of hourly forecasts in one shot. This essentially turns the task into a sequence-to-sequence (Seq2Seq) problem:

• Input: a sequence of past load values (past week)
• Output: another sequence — the next 24 hours, aligned to the same client signal

Training windows are now: past week in → next day out.

Truncated Backpropagation Through Time (TBPTT)

The training configuration now includes a new parameter: truncated_bptt_steps.

Why TBPTT?

If we backpropagate through each step of a long sequence, the gradients can become more difficult to optimize and the training becomes more cumbersome.

Solution: we process the sequence by chunks, and between the chunks we detach the hidden state so that the calculation graph does not extend to the beginning.

def train_model_with_tbptt(model, train_loader, n_epochs, lr, 
                            tbptt_steps=None):
    """
    Entraînement avec Truncated Backpropagation Through Time (TBPTT).
    
    Au lieu de backpropager à travers toute la fenêtre de contexte longue,
    on traite la séquence en chunks et on détache le hidden state entre les chunks.
    
    Cela :
    - Garde la mémoire gérable
    - Rend l'entraînement bien plus stable sur les longues séquences
    - Permet quand même au modèle de transporter l'état en avant sur toute la semaine
      (on n'empêche pas les états de se propager, seulement les gradients)
    """
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.MSELoss()
    
    for epoch in range(n_epochs):
        model.train()
        
        for batch_x, batch_y in train_loader:
            optimizer.zero_grad()
            hidden = None
            
            T = batch_x.size(1)  # Longueur de la séquence
            
            if tbptt_steps is not None:
                # Traitement par chunks de tbptt_steps
                for t in range(0, T, tbptt_steps):
                    chunk = batch_x[:, t:t + tbptt_steps, :]
                    chunk_out, hidden = model.rnn(chunk, hidden)
                    
                    # *** detach est la clé ***
                    # Coupe le chemin du gradient → les gradients ne 
                    # se propagent pas au-delà de ce chunk
                    if isinstance(hidden, tuple):
                        # LSTM : (h_n, c_n)
                        hidden = tuple(h.detach() for h in hidden)
                    else:
                        # RNN/GRU : h_n
                        hidden = hidden.detach()
            else:
                # Backprop normale sur toute la séquence
                chunk_out, hidden = model.rnn(batch_x)
            
            # Prédiction et loss sur la partie horizon
            # ...

Crucial point: We still let the model carry its state forward over the whole week, but we do not force the gradients to propagate over the whole week. This keeps memory usage manageable and makes training much more stable over long sequences.

The same pattern reappears when encoding the entire context window before decoding the horizon. If the variable truncated_steps is defined, we go through the context in chunks, and after each chunk we detach the hidden state.

Encoding and Decoding

Two fundamental terms in Seq2Seq models:

Teacher Forcing

Teacher Forcing is a common training strategy in Seq2Seq models.

The problem without Teacher Forcing:

When running a model on its own predictions, early errors become part of the input distribution for subsequent steps. The model starts to see its own errors, and this can compensate for itself quickly, especially for longer horizons.

The Teacher Forcing solution:

def train_recurrent_seq2seq(model, train_loader, n_epochs, lr,
                             teacher_forcing: bool = False,
                             teacher_forcing_ratio: float = 1.0):
    """
    Entraînement Seq2Seq avec ou sans Teacher Forcing.
    
    Sans Teacher Forcing : on re-injecte la prédiction du modèle comme
    entrée pour le prochain pas du décodeur → les erreurs peuvent s'accumuler
    
    Avec Teacher Forcing : on injecte la vraie valeur cible (ground truth)
    comme entrée pour le prochain pas du décodeur → le modèle est toujours
    conditionné sur la valeur précédente correcte pendant l'entraînement
    """
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.MSELoss()
    
    for epoch in range(n_epochs):
        model.train()
        
        for batch_x, batch_y in train_loader:
            # batch_y : (batch, horizon) — vraies valeurs cibles
            optimizer.zero_grad()
            
            # Phase d'encodage
            _, hidden = model.encode(batch_x)
            
            # Phase de décodage : génération pas à pas
            predictions = []
            decoder_input = batch_x[:, -1:, :]  # Dernière valeur connue
            
            for step in range(horizon):
                pred, hidden = model.decode_step(hidden, decoder_input)
                predictions.append(pred)
                
                if teacher_forcing:
                    # Injecter la vraie valeur cible (ground truth)
                    decoder_input = batch_y[:, step:step+1].unsqueeze(-1)
                else:
                    # Injecter la propre prédiction du modèle
                    decoder_input = pred.unsqueeze(-1)
            
            predictions = torch.cat(predictions, dim=1)
            loss = criterion(predictions, batch_y)
            loss.backward()
            optimizer.step()

Why Teacher Forcing?

• Stabilizes training by keeping the decoder on data distribution during training
• Tends to help the model learn the local dynamics of how to take a step forward without being immediately sidetracked by its own imperfect output

The nuance of Teacher Forcing:

Teacher Forcing can help the model learn clean step transitions, and we often see this manifest early in the horizon because steps 1 and 6 are still relatively close to the true context. But if a model is trained almost exclusively on perfect previous inputs, it may be less prepared for the reality that at the time of inference it will be conditioned on its own imperfect outputs.

This is why it is important to inspect multiple points on the horizon rather than reporting a single aggregate number, which could almost certainly mask this behavior.

Results with/without Teacher Forcing:

# Tableau des résultats (validation) sur horizon de 24 steps
# MAE step 1 = erreur à la prochaine heure (très proche du contexte connu)
# MAE step 24 = erreur à 1 jour complet (plus exposé à l'accumulation d'erreurs)

results_table = {
    "LSTM + teacher_forcing":     {"MAE_overall": "meilleur", "RMSE_overall": "meilleur"},
    "LSTM - teacher_forcing":     {"MAE_overall": "+", "RMSE_overall": "+"},
    "GRU + teacher_forcing":      {"MAE_overall": "~", "RMSE_overall": "~"},
    "GRU - teacher_forcing":      {"MAE_overall": "~", "RMSE_overall": "~"},
}
# → La meilleure validation loss globale : LSTM avec teacher_forcing
# → Meilleurs modèles sauvés : GRU sans teacher_forcing, LSTM avec teacher_forcing

Step view of the horizon:

Architecture TinyTransformerForecaster

The Transformer is now introduced as an alternative to recurrent models for sequence modeling.

import math

class PositionalEncoding(nn.Module):
    """
    Encodage positionnel sinusoïdal.
    
    Dans un transformer, contrairement aux RNNs, toute la séquence est traitée
    en parallèle → il n'y a pas de notion intégrée d'ordre temporel.
    L'encodage positionnel injecte explicitement cette information.
    
    Pour chaque position pos et chaque dimension i :
    PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
    PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
    """
    def __init__(self, d_model: int, max_len: int = 5000):
        super().__init__()
        
        # Précalcul d'une table de vecteurs de position avec
        # des patterns sine et cosine à différentes fréquences
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len).unsqueeze(1).float()
        div_term = torch.exp(
            torch.arange(0, d_model, 2).float() 
            * (-math.log(10000.0) / d_model)
        )
        pe[:, 0::2] = torch.sin(position * div_term)   # dimensions paires
        pe[:, 1::2] = torch.cos(position * div_term)   # dimensions impaires
        
        self.register_buffer('pe', pe.unsqueeze(0))  # (1, max_len, d_model)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Ajouter le vecteur de position approprié à chaque token
        return x + self.pe[:, :x.size(1), :]

class TinyTransformerForecaster(nn.Module):
    """
    Transformer compact pour la prévision de séries temporelles.
    
    Architecture :
    1. input_projection  : 1 feature → d_model (espace de représentation)
    2. positional_encoding : injection de l'ordre temporel
    3. encoder_stack : N couches encoder avec self-attention + FFN
    4. head : d_model → horizon (prédiction directe de 24 steps)
    
    Avantage clé vs RNNs : l'attention permet à chaque pas de temps de 
    regarder directement chaque autre pas de temps dans la fenêtre d'entrée
    → pas besoin de boucle récurrente pour propager l'information
    """
    def __init__(self, 
                 input_size: int = 1,
                 d_model: int = 64,         # Dimension interne de représentation
                 nhead: int = 4,            # Nombre de têtes d'attention
                 num_encoder_layers: int = 2,
                 dim_feedforward: int = 128,
                 dropout: float = 0.1,
                 horizon: int = 24):
        super().__init__()
        
        # 1. Projection de l'entrée vers l'espace de représentation
        # Au lieu que chaque heure soit un seul scalaire, chaque heure 
        # devient un vecteur appris de dimension d_model
        self.input_projection = nn.Linear(input_size, d_model)
        
        # 2. Encodage positionnel (sinusoïdal, déterministe)
        self.pos_encoding = PositionalEncoding(d_model)
        
        # 3. Stack d'encodeurs Transformer
        # Chaque couche contient : multi-head attention + FFN 2-couches
        # + layer norms + dropout
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model,
            nhead=nhead,
            dim_feedforward=dim_feedforward,
            dropout=dropout,
            batch_first=True
        )
        self.encoder = nn.TransformerEncoder(
            encoder_layer, 
            num_layers=num_encoder_layers
        )
        
        # 4. Tête de prédiction : représentation → horizon
        # Prend la représentation au dernier pas de temps
        # et projette directement vers les 24 sorties
        self.head = nn.Linear(d_model, horizon)
    
    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # x : (batch, seq_len, input_size)
        x = self.input_projection(x)   # (batch, seq_len, d_model)
        x = self.pos_encoding(x)       # + information de position
        x = self.encoder(x)            # self-attention sur toute la fenêtre
        last_token = x[:, -1, :]       # Représentation du dernier pas de temps
        return self.head(last_token)   # (batch, horizon) — horizon direct !

Sinusoidal Positional Encoding

The mathematical equations of positional encoding:

$$PE_{(pos,, 2i)} = \sin!\left(\frac{pos}{10000^{\frac{2i}{d_{model}}}}\right)$$

$$PE_{(pos,, 2i+1)} = \cos!\left(\frac{pos}{10000^{\frac{2i}{d_{model}}}}\right)$$

Where:

• $pos$ is the position in the sequence
• $i$ is the dimension index
• $d_{model}$ is the dimension of the internal representation

Analogy with recurrent models: In a recurrent model, the order is integrated naturally because we process one step after the other and the hidden state carries the information forward. In a Transformer, you process the entire window in parallel, so the order must be injected explicitly — and positional encoding is simply how you do it.

Each hour gets:

• Same content features learned from projection layer
• A deterministic position signature that tells the model where it is located in the window

Self-Attention and Encode Transform

The payable advantage of the Transformer architecture is that, instead of carrying state forward one step at a time like an RNN, attention allows each time step to look directly at every other time step in the input window.

When the model tries to predict what comes next, it can learn patterns like:

• “this time tends to correlate with the same time yesterday”
• “this time tends to correlate with the same time last week”

…without needing a recurring loop to physically pass this information step by step.

flowchart TB
    subgraph TF["TinyTransformerForecaster"]
        direction TB
        Input["Entrée\n(batch, 168, 1)"]
        IP["input_projection\nLinear(1 → 64)"]
        PE["positional_encoding\nPE sinusoïdal"]
        
        subgraph ENC["Encoder Stack (×2)"]
            direction TB
            MHA["Multi-Head\nSelf-Attention\n(4 têtes)"]
            LN1["LayerNorm"]
            FFN["FFN 2-couches\n64 → 128 → 64"]
            LN2["LayerNorm"]
            DROP["Dropout"]
        end
        
        LAST["Dernier token\n(batch, 64)"]
        HEAD["head\nLinear(64 → 24)"]
        OUTPUT["Output\n(batch, 24)\n→ 24 prédictions directes"]
        
        Input --> IP --> PE --> ENC --> LAST --> HEAD --> OUTPUT
    end
    
    style OUTPUT fill:#dfd,stroke:#333
    style Input fill:#ffd,stroke:#333

The important final detail: The head takes the representation at the last time step and projects it into 24 outputs directly. This changes the prediction style: the Transformer can predict the entire 24 hour horizon in a single forward pass instead of rolling out one step into the next.

3.2 Multi-step forecasting with Transformers

Transformer instantiation and architecture

# Instanciation du TinyTransformerForecaster
transformer = TinyTransformerForecaster(
    input_size=1,
    d_model=64,
    nhead=4,
    num_encoder_layers=2,
    dim_feedforward=128,
    dropout=0.1,
    horizon=24
)

# Aperçu de l'architecture :
# input_projection    : 1 → 64
# positional_encoding : (déterministe, sinusoïdal)
# encoder_stack       : 2 couches encoder
#   chaque couche     : multi-head attention (4 têtes) 
#                       + FFN (64 → 128 → 64)
#                       + layer norms + dropout
# head                : 64 → 24 (un output par pas d'horizon)
#
# Nombre de paramètres : 68 632

print(f"Paramètres entraînables : {sum(p.numel() for p in transformer.parameters() if p.requires_grad):,}")
# Output : Paramètres entraînables : 68,632

Transformer variations tested:

transformer_configs = [
    {"d_model": 32, "num_encoder_layers": 1, "dim_feedforward": 64},
    {"d_model": 32, "num_encoder_layers": 2, "dim_feedforward": 64},
    {"d_model": 64, "num_encoder_layers": 1, "dim_feedforward": 128},  
    {"d_model": 64, "num_encoder_layers": 2, "dim_feedforward": 128},  # ← Meilleur
    {"d_model": 128, "num_encoder_layers": 2, "dim_feedforward": 256},
]
# Meilleur run : d_model=64, num_encoder_layers=2
# → Sélectionné sur validation loss

Note on training time: All of these runs train longer than the recurring models in this notebook. Partly because of the number of parameters, and partly because self-attention does more work per window because it considers relationships across the entire input window.

Rollout vs Direct Multi-step Output

Two inference strategies for multi-step forecasts:

1. One-step rollout (for recurring models):

def rollout_one_step_model(model, initial_window, horizon, device):
    """
    Déroule autoregressivement un modèle one-step pour la prévision multi-step.
    
    Commence depuis initial_window, ajoute chaque prédiction, 
    avance la fenêtre, répète jusqu'à générer l'horizon complet de 24h.
    
    → Chaque nouveau pas dépend de la sortie précédente du modèle
    → Les erreurs peuvent s'accumuler au fil du déroulent
    """
    model.eval()
    window = initial_window.clone()  # (batch, window_length, 1)
    predictions = []
    
    with torch.no_grad():
        for _ in range(horizon):
            # Prédire 1 pas en avant
            pred = model(window)            # (batch, 1)
            predictions.append(pred)
            
            # Avancer la fenêtre :
            # supprimer la valeur la plus ancienne, 
            # ajouter la prédiction à la fin
            window = torch.cat(
                [window[:, 1:, :], pred.unsqueeze(-1)], 
                dim=1
            )
    
    return torch.cat(predictions, dim=1)   # (batch, horizon)

2. Direct multi-step output (for the Transformer):

def direct_multistep_output(model, input_window):
    """
    Prédiction directe de l'horizon complet en un seul forward pass.
    
    → Pas d'accumulation d'erreurs step-by-step
    → Le modèle conditionne sur la fenêtre de contexte complète
      pour émettre l'horizon entier en une fois
    """
    model.eval()
    with torch.no_grad():
        return model(input_window)  # (batch, horizon) directement

Final comparative results

Evaluation on the retained test set (hold-out):

# Évaluation finale sur test set
# - Modèles récurrents : rollout one-step
# - Transformer : direct multi-step output

final_results = evaluate_all_models(
    models={
        "LSTM (teacher_forcing)": best_lstm_tf,
        "GRU (no teacher_forcing)": best_gru_no_tf,
        "Transformer (d=64, L=2)": best_transformer,
    },
    test_loader=test_loader,
    rollout_fn=rollout_one_step_model,
    direct_fn=direct_multistep_output,
)

Overall results table:

Interesting observation: LSTM has the lowest final commit loss in this run. But when we look at the prediction accuracy on the test set in the original units, the Transformer is the best by a wide margin on the overall MAE and RMSE.

Detailed table by horizon step:

Explanation of the discrepancy between validation loss and test accuracy:

This is an important reminder that for multi-step forecasting, the training objective chosen and the way one generates the predictions at inference time can create discrepancies between “looks good on validation loss” and “holds up over a long horizon in real units”.

Why does Transformer perform better over long horizons?

The gap between step 1 and step 24 is exactly what we expect when:

• A model must roll its predictions into the next one (LSTM, GRU)
• The other can condition on the full context window and emit the entire horizon at once (Transform)

The Transformer does not accumulate its own errors step by step during inference, and attention gives it a more direct way to connect distant parts of the input window when it constructs this horizon vector.

Analysis of drive dynamics

Plot Validation Loss:

• LSTM has the lowest validation loss — good reminder of the mismatch described above

Plot Gradient Norm:

• Transformer norm gradient remains relatively stable in a narrow band across epochs
• Recurring runs show more movement
• GRU without teacher_forcing rises by last epoch

It’s a short run, but we can already see that these models live in different optimization regimes.

MAE plot by horizon step:

• The long-range effect is clearly visible
• Once you go beyond the first step, the Transformer stays below it as you move further into the horizon

Predictions vs. Reality Plot (single customer):

• All three models roughly follow the general shape
• Transformer tends to stay closer on long swings
• Recurring models may drift further as rollout continues

Conclusion and summary of the evolution of the models

timeline
    title Évolution des modèles de séquences
    section RNN Vanilla
        Hidden state temporel : Feedback loop
        Variable-length sequences : Shared weights
        Limitation : Vanishing/Exploding gradient
    section LSTM & GRU
        LSTM : Cell state (long-term path)
        LSTM : 3 gates (forget, input, output)
        GRU : 2 gates (reset, update)
        GRU : Plus léger que LSTM
        Solution : Gradient flow stable
    section Transformer
        Self-Attention : Toute la séquence en parallèle
        Positional Encoding : Ordre injecté explicitement
        Direct multi-step : Horizon complet en 1 forward pass
        Long-range : Relations distantes directes

Summary: Recurrent models have taught us a lot about sequence modeling. Hidden state, unrolling, training stability, teacher forcing — all of these shape the way we think about sequential problems. Transformers keep that same central goal — modeling dependencies across time — but they do it with attention instead of recursion, which is why they can be such a strong option when you really care about longer horizons.

5. Appendices

Complete Math Equations

RNN Vanilla

$$h_t = \tanh(W_{hh} \cdot h_{t-1} + W_{xh} \cdot x_t + b_h)$$

$$\hat{y}t = W{hy} \cdot h_t + b_y$$

LSTM

$$f_t = \sigma(W_f \cdot [h_{t-1},, x_t] + b_f) \quad \text{(forget gate)}$$

$$i_t = \sigma(W_i \cdot [h_{t-1},, x_t] + b_i) \quad \text{(input gate)}$$

$$\tilde{C}t = \tanh(W_C \cdot [h{t-1},, x_t] + b_C) \quad \text{(candidate cell)}$$

$$C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \quad \text{(cell state update)}$$

$$o_t = \sigma(W_o \cdot [h_{t-1},, x_t] + b_o) \quad \text{(output gate)}$$

$$h_t = o_t \odot \tanh(C_t) \quad \text{(hidden state update)}$$

GRU

$$r_t = \sigma(W_r \cdot [h_{t-1},, x_t]) \quad \text{(reset gate)}$$

$$z_t = \sigma(W_z \cdot [h_{t-1},, x_t]) \quad \text{(update gate)}$$

$$\tilde{h}t = \tanh(W \cdot [r_t \odot h{t-1},, x_t]) \quad \text{(hidden candidate)}$$

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \quad \text{(hidden state update)}$$

Positional Encoding (Transformer)

$$PE_{(pos,, 2i)} = \sin!\left(\frac{pos}{10000^{\frac{2i}{d_{model}}}}\right)$$

$$PE_{(pos,, 2i+1)} = \cos!\left(\frac{pos}{10000^{\frac{2i}{d_{model}}}}\right)$$

Evaluation Metrics

$$MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|$$

$$RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}$$

Gradient problems (simplification with W alone)

Exploding gradient: $$\text{gradient} \propto W^T \xrightarrow{W=2} 2^T \to +\infty$$

Vanishing gradient: $$\text{gradient} \propto W^T \xrightarrow{W=0.5} 0.5^T \to 0$$

Architecture diagrams

General RNN architecture (rolled over T steps)

flowchart LR
    H0["h₀\n(init)"]
    X1["x₁"] --> C1["Cellule\nRNN"]
    H0 --> C1
    C1 --> Y1["ŷ₁"]
    C1 --> H1["h₁"]
    
    X2["x₂"] --> C2["Cellule\nRNN"]
    H1 --> C2
    C2 --> Y2["ŷ₂"]
    C2 --> H2["h₂"]
    
    X3["x₃"] --> C3["Cellule\nRNN"]
    H2 --> C3
    C3 --> Y3["ŷ₃"]
    C3 --> H3["..."]
    
    XT["x_T"] --> CT["Cellule\nRNN"]
    H3 --> CT
    CT --> YT["ŷ_T\n(prédiction)"]
    
    style H0 fill:#eee,stroke:#999
    style YT fill:#dfd,stroke:#333

Comparison of recurrent architectures

classDiagram
    class RNN {
        +h_t : hidden state
        +W_hh : poids hidden→hidden
        +W_xh : poids input→hidden
        +tanh(W_hh·h + W_xh·x)
        -Vanishing gradient risk
        -Exploding gradient risk
    }
    
    class LSTM {
        +h_t : hidden state (court terme)
        +C_t : cell state (long terme)
        +f_t : forget gate (σ)
        +i_t : input gate (σ)
        +o_t : output gate (σ)
        +C̃_t : cell candidate (tanh)
        +Gradient flow stable
    }
    
    class GRU {
        +h_t : hidden state (intègre LT+CT)
        +r_t : reset gate (σ)
        +z_t : update gate (σ)
        +h̃_t : hidden candidate (tanh)
        +Plus léger que LSTM
        +Gradient flow stable
    }
    
    RNN <|-- LSTM : extension
    RNN <|-- GRU : simplification

Full forecast pipeline

flowchart TD
    RAW["Données brutes\n(séries temporelles horaires\n370 clients)"]
    SPLIT["Split temporel\ntrain / val / test"]
    SCALE["Normalisation\n(per-client, train only)"]
    WINDOW["Fenêtrage glissant\nwindow_length=24 ou 168\nstride=4"]
    MODEL["Modèle\nRNN / LSTM / GRU / Transformer"]
    PRED["Prédictions\n(scaled)"]
    INVERSE["Inverse Transform\n(par client)"]
    EVAL["Évaluation\nMAE / RMSE"]
    
    RAW --> SPLIT --> SCALE --> WINDOW --> MODEL --> PRED --> INVERSE --> EVAL
    
    style RAW fill:#ffd,stroke:#888
    style MODEL fill:#ddf,stroke:#888
    style EVAL fill:#dfd,stroke:#888

Summary of hyperparameters used

Module 1 — RNN Baseline

Module 2 — LSTM/GRU/RNN Comparison

Fixed parameters: learning_rate=0.001, weight_decay=0.0 Epochs tuning: 2 | Final training epochs: 6

Module 3 — Seq2Seq & Transform

Final Model Comparison (Module 3)

Search Terms

recurrent · neural · networks · rnns · deep · machine · data · science · rnn · architecture · comparison · gradient · gru · lstm · forecasting · series · time · baseline · encoding · model · multi-step · seq2seq · analysis · attention