Deep Learning Model Evaluation, Tuning, and Optimization

Training: Deep Learning Model Evaluation, Tuning, and Optimization

Table of Contents

1. Module 1 — Explain the importance of evaluation to assess the generalizability and reliability of a model

• 1.1 Why assessment is important and what can go wrong
• 1.2 Key metrics for classification and regression
• 1.3 Confusion matrices and practical error analysis
• 1.4 Training curve validation and diagnostic strategies

2. Module 2 — Optimize model performance through hyperparameter tuning and regularization techniques

• 2.1 Understand the hyperparameters that shape model behavior
• 2.2 Regularization techniques to prevent overfitting
• 2.3 Hyperparameter tuning strategies
• 2.4 Learning rate planning and convergence acceleration

3. Module 3 — Improving training stability and efficiency through optimization best practices

• 3.1 Stabilize training with BatchNorm, LayerNorm and Gradient Clipping
• 3.2 Mixed Precision Training and GPU efficiency
• 3.3 Building Efficient Input Pipelines
• 3.4 Monitor models with TensorBoard or Weights & Biases

1. Introduction

Deep learning models can appear perfect during training and fail as soon as they encounter real data. Most of the time, the problem isn’t the architecture — it’s the evaluation, tuning, and optimization behind the model. In this training, we will learn how to evaluate models correctly, tune them scientifically, and optimize their training so that they remain reliable long after deployment.

2.

3. Explain the importance of evaluation to assess the generalizability and reliability of a model

1.1 Why assessment is important and what can go wrong

The myth of high training precision

Let’s start with one of the biggest misunderstandings in deep learning. When our model shows 97%, 98%, or even 99% training accuracy, the first reaction is: “Wow, that’s amazing!” » But the truth is that high training precision means almost nothing in itself. Deep learning models are incredibly good at memorizing. They can adapt well to the training dataset, even when they don’t understand the underlying patterns.

So in the graph, the training accuracy can touch almost 100%, but the validation accuracy drops dramatically. And that gap tells the whole story — it’s the part that tells us whether the model is truly learning or just memorizing.

The key message: Training accuracy can give you confidence, but it doesn’t tell you whether the model will work in the real world.

Why evaluation pipelines break easily

Here are the most common reasons why the assessment fails:

1. Data leakage — The most dangerous of all. Leakage occurs when the model sees information during training that should only exist in the test set. Sometimes it’s a timestamp, sometimes an ID, sometimes a hidden correlation. And the scary part is that everything looks perfect — the metrics are high, the accuracy is perfect, but the model is cheating.

2. Biased splits — If the validation data is not representative — maybe a class is missing, maybe the data was split incorrectly — your assessment becomes meaningless. The model looks good only because the validation set was easy.

3. Shortcut learning — This is where the model learns the wrong pattern, such as detecting the background instead of the object. Even when everything seems correct, the assessment may be fundamentally broken underneath.

4. Silent failures in production — When the evaluation is low, the model works perfectly during development and then suddenly collapses once in contact with real users or real data. There is no error message, no warning. The model simply starts giving bad predictions, and these errors turn into business impact.

A reliable evaluation workflow

To prevent these issues, a reliable assessment workflow requires a few essential elements:

1. Clean splits — train, validation and test — especially when the dataset is unbalanced.
2. The Right Metrics — Precision is great for simple tasks, but in real-world scenarios, precision, recall, F1, or AUC often tell the real truth.
3. Confusion matrices and error analysis — This is where you’ll find the hidden blind spots that metrics alone never reveal.
4. Diagnosis of training and validation curves to detect overfitting, underfitting or data leak early.
5. Check for drifts or unexpected data patterns, because the data always changes once the model is deployed.

1.2 Key Metrics for Classification and Regression

Classification Metrics

Let’s break down the metrics every deep learning practitioner needs to master.

Accuracy

• Measures how often the model is correct overall.
• Works well only when the dataset is balanced.
• As soon as one class dominates the dataset, precision becomes almost useless.
• Example: A dataset where 98% of the samples belong to class A and only 2% to class B. If the model predicts class A for each sample, its accuracy is still 98% — but has it learned anything significant? No way.

Precision

• Of all the predictions marked as positive by the model, how many were actually correct?
• It’s like testing the model’s prudence.

Recall

• Of all the actual positive cases present in the data, how many did the model detect?
• While precision is about correctness, recall is about completeness.

F1 Score

• The harmonic mean of precision and recall.
• Gives a balanced view when you need both — accuracy and completeness.

Rule: While precision is a good starting number, precision, recall and F1 really reveal how your model performs under real-world constraints.

ROC-AUC (Receiver Operating Characteristic - Area Under the Curve)

This metric is almost magical once you understand it. Instead of checking whether predictions are simply correct or incorrect, ROC-AUC looks at how much higher the model ranks positives than negatives. It tells you: if you randomly choose a positive and a negative sample, what is the probability that the model will assign a higher score to the positive?

This makes ROC-AUC extremely powerful in real-world tasks, especially when:

• Thresholds change
• Classes are unbalanced
• We care about ranking quality rather than raw accuracy

Regression Metrics

For regression, the model predicts a continuous value rather than a class.

How to choose the right metric:

• The answer always depends on the business issues.
• If you predict house prices, big errors can cost millions → MSE or RMSE.
• If you are predicting delivery times or traffic estimates → MAE is often better because it is not disrupted by rare outliers.
• In summary, the chosen metric should reflect the true cost of a model error.

Code example — Metrics on a fraud detection dataset

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
                              f1_score, confusion_matrix, roc_auc_score)

# Chargement du dataset de détection de fraude
df = pd.read_csv("/content/fraud_detection_demo_dataset.csv")
df.head()

# Distribution de la classe cible
df['is_fraud'].value_counts()

# Entraînement du modèle
X = df.drop("is_fraud", axis=1)
y = df["is_fraud"]

model = LogisticRegression(max_iter=500)
model.fit(X, y)

# Évaluation des métriques
y_pred = model.predict(X)

print("Accuracy:", accuracy_score(y, y_pred))
print("Precision:", precision_score(y, y_pred))
print("Recall:", recall_score(y, y_pred))
print("F1-score:", f1_score(y, y_pred))

# ROC-AUC
y_prob = model.predict_proba(X)[:, 1]
auc = roc_auc_score(y, y_prob)
print("ROC-AUC:", auc)

Observed results: Overall accuracy appears strong, but recall is significantly lower. This means that the model misses a lot of positive cases — something that accuracy fails to reveal. If we look at the confusion matrix, we can see why: there are several false negatives. These are cases where the model predicted 0, but the true label was 1. These false negatives directly reduce recall.

1.3 Confusion matrices and practical error analysis

Anatomy of a confusion matrix

The confusion matrix is ​​one of the simplest but most revealing tools in evaluation. It’s just a 2x2 grid, but each cell has a meaning:

• TP (True Positives) — Predicted positive and it is correct.
• TN (True Negatives) — Predicted negative and it is correct.
• FP (False Positives) — The model says positive, but the truth is negative.
• FN (False Negatives) — The model says negative, but the truth is positive.

Green cells (TP and TN) show where the model is correct. The red cells (FP and FN) show the errors — and these errors tell much more about the behavior of the model than accuracy ever can.

What error patterns mean in practice

Many false positives (FP):

• The model is too sensitive — it triggers alarms even when everything is fine.
• In fraud detection: blocking of legitimate transactions.
• In spam detection: marking legitimate emails as spam.

Many false negatives (FN):

• Model misses actual positive cases — much more dangerous in some scenarios.
• In medical diagnosis: not detecting an illness can have serious consequences.
• Low recall often manifests itself as a high number of FNs, and this is a red flag that should never be ignored.

Class imbalance pattern:

• The model simply doesn’t see enough examples of a certain class to learn it well.

Confusion between specific categories:

• Model is confused between some categories due to similar features, lighting or noise.

Insight: The confusion matrix isn’t just a table of numbers — it’s a window into the model’s blind spots.

Improvement loop based on error analysis

Model improvement starts with collecting misclassified samples, including FP and FN. These examples reveal what the model failed to learn:

• Missing Classes
• Weak Features
• Label noise (noise in labels)
• Insufficient increase

Process:

1. Collect errors
2. Analyze patterns
3. Apply the fix
4. Retrain

Code example — Error analysis with confusion matrix

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

# Chargement du dataset
df = pd.read_csv("/content/fraud_detection_demo_dataset.csv")
X = df.drop("is_fraud", axis=1)
y = df["is_fraud"]

# Entraînement
model = LogisticRegression(max_iter=500)
model.fit(X, y)
y_pred = model.predict(X)

# Matrice de confusion
cm = confusion_matrix(y, y_pred)

# Visualisation
plt.figure(figsize=(3, 3))
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
            xticklabels=["Pred Legit", "Pred Fraud"],
            yticklabels=["True Legit", "True Fraud"])
plt.show()

# Extraction des erreurs
false_positives = df[(y == 0) & (y_pred == 1)]
false_negatives = df[(y == 1) & (y_pred == 0)]

print("Faux positifs (transactions légitimes marquées frauduleuses):")
false_positives.head()

print("Faux négatifs (fraudes manquées):")
false_negatives.describe().T

Observed results:

• False positives: Represent over-triggering — legitimate transactions that the model thought were suspicious. Often include high transaction amounts or international activities.
• False Negatives: Represents misses — fraudulent transactions that have escaped the filter. Most resemble normal customer behavior (moderate amounts, high trust scores, transactions at normal times of day). In other words, they look like legitimate customer behavior, so the model isn’t confident enough to report them.

This is the central idea of ​​error analysis: instead of looking only at accuracy or F1 score, we look at errors directly — this is how we discover why the model behaves the way it does and where it needs improvement the most.

1.4 Validation strategies and training curve diagnostics

Importance of a Strong Validation Configuration

Before we tune or optimize anything, we need to make sure the validation setup itself is solid, because how you split your data affects everything you believe about your model.

Division methods (splits)

Practical rules:

• For large datasets: a basic split holdout is generally sufficient.
• For unbalanced datasets: always use a stratified split. It prevents unrealistic validation results.
• For small datasets: k-fold cross-validation gives a much more reliable image by using several train/validation splits.
• Whenever you observe strange validation behavior — extremely low loss, sudden spikes in accuracy — first suspect a data leak.

Rule: The choice of validation method really depends on the size of the dataset, the balance of classes, and the reliability required for the evaluations.

Read and diagnose drive curves

Training and validation curves are the most direct tools for diagnosing what is happening during training.

Code Example — Stratified Split and Training Curves

import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
import matplotlib.pyplot as plt

# Chargement du dataset de churn
df = pd.read_csv("/content/churn_demo_dataset.csv")
df.head()

# Distribution de la cible
df['is_churn'].value_counts(normalize=True)
# Résultat : environ 30% de churn - balance permettant un split stable

# Préparation des données
X = df.drop("is_churn", axis=1)
y = df["is_churn"].values

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Split stratifié en trois parties : train / validation / test
X_train, X_temp, y_train, y_temp = train_test_split(
    X_scaled, y, test_size=0.3, stratify=y, random_state=42
)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, stratify=y_temp, random_state=42
)

# Construction d'un petit réseau de neurones
model = Sequential([
    Dense(16, activation='relu', input_shape=(X_train.shape[1],)),
    Dense(8, activation='relu'),
    Dense(1, activation='sigmoid')
])

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Entraînement sur 25 epochs
history = model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=25,
    batch_size=32,
    verbose=0
)

# Visualisation des courbes
plt.plot(history.history['loss'], label='Train Loss')
plt.plot(history.history['val_loss'], label='Val Loss')
plt.legend()
plt.show()

Interpretation of the curves: Initially, the two curves descend together, which means that the model learns useful and generalizable patterns. If the training loss continues to fall but the validation loss begins to rise, this is a classic sign of overfitting. And if the two curves remain high and close to each other, this is underfitting.

4.

5. Optimizing model performance through hyperparameter tuning and regularization techniques

2.1 Understanding the hyperparameters that shape model behavior

Learning rate: the most important hyperparameter

When we talk about hyperparameters in deep learning, one of them stands out by far: the learning rate. The learning rate controls the size of each step during training. Each time the model updates its weights, it essentially takes a step across the loss landscape. And the learning rate decides the length of this step.

This is why learning rate is said to be the most important hyperparameter to tune — because it directly controls how your model learns.

Batch size and optimizers

Batch size:

• Small batch size → Noisy gradients, different updates, help with generalization, models often better avoid overfitting.
• Large batch size → Stable and smooth gradients, faster training on modern hardware, but large batches sometimes converge to sharper minima, which can harm generalization.

Optimizers:

Each optimizer shapes how the model moves across the loss surface — whether the steps are smooth and constant, fast and adaptive, or very sensitive to noise.

The loss landscape and the LR Finder

Training a model is like navigating a massive terrain full of peaks, valleys and flat regions. Your hyperparameters, particularly the learning rate, decide the path your model takes through this landscape.

LR Finder: To find this good learning rate without guessing, we use the LR Finder. We start with a tiny learning rate, gradually increase it, and observe how the loss responds. The resulting curve shows exactly where the loss starts to improve quickly and where it explodes. This sweet spot is your ideal learning rate, revealed in a powerful visualization.

Code example — Comparing learning rates and batch sizes

Basic configuration and data:

import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Dataset : propension à l'achat (e-commerce)
df = pd.read_csv("/content/purchase_propensity_demo_dataset.csv")

X = df.drop("made_purchase", axis=1)
y = df["made_purchase"].values

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_val, y_train, y_val = train_test_split(
    X_scaled, y, test_size=0.2, stratify=y, random_state=42
)

# Construction du modèle de base
def build_model():
    model = Sequential([
        Dense(16, activation='relu', input_shape=(X_train.shape[1],)),
        Dense(8, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    return model

Baseline training (lr=0.001):

baseline_model = build_model()
baseline_model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy']
)

history_baseline = baseline_model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=20,
    batch_size=32,
    verbose=0
)

# Courbes saines : training loss descend en douceur, val loss suit de près
plt.plot(history_baseline.history['loss'], label='Train Loss (baseline)')
plt.plot(history_baseline.history['val_loss'], label='Val Loss (baseline)')
plt.xlabel('Epoch'); plt.ylabel('Loss'); plt.legend(); plt.show()

Learning rate too high (lr=0.1):

high_lr_model = build_model()
high_lr_model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.1),
    loss='binary_crossentropy',
    metrics=['accuracy']
)

history_high_lr = high_lr_model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=20,
    batch_size=32,
    verbose=0
)

# Comparaison : lr=0.001 descend en douceur, lr=0.1 rebondit constamment
plt.plot(history_baseline.history['loss'], label='Train Loss (lr=0.001)')
plt.plot(history_high_lr.history['loss'], label='Train Loss (lr=0.1)')
plt.xlabel('Epoch'); plt.ylabel('Loss'); plt.legend(); plt.show()

Comparison of batch sizes:

# Petit batch (batch=8)
small_batch_model = build_model()
small_batch_model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy']
)
history_small_batch = small_batch_model.fit(
    X_train, y_train, validation_data=(X_val, y_val),
    epochs=20, batch_size=8, verbose=0
)

# Grand batch (batch=256)
large_batch_model = build_model()
large_batch_model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy']
)
history_large_batch = large_batch_model.fit(
    X_train, y_train, validation_data=(X_val, y_val),
    epochs=20, batch_size=256, verbose=0
)

plt.plot(history_small_batch.history['val_loss'], label='Val Loss (batch=8)')
plt.plot(history_large_batch.history['val_loss'], label='Val Loss (batch=256)')
plt.xlabel('Epoch'); plt.ylabel('Validation Loss'); plt.legend(); plt.show()

Observed results:

• Batch size 8 (blue curve): The val loss goes down quickly but shakes a little along the way. This noise comes from using only a few examples per update. Noisy but flexible gradients sometimes give better final results.
• Batch size 256 (orange curve): The curve is beautifully smooth, and learns much more slowly. The model cannot explore the loss landscape as freely, so it converges cautiously.

2.2 Regularization techniques to prevent overfitting

What is overfitting?

When training deep learning models, one of the most common problems is overfitting. The model performs beautifully on training data, but breaks down as soon as we test it on new examples. Overfitting occurs because the model begins to memorize patterns in the training data, including noise and idiosyncrasies that do not generalize.

Regularization is our toolkit to prevent this. It keeps the model honest, general, and prevents it from becoming overconfident about things it shouldn’t be confident about.

Visual signature of overlearning: The training loss falls steadily while the validation loss begins to rise. Regularization smooths this out — it brings these curves together, helping the model perform better on data it has never seen.

The three most common regularization techniques

1. Dropout

The dropout is probably the most visual. Inside the neural network, the dropout randomly deactivates part of the neurons during training. Like temporarily removing pieces from the network, this forces the model to not rely on a single neuron or shortcut. It must learn robust representations instead of memorizing specific patterns.

2. L2 Regularization (Weight Decay)

L2 discourages the model from assigning very large weights to a feature. And that’s a good thing, because large weights often mean the model is overlearning on certain inputs. By gently bringing these weights towards 0, L2 helps models stay simpler, smoother, and more general.

3. Early Stopping

This is one of the simplest yet most powerful techniques. Instead of training for a fixed number of epochs, you watch the validation loss, and if it stops improving, you stop training. This way you capture the best version of your model before it starts memorizing noise.

Code example — Comparison of regularization techniques

Setup of the dataset with intentionally little data to cause overfitting:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.regularizers import l2
from tensorflow.keras.callbacks import EarlyStopping
import matplotlib.pyplot as plt

# Chargement du dataset
df = pd.read_csv("/content/purchase_propensity_demo_dataset.csv")
X = df.drop("made_purchase", axis=1)
y = df["made_purchase"].values

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_val, y_train, y_val = train_test_split(
    X_scaled, y, test_size=0.2, stratify=y, random_state=42
)

# Petit ensemble d'entraînement pour encourager intentionnellement l'overfitting
X_train_small = X_train[:400]
y_train_small = y_train[:400]

print("Train small:", X_train_small.shape, "Val:", X_val.shape)

Without regularization (baseline):

tf.random.set_seed(42)
np.random.seed(42)

def build_base_model():
    model = Sequential([
        Dense(64, activation='relu', input_shape=(X_train.shape[1],)),
        Dense(32, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    return model

model_no_reg = build_base_model()
model_no_reg.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history_no_reg = model_no_reg.fit(
    X_train_small, y_train_small,
    validation_data=(X_val, y_val),
    epochs=40, batch_size=32, verbose=0
)

plt.plot(history_no_reg.history['loss'], label='Train Loss (no reg)')
plt.plot(history_no_reg.history['val_loss'], label='Val Loss (no reg)')
plt.xlabel('Epoch'); plt.ylabel('Loss'); plt.legend(); plt.show()

With Dropout:

tf.random.set_seed(42)
np.random.seed(42)

def build_dropout_model():
    model = Sequential([
        Dense(64, activation='relu', input_shape=(X_train.shape[1],)),
        Dropout(0.4),
        Dense(32, activation='relu'),
        Dropout(0.3),
        Dense(1, activation='sigmoid')
    ])
    return model

model_dropout = build_dropout_model()
model_dropout.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history_dropout = model_dropout.fit(
    X_train_small, y_train_small,
    validation_data=(X_val, y_val),
    epochs=40, batch_size=32, verbose=0
)

With L2 Regularization:

tf.random.set_seed(42)
np.random.seed(42)

def build_l2_model():
    model = Sequential([
        Dense(64, activation='relu', kernel_regularizer=l2(1e-3),
              input_shape=(X_train.shape[1],)),
        Dense(32, activation='relu', kernel_regularizer=l2(1e-3)),
        Dense(1, activation='sigmoid')
    ])
    return model

model_l2 = build_l2_model()
model_l2.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

history_l2 = model_l2.fit(
    X_train_small, y_train_small,
    validation_data=(X_val, y_val),
    epochs=40, batch_size=32, verbose=0
)

With Early Stopping:

tf.random.set_seed(42)
np.random.seed(42)

model_early_stop = build_base_model()
model_early_stop.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

early_stop = EarlyStopping(
    monitor='val_loss',
    patience=3,
    restore_best_weights=True  # restaure les meilleurs poids
)

history_early = model_early_stop.fit(
    X_train_small, y_train_small,
    validation_data=(X_val, y_val),
    epochs=40,
    batch_size=32,
    callbacks=[early_stop],
    verbose=0
)

print("Stopped at epoch:", len(history_early.history['loss']))

Comparison of all techniques:

plt.figure(figsize=(10, 5))
plt.plot(history_no_reg.history['val_loss'], label='Val Loss (no reg)')
plt.plot(history_dropout.history['val_loss'], label='Val Loss (dropout)')
plt.plot(history_l2.history['val_loss'], label='Val Loss (L2)')
plt.plot(history_early.history['val_loss'], label='Val Loss (early stop)')
plt.xlabel('Epoch')
plt.ylabel('Validation Loss')
plt.title('Purchase Propensity — Validation Loss Comparison')
plt.legend()
plt.show()

What we observe:

• Without regularization: The training loss goes down beautifully, but the validation loss quickly starts to rise. The model clearly remembers the training data.
• With Dropout: The model becomes more stable. The validation loss more closely follows the training loss. Although the training may seem a bit noisy, the final model is much better at generalizing.
• With L2: The training curve becomes smoother, and the validation curve stabilizes. The model stops relying on specific neurons or peak weights. Everything becomes more controlled.
• With Early Stopping: Training stops automatically when the validation loss stops improving.

2.3 Hyperparameter tuning strategies

The three classic tuning strategies

When it comes time to tune hyperparameters, there are three classic strategies:

1. Manual Tuning – This is where most beginners start.

• We try a learning rate, a dropout rate, a layer size, and we continue to adjust based on intuition.
• This sometimes works, but it’s inconsistent and very dependent on your intuition.
• No guarantee of landing near optimal configuration.

2. Grid Search

• We define a fixed grid of hyperparameters, we try each combination, and we wait for the results.
• Seems routine, but gets expensive quickly.
• The more hyperparameters we add, the more the grid explodes in size.

3. Random Search

• Instead of trying every possible combination, we sample a few at random.
• The magic: random search often finds better models with fewer tries because it explores space more freely.
• It doesn’t waste time checking unimportant dimensions.
• Trip over good regions much more efficiently.

Key search insight: Bergstra & Bengio (2012) showed that random search outperforms grid search in many practical cases, especially for high-dimensional spaces.

Practical search space design

Choosing the right beaches is half the battle:

Why the log scale for learning rate? This captures the huge range where LR actually matters. A jump from 0.001 to 0.01 is very different from a jump from 0.1 to 0.2 — the log scale captures this difference.

Code Example — Grid Search vs Random Search

Setup of the dataset and helper functions:

import pandas as pd
import numpy as np
import itertools
import random
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense, Dropout

# Chargement du dataset
df = pd.read_csv("/content/purchase_propensity_demo_dataset.csv")
X = df.drop("made_purchase", axis=1)
y = df["made_purchase"].values

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

X_train, X_val, y_train, y_val = train_test_split(
    X_scaled, y, test_size=0.2, stratify=y, random_state=42
)

# Subset plus petit pour que le tuning soit plus rapide
X_train_small = X_train[:800]
y_train_small = y_train[:800]

# Fonction pour construire le modèle
def build_model(learning_rate=0.001, dropout_rate=0.0):
    model = Sequential([
        Dense(16, activation='relu', input_shape=(X_train.shape[1],)),
        Dropout(dropout_rate),
        Dense(8, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    opt = tf.keras.optimizers.Adam(learning_rate=learning_rate)
    model.compile(optimizer=opt, loss='binary_crossentropy', metrics=['accuracy'])
    return model

# Fonction pour entraîner et évaluer une configuration
def train_and_eval(params, epochs=8):
    lr = params["lr"]
    dr = params["dropout"]
    bs = params["batch"]

    model = build_model(learning_rate=lr, dropout_rate=dr)
    history = model.fit(
        X_train_small, y_train_small,
        validation_data=(X_val, y_val),
        epochs=epochs,
        batch_size=bs,
        verbose=0
    )
    best_val_loss = min(history.history["val_loss"])
    best_val_acc = max(history.history["val_accuracy"])
    return best_val_loss, best_val_acc

Search space definition:

lr_values = [1e-4, 3e-4, 1e-3, 3e-3]
dropout_values = [0.0, 0.3, 0.5]
batch_values = [32, 64]

total_grid_combos = len(lr_values) * len(dropout_values) * len(batch_values)
print(f"Total combinaisons Grid Search: {total_grid_combos}")  # = 24

Grid Search — all 24 combinations:

results_grid = []

for lr in lr_values:
    for dr in dropout_values:
        for bs in batch_values:
            params = {"lr": lr, "dropout": dr, "batch": bs}
            val_loss, val_acc = train_and_eval(params, epochs=8)
            results_grid.append({
                "strategy": "grid",
                "lr": lr, "dropout": dr, "batch": bs,
                "val_loss": val_loss, "val_acc": val_acc
            })

df_grid = pd.DataFrame(results_grid)
df_grid_sorted = df_grid.sort_values("val_loss").reset_index(drop=True)
df_grid_sorted.head(5)

Random Search — only 8 random tries:

def sample_random_configs(n_samples=8):
    all_combos = list(itertools.product(lr_values, dropout_values, batch_values))
    sampled = random.sample(all_combos, n_samples)
    configs = [{"lr": lr, "dropout": dr, "batch": bs} for lr, dr, bs in sampled]
    return configs

random_configs = sample_random_configs(n_samples=8)

results_random = []
for params in random_configs:
    val_loss, val_acc = train_and_eval(params, epochs=8)
    results_random.append({
        "strategy": "random",
        "lr": params["lr"], "dropout": params["dropout"], "batch": params["batch"],
        "val_loss": val_loss, "val_acc": val_acc
    })

df_random = pd.DataFrame(results_random)
df_random_sorted = df_random.sort_values("val_loss").reset_index(drop=True)

Comparing results:

df_grid_top = df_grid_sorted.head(5)[["lr", "dropout", "batch", "val_loss", "val_acc"]]
df_random_top = df_random_sorted.head(5)[["lr", "dropout", "batch", "val_loss", "val_acc"]]

print("Top 5 configs from GRID search:")
print(df_grid_top)

print("\nTop configs from RANDOM search:")
print(df_random_top)

Typical results observed:

• Grid Search: Explores the 24 combinations and finds a val loss around 0.38, val accuracy ~85%. The best results are grouped around a learning rate of around 0.003 with a batch size of 32 or 64.
• Random Search: With only 8 tries, its best val loss is around 0.44 — a little worse than the best in the grid, but still reasonably good. And that’s the key idea: Grid search gives very good results, but trying every combination. Random search uses far fewer experiments and can still land in a good region of space even if it doesn’t always hit the absolute best configuration.

Advanced practice: For large models and large search spaces, we often combine these ideas. We use random search or a smarter sampler to explore broadly, then zoom in with a more targeted search once we know where the right hyperparameters live.

2.4 Learning rate planning and acceleration of convergence

Why are learning rate schedules important?

When it comes to optimizing training speed and improving final performance, learning rate schedules are one of the most powerful tools available. A fixed learning rate works, but almost never gives the best results.

The idea is simple:

• At the start of training: We want the model to take large, aggressive steps. He explores the landscape of loss, tries to find direction. A high learning rate helps the model move quickly and get out of bad regions early.
• As training progresses: These big steps become more of a liability. The model has to slow down, taking smaller, more careful steps to refine the details — almost like zooming into the valley and looking for the absolute lowest point. This is where the lower learning rate becomes essential.

Learning rate schedules automate this entire process — starting high to move quickly and gradually reducing the learning rate so the model can fine-tune itself. This combination often leads to faster convergence, smoother curves and better final performance.

The three most used schedules

1. Step Decay

• The simplest: we start with a fixed learning rate, and after a number of epochs, we lower it — generally by a factor of 10.
• Gives the optimizer a precision hit at the right time.
• Like changing gears in a car — high power early, finer control later.

2. Cosine Annealing

• More elegant: instead of sharp drops, it follows a smooth cosine curve.
• The learning rate slowly decreases over time.
• Gives a smoother convergence path and often helps the optimizer slide through the loss surface.

3. Warm Restarts (Cosine with Restarts)

• Almost like cosine annealing with periodic resets.
• The learning rate cycles down, then up, then down again, repeating in waves.
• These jumps help the optimizer escape local minima and explore new regions.
• Surprisingly effective for models that tend to crash early.

When to choose which schedule?

Code example — Comparing learning rate schedules

Definition of schedules:

import tensorflow as tf
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense

def build_model():
    model = Sequential([
        Dense(16, activation='relu', input_shape=(X_train.shape[1],)),
        Dense(8, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    return model

# Schedule 1 : Constant (baseline)
baseline_model = build_model()
baseline_model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss='binary_crossentropy',
    metrics=['accuracy']
)
history_const = baseline_model.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=10, batch_size=32, verbose=0
)

# Schedule 2 : Step Decay
def step_decay(epoch):
    if epoch < 5:
        return 0.001
    else:
        return 0.0003  # baisse par un facteur ~3 après l'epoch 5

step_callback = tf.keras.callbacks.LearningRateScheduler(step_decay)

model_step = build_model()
model_step.compile(optimizer=tf.keras.optimizers.Adam(), loss='binary_crossentropy')
history_step = model_step.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=10, batch_size=32,
    callbacks=[step_callback], verbose=0
)

# Schedule 3 : Cosine Annealing
def cosine_annealing(epoch, total_epochs=10, lr_max=0.001, lr_min=0.0001):
    import math
    cos_inner = (math.pi * (epoch % total_epochs)) / total_epochs
    return lr_min + (lr_max - lr_min) * (1 + math.cos(cos_inner)) / 2

cosine_callback = tf.keras.callbacks.LearningRateScheduler(cosine_annealing)
model_cosine = build_model()
model_cosine.compile(optimizer=tf.keras.optimizers.Adam(), loss='binary_crossentropy')
history_cosine = model_cosine.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=10, batch_size=32,
    callbacks=[cosine_callback], verbose=0
)

# Schedule 4 : Cosine avec Warm Restarts
def cosine_with_restarts(epoch, cycle_length=5, lr_max=0.001, lr_min=0.0001):
    import math
    cycle_epoch = epoch % cycle_length
    cos_inner = (math.pi * cycle_epoch) / cycle_length
    return lr_min + (lr_max - lr_min) * (1 + math.cos(cos_inner)) / 2

restart_callback = tf.keras.callbacks.LearningRateScheduler(cosine_with_restarts)
model_restart = build_model()
model_restart.compile(optimizer=tf.keras.optimizers.Adam(), loss='binary_crossentropy')
history_restart = model_restart.fit(
    X_train, y_train,
    validation_data=(X_val, y_val),
    epochs=10, batch_size=32,
    callbacks=[restart_callback], verbose=0
)

Comparative visualization:

plt.figure(figsize=(7, 5))
plt.plot(history_const.history['val_loss'], label='Constant LR')
plt.plot(history_step.history['val_loss'], label='Step Decay')
plt.plot(history_cosine.history['val_loss'], label='Cosine Annealing')
plt.plot(history_restart.history['val_loss'], label='Cosine + Restarts')
plt.xlabel("Epoch")
plt.ylabel("Validation Loss")
plt.legend()
plt.show()

Interpretation of curves:

• LR Constant (blue): Smooth, consistent, reliable — our benchmark.
• Step Decay (orange): Goes down quickly and keeps a good lead at the start. That’s the benefit of those initial aggressive steps. This schedule works like shifting gears — high performance early, then precise descent later.
• Cosine Annealing (green): Descends more gracefully. Not the fastest, but incredibly stable. The learning rate follows a smooth cosine wave, taking increasingly smaller and more deliberate steps.
• Cosine + Warm Restarts (red): Shows those slight bumps at the start (the “warm restarts”), but look at the end — it catches up beautifully. These bumps aren’t random — they’re small, deliberate pushes that help the model escape when it’s stuck.

6.

7. Improving Drive Stability and Efficiency with Optimization Best Practices

3.1 Stabilize training with BatchNorm, LayerNorm and Gradient Clipping

The problem of drive instability

As deep learning models become deeper and more complex, one thing becomes very clear: training can become unstable very quickly. Sometimes the loss goes down smoothly, and other times it goes up aggressively, swings wildly, or explodes completely.

Most of this instability comes from what is happening inside the network:

• Activations in deeper layers may drift, amplify or explode.
• Since each layer depends on the previous one, even small fluctuations are amplified across the network.
• gradients can become extremely noisy, or in the worst case, grow so large that the model diverges.

This is why training a deep network without stabilization techniques often feels unpredictable.

The three stabilization methods

1. Batch Normalization (BatchNorm)

• What it does: Normalizes intermediate activations across the batch.
• No matter how activations drift or scale, the network returns them to a stable range.
• Ideal for: CNNs and most standard deep learning architectures.

2. Layer Normalization (LayerNorm)

• What it does: Instead of normalizing across the batch, LayerNorm normalizes across the feature dimension.
• This makes it independent of batch size.
• Ideal for: NLP and transform models that work with variable length sequences or tokens.

3. Gradient Clipping

• What it does: Places a cap on the size that gradients can reach, preventing the drive from spiraling out of control.
• Simple but can save training instantly when things start to blow up.
• Ideal for: Deep models with high learning rates.

Together, these three techniques — BatchNorm, LayerNorm, and Gradient Clipping — form the basis for training stability across modern deep learning architectures.

Code example — BatchNorm in action

Setup of the MNIST dataset:

import numpy as np
import tensorflow as tf
from tensorflow.keras import layers, models
import matplotlib.pyplot as plt

# Chargement de MNIST
(x_train_full, y_train_full), (x_test, y_test) = tf.keras.datasets.mnist.load_data()

# Normalisation à [0, 1]
x_train_full = x_train_full.astype("float32") / 255.0
x_test = x_test.astype("float32") / 255.0

# Sous-ensemble pour la vitesse
x_train = x_train_full[:10000]
y_train = y_train_full[:10000]

# Ajout de la dimension canal pour CNN
x_train_cnn = np.expand_dims(x_train, -1)
x_test_cnn = np.expand_dims(x_test, -1)

CNN baseline (without BatchNorm):

def build_baseline_cnn():
    model = models.Sequential([
        layers.Conv2D(16, (3, 3), activation="relu", padding="same",
                      input_shape=(28, 28, 1)),
        layers.Conv2D(32, (3, 3), activation="relu", padding="same"),
        layers.Flatten(),
        layers.Dense(64, activation="relu"),
        layers.Dense(10, activation="softmax")
    ])
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"]
    )
    return model

baseline_cnn = build_baseline_cnn()
baseline_cnn.summary()

history_baseline = baseline_cnn.fit(
    x_train_cnn, y_train,
    validation_data=(x_test_cnn, y_test),
    epochs=5, batch_size=64, verbose=0
)

CNN with BatchNorm:

def build_bn_cnn():
    model = models.Sequential([
        layers.Conv2D(16, (3, 3), padding="same", input_shape=(28, 28, 1)),
        layers.BatchNormalization(),   # BN après la convolution, avant ReLU
        layers.Activation("relu"),

        layers.Conv2D(32, (3, 3), padding="same"),
        layers.BatchNormalization(),
        layers.Activation("relu"),

        layers.Flatten(),
        layers.Dense(64, activation="relu"),
        layers.Dense(10, activation="softmax")
    ])
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"]
    )
    return model

bn_cnn = build_bn_cnn()
history_bn = bn_cnn.fit(
    x_train_cnn, y_train,
    validation_data=(x_test_cnn, y_test),
    epochs=5, batch_size=64, verbose=0
)

# Comparaison
plt.plot(history_baseline.history["val_loss"], label="Baseline (No BN)")
plt.plot(history_bn.history["val_loss"], label="With BatchNorm")
plt.xlabel("Epoch"); plt.ylabel("Validation Loss"); plt.legend(); plt.show()

Results observed for BatchNorm: The BatchNorm model starts with a higher validation loss — completely normal because BatchNorm needs a few batches to stabilize its current statistics. But then look what happens: the BatchNorm curve goes down quickly and becomes much more consistent across epochs. The takeaway: Even though BatchNorm starts smoothly, it converges faster and stabilizes the overall training process.

LayerNorm for sequential models:

# Reshape des images en séquences : 28 time steps, 28 features
x_train_seq = x_train.reshape(-1, 28, 28)
x_test_seq = x_test.reshape(-1, 28, 28)

def build_seq_mlp(use_layernorm=False):
    inputs = layers.Input(shape=(28, 28))  # 28 steps, 28 features

    x = inputs
    if use_layernorm:
        x = layers.LayerNormalization(axis=-1)(x)  # normalise à travers les features

    x = layers.TimeDistributed(layers.Dense(64, activation="relu"))(x)
    x = layers.GlobalAveragePooling1D()(x)
    outputs = layers.Dense(10, activation="softmax")(x)

    model = models.Model(inputs, outputs)
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=0.005),
        loss="sparse_categorical_crossentropy",
        metrics=["accuracy"]
    )
    return model

seq_no_ln = build_seq_mlp(use_layernorm=False)
seq_ln = build_seq_mlp(use_layernorm=True)

history_seq_no_ln = seq_no_ln.fit(
    x_train_seq, y_train,
    validation_data=(x_test_seq, y_test),
    epochs=5, batch_size=128, verbose=0
)
history_seq_ln = seq_ln.fit(
    x_train_seq, y_train,
    validation_data=(x_test_seq, y_test),
    epochs=5, batch_size=128, verbose=0
)

plt.plot(history_seq_no_ln.history["val_loss"], label="No LayerNorm")
plt.plot(history_seq_ln.history["val_loss"], label="With LayerNorm")
plt.xlabel("Epoch"); plt.ylabel("Validation Loss"); plt.legend(); plt.show()

LayerNorm results: Without LayerNorm, the validation loss remains higher and the curve is slightly irregular. With LayerNorm applied across features at each time step, the curves become smoother and consistently lower from the first epoch. This is exactly why LayerNorm is so effective in transform models — it stabilizes sequence learning instantly.

Gradient Clipping:

# Modèle dense sur pixels aplatis
x_train_flat = x_train.reshape(-1, 784)
x_test_flat = x_test.reshape(-1, 784)

def build_dense_model(clipnorm=None):
    inputs = layers.Input(shape=(784,))
    x = layers.Dense(128, activation="relu")(inputs)
    x = layers.Dense(64, activation="relu")(x)
    outputs = layers.Dense(10, activation="softmax")(x)

    # SGD avec learning rate élevé pour démontrer les gradients explosifs
    opt = tf.keras.optimizers.SGD(learning_rate=0.1, clipnorm=clipnorm)
    model = models.Model(inputs, outputs)
    model.compile(optimizer=opt,
                  loss="sparse_categorical_crossentropy",
                  metrics=["accuracy"])
    return model

dense_no_clip = build_dense_model(clipnorm=None)
dense_clip = build_dense_model(clipnorm=1.0)  # plafond à norme de gradient = 1.0

history_no_clip = dense_no_clip.fit(
    x_train_flat, y_train,
    validation_data=(x_test_flat, y_test),
    epochs=5, batch_size=128, verbose=0
)
history_clip = dense_clip.fit(
    x_train_flat, y_train,
    validation_data=(x_test_flat, y_test),
    epochs=5, batch_size=128, verbose=0
)

plt.plot(history_no_clip.history["val_loss"], label="No Gradient Clipping")
plt.plot(history_clip.history["val_loss"], label="With Gradient Clipping")
plt.xlabel("Epoch"); plt.ylabel("Validation Loss"); plt.legend(); plt.show()

Gradient Clipping results: With a high learning rate, the model without gradient clipping suddenly spikes losses around epoch 2 — classic explosive gradient behavior. Once clipping is enabled, the validation loss follows a much more controlled downward curve without sudden jumps. Gradient clipping acts like a seat belt — it prevents your model from taking too big steps and keeps the workout on track.

3.2 Mixed Precision Training and GPU efficiency

The problem of precision

When training deep learning models on GPUs, one of the biggest bottlenecks is precision — specifically how many bits we use to represent each number the network processes.

Traditionally, we use FP32 (32-bit floating point), which means that each number takes 32 bits of space. It’s accurate, but it’s memory intensive and slower on modern GPUs.

Mixed Precision Training

With mixed precision training, we pass part of the calculation to FP16 (16-bit), which only uses 16 bits. This:

• Instantly cuts memory usage in half
• Unlocks specialized hardware like Tensor Cores which handle FP16 operations much faster
• Does not lose significant accuracy for most operations

With mixed precision, you can often train larger models, larger batches, and train them faster — all without changing your architecture.

AMP — Automatic Mixed Precision

AMP handles the entire precision switching process automatically. We don’t have to rewrite everything. We simply wrap our forward pass in an autocast context. Inside this block, AMP decides which operations can safely use FP16 and which should stay in FP32 for stability.

Why do some operations remain in FP32? Operations like softmax, log, and some normalization steps do not behave well at half-precision.

GradScaler: It scales the loss to prevent underflow, then reverses the gradients during each backward pass. This keeps the FP16 drive stable without explosive or vanishing gradients.

Code Example — FP32 vs AMP

Construction of the test CNN:

import tensorflow as tf
import time

def build_cnn():
    model = tf.keras.Sequential([
        tf.keras.layers.Conv2D(32, 3, activation="relu"),
        tf.keras.layers.Conv2D(32, 3, activation="relu"),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(64, activation="relu"),
        tf.keras.layers.Dense(10, activation="softmax")
    ])
    return model

Training in FP32 (baseline):

model_fp32 = build_cnn()
model_fp32.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

start = time.time()
history_fp32 = model_fp32.fit(
    x_train_cnn, y_train,
    validation_data=(x_val_cnn, y_val),
    epochs=2, batch_size=128, verbose=0
)
fp32_time = time.time() - start
print(f"FP32 Time: {fp32_time:.2f} sec")

Activation of AMP:

# Activer la politique mixed precision
tf.keras.mixed_precision.set_global_policy("mixed_float16")

Training with AMP:

model_amp = build_cnn()
model_amp.compile(
    optimizer=tf.keras.optimizers.Adam(1e-3),
    loss="sparse_categorical_crossentropy",
    metrics=["accuracy"]
)

start = time.time()
history_amp = model_amp.fit(
    x_train_cnn, y_train,
    validation_data=(x_val_cnn, y_val),
    epochs=2, batch_size=128, verbose=0
)
amp_time = time.time() - start
print(f"AMP Time: {amp_time:.2f} sec")

Performance comparison:

# Visualisation des courbes de val loss
plt.plot(history_fp32.history["val_loss"], label="FP32")
plt.plot(history_amp.history["val_loss"], label="AMP")
plt.xlabel("Epoch"); plt.ylabel("Validation Loss")
plt.legend(); plt.show()

# Résumé de performance
print("=== PERFORMANCE SUMMARY ===")
print(f"FP32 Time: {fp32_time:.2f} sec")
print(f"AMP  Time: {amp_time:.2f} sec")
print(f"Speedup: {fp32_time/amp_time:.2f}x")

Typical results observed:

• FP32: ~10.5 seconds
• AMP: ~9.3 seconds
• Speedup: ~1.13x

This is only a very small 2 epoch model. With deeper CNNs or transformer style models, AMP speed-ups can easily reach 1.5x to 2x or more. This is why all modern training pipelines — from Keras to PyTorch Lightning to Hugging Face — use mixed precision by default.

Takeaway: AMP trains faster, uses less memory, and maintains almost the same accuracy as FP32. This is why mixed precision is an essential optimization technique in deep learning.

3.3 Building efficient input pipelines

The invisible bottleneck

When thinking about training efficiency, most people focus on the model — the layers, the optimizer, the learning rate, the GPU used. But one thing that silently slows down training much more than people expect is the input pipeline.

If the data loader cannot prepare batches fast enough, the GPU finds itself idle — literally waiting for the next batch to arrive. And every millisecond the GPU waits is wasted computation. This delay can come from anywhere:

• Slow disk reads
• Expensive data transformations on the fly
• Too few CPU workers
• Inefficiently organized dataset

Key rule: If your input pipeline is slow, your GPU becomes the victim of the bottleneck.

Key Techniques for an Effective Pipeline

| Technical | Description | Impact | |----------|--------||--------| | Parallel data loading | Increase num_workers to load batches in parallel with multiple CPU threads | Can cut loading time in half | | Prefetching | The loader starts preparing batch N+1 while the GPU trains on batch N | Keeps pipeline full, GPU never waits | | Caching | Cache the dataset in memory or store preprocessed tensors | Eliminate repeated disk reads in subsequent epochs | | Shuffling | Improves generalization and prevents pipeline from reading sequential data | Avoid slow I/O patterns |

Sample Code — PyTorch Pipeline: From Slow to Fast

Setup — imports, dataset, CNN model:

import time
import numpy as np
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader, Subset
from torchvision import datasets, transforms

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# Dataset MNIST avec transform simple
transform = transforms.Compose([transforms.ToTensor()])
mnist_train = datasets.MNIST(root="./data", train=True, download=True, transform=transform)

# Sous-ensemble pour garder la démo rapide
subset_indices = list(range(0, 1024))
mnist_subset = Subset(mnist_train, subset_indices)

# Dataset avec délai artificiel pour simuler un I/O lent (réseau ou disque)
class DelayedDataset(Dataset):
    def __init__(self, base_dataset, delay_per_sample=0.003):
        self.base = base_dataset
        self.delay = delay_per_sample

    def __len__(self):
        return len(self.base)

    def __getitem__(self, idx):
        x, y = self.base[idx]
        time.sleep(self.delay)  # Simule la latence disque/réseau
        return x, y

slow_dataset = DelayedDataset(mnist_subset, delay_per_sample=0.003)

CNN model and timing helper:

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.Conv2d(16, 32, kernel_size=3, padding=1),
            nn.ReLU(),
            nn.AdaptiveAvgPool2d((4, 4)),
            nn.Flatten(),
            nn.Linear(32 * 4 * 4, 10)
        )

    def forward(self, x):
        return self.net(x)

criterion = nn.CrossEntropyLoss()

def run_one_epoch(loader, model, optimizer=None):
    model.to(device)
    model.train()
    batch_times = []
    start_epoch = time.time()

    for xb, yb in loader:
        t0 = time.time()
        xb, yb = xb.to(device), yb.to(device)
        preds = model(xb)
        loss = criterion(preds, yb)

        if optimizer is not None:
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        batch_times.append(time.time() - t0)

    total = time.time() - start_epoch
    return total, float(np.mean(batch_times))

Pipeline slow — num_workers=0:

slow_loader = DataLoader(
    slow_dataset,
    batch_size=64,
    shuffle=True,
    num_workers=0,     # Processus principal charge chaque batch — pas de parallélisme
    pin_memory=False
)

model_slow = SmallCNN()
optimizer_slow = torch.optim.SGD(model_slow.parameters(), lr=0.01)
slow_epoch_time, slow_batch_time = run_one_epoch(slow_loader, model_slow, optimizer_slow)

print("=== Slow pipeline (num_workers=0) ===")
print(f"Epoch time : {slow_epoch_time:.2f} sec")
print(f"Avg time/batch : {slow_batch_time:.4f} sec")

Faster pipeline — num_workers=2:

fast_loader = DataLoader(
    slow_dataset,
    batch_size=64,
    shuffle=True,
    num_workers=2,     # Deux processus de fond chargent des batchs en parallèle
    pin_memory=True    # Mémoire épinglée pour des transferts GPU plus rapides
)

model_fast = SmallCNN()
optimizer_fast = torch.optim.SGD(model_fast.parameters(), lr=0.01)
fast_epoch_time, fast_batch_time = run_one_epoch(fast_loader, model_fast, optimizer_fast)

print("\n=== Faster pipeline (num_workers=2, pin_memory=True) ===")
print(f"Epoch time : {fast_epoch_time:.2f} sec")
print(f"Avg time/batch : {fast_batch_time:.4f} sec")

With prefetching:

prefetch_loader = DataLoader(
    slow_dataset,
    batch_size=64,
    shuffle=True,
    num_workers=2,
    pin_memory=True,
    prefetch_factor=2      # Chaque worker prépare 2 batchs à l'avance
)

model_prefetch = SmallCNN()
optimizer_prefetch = torch.optim.SGD(model_prefetch.parameters(), lr=0.01)
pref_epoch_time, pref_batch_time = run_one_epoch(prefetch_loader, model_prefetch, optimizer_prefetch)

print("\n=== Faster + Prefetch (num_workers=2, prefetch_factor=2) ===")
print(f"Epoch time : {pref_epoch_time:.2f} sec")
print(f"Avg time/batch : {pref_batch_time:.4f} sec")

With caching:

class CachedDataset(Dataset):
    def __init__(self, delayed_dataset):
        # Matérialiser une seule fois (inclut le délai artificiel)
        self.cache = [delayed_dataset[i] for i in range(len(delayed_dataset))]

    def __len__(self):
        return len(self.cache)

    def __getitem__(self, idx):
        return self.cache[idx]  # Retourne directement depuis la RAM

cached_dataset = CachedDataset(slow_dataset)

cached_loader = DataLoader(
    cached_dataset,
    batch_size=64,
    shuffle=True,
    num_workers=2,
    pin_memory=True
)

model_cached = SmallCNN()
optimizer_cached = torch.optim.SGD(model_cached.parameters(), lr=0.01)

# Epoch 1 (données déjà en cache)
cached_epoch1_time, cached_batch1_time = run_one_epoch(cached_loader, model_cached, optimizer_cached)
# Epoch 2 (même données en cache, zéro I/O disque)
cached_epoch2_time, cached_batch2_time = run_one_epoch(cached_loader, model_cached, optimizer_cached)

print("\n=== Cached dataset (même DataLoader, 2 epochs) ===")
print(f"Epoch 1 time : {cached_epoch1_time:.2f} sec,  Avg time/batch: {cached_batch1_time:.4f} sec")
print(f"Epoch 2 time : {cached_epoch2_time:.2f} sec,  Avg time/batch: {cached_batch2_time:.4f} sec")

Performance Summary:

print("\n=== SUMMARY ===")
print(f"Slow loader (workers=0)        ~ {slow_batch_time:.4f} sec / batch")
print(f"Fast loader (workers=2)        ~ {fast_batch_time:.4f} sec / batch")
print(f"Fast + prefetch                ~ {pref_batch_time:.4f} sec / batch")

Typical results observed:

• num_workers=0: ~0.026 sec/batch — GPU mostly idle, waiting for data
• num_workers=2: ~0.0025 sec/batch — reduced batch loading time by ~10x
• prefetch_factor=2: Even smoother pipeline on heavier datasets
• Caching: Both epochs are extremely fast — zero disk I/O when cached

Takeaway: A fast model doesn’t count if your data loader is slow. Optimizing data pipelines is one of the simplest and most effective ways to speed up training. The difference between a GPU that is always busy and a GPU that is just waiting for data.

3.4 Monitor models with TensorBoard or Weights & Biases

Why monitor training visually?

When training deep learning models, the numbers you display in the console only tell a small part of the story. You can see the loss going down, you can see the accuracy improving, but you really don’t know if the training is stable until you start monitoring it visually.

Tools like TensorBoard and Weights & Biases (W&B) allow you to see the entire training process in motion:

• Loss curves
• Precision curves
• Learning rate planning
• The flow of gradients
• Weight distributions

Anything that indicates whether your model learns well or struggles silently.

What becomes obvious with visual monitoring:

• Overlearning: The training loss continues to go down while the validation loss goes up.
• Explosive gradients: Appear like wild spikes.
• Vanishing gradients: Appear as gradients shrinking towards 0.
• Unstable training: Curves seem noisy or chaotic.

Without supervision, you are essentially training blind.

What we should follow in the dashboards

Sample code — TensorBoard with two experiments

TensorBoard setup:

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
from torchvision import datasets, transforms

# Transform basique : convertir les images en tenseurs
transform = transforms.Compose([transforms.ToTensor()])

# Datasets MNIST train & validation
train_ds = datasets.MNIST(root="./data", train=True, download=True, transform=transform)
val_ds   = datasets.MNIST(root="./data", train=False, transform=transform)

train_loader = DataLoader(train_ds, batch_size=64, shuffle=True)
val_loader   = DataLoader(val_ds, batch_size=64, shuffle=False)

Compact CNN model architecture:

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv = nn.Conv2d(1, 16, kernel_size=3, padding=1)
        self.relu = nn.ReLU()
        self.fc   = nn.Linear(16 * 28 * 28, 10)

    def forward(self, x):
        x = self.relu(self.conv(x))
        x = x.view(x.size(0), -1)
        return self.fc(x)

Training function with TensorBoard logging:

def train_one_experiment(run_name, lr):
    writer = SummaryWriter(f"runs/{run_name}")   # Crée un dossier de run séparé

    model = SmallCNN()
    optimizer = optim.Adam(model.parameters(), lr=lr)
    criterion = nn.CrossEntropyLoss()

    step = 0
    for epoch in range(2):  # 2 epochs pour une démo rapide
        model.train()
        for batch, (x, y) in enumerate(train_loader):
            optimizer.zero_grad()
            preds = model(x)
            loss = criterion(preds, y)
            loss.backward()
            optimizer.step()

            # --- Logging TensorBoard par step d'entraînement ---
            writer.add_scalar("Loss/train", loss.item(), step)
            writer.add_scalar("LR", lr, step)
            writer.add_histogram("Weights/conv", model.conv.weight, step)

            step += 1

        # --- Validation à la fin de chaque epoch ---
        model.eval()
        val_loss = 0.0
        correct = 0
        with torch.no_grad():
            for x, y in val_loader:
                preds = model(x)
                val_loss += criterion(preds, y).item()
                correct += (preds.argmax(1) == y).sum().item()

        val_loss /= len(val_loader)
        val_acc = correct / len(val_ds)

        writer.add_scalar("Loss/val", val_loss, epoch)
        writer.add_scalar("Accuracy/val", val_acc, epoch)

    writer.close()

Launch of two experiments with different learning rates:

# Expérience A : learning rate plus élevé
train_one_experiment("exp_lr_01", lr=0.01)

# Expérience B : learning rate plus faible
train_one_experiment("exp_lr_001", lr=0.001)

Launching TensorBoard in Jupyter:

%load_ext tensorboard
%tensorboard --logdir runs

Interpreting TensorBoard views

SCALARS tab:

• The first destination: display of the val accuracy curves for the two runs.
• Two separate lines are visible. The blue run is the lower learning rate, the orange run is the higher rate.
• Even with just a few steps, TensorBoard makes the comparison obvious — one run goes up slightly faster, another is a bit smoother. This visual alone already gives a feeling of what learning rate could be more stable.

DISTRIBUTIONS tab:

• This is where TensorBoard becomes powerful.
• We see the weight distributions evolving over time.
• Orange distribution (higher LR) spreads a bit more aggressively, while blue stays tighter.
• These subtle differences are exactly how you catch problems like unstable training or too large updates before they explode.

HISTOGRAMS view:

• TensorBoard renders the same evolution of weights in a 3D animation style.
• Both runs seem stable here, but we can clearly see different movement patterns between the two learning rates.

8. Summary and conclusion

What we learned in this training

This training gave us the skills to:

1. Evaluate deep learning models with confidence — go beyond simple accuracy, use context-specific metrics, analyze errors, and diagnose training curves.

2. Scientifically tune hyperparameters — understand the impact of learning rate, batch size and optimizers, design efficient search spaces, and choose between grid search and random search.

3. Optimize training — stabilize deep networks with BatchNorm, LayerNorm and gradient clipping, accelerate with mixed precision training, optimize the data pipeline and monitor with tools like TensorBoard.

Summary table of techniques

Full workflow recommended

1. PRÉPARER
   ├── Nettoyage des données
   ├── Split stratifié (train/val/test)
   └── Vérification contre la data leakage

2. ÉVALUER
   ├── Choisir les bonnes métriques (accuracy + precision + recall + AUC)
   ├── Analyser la matrice de confusion
   └── Identifier les FP et FN

3. TUNER
   ├── LR Finder pour trouver le learning rate optimal
   ├── Random Search pour explorer l'espace d'hyperparamètres
   └── Appliquer la régularisation (dropout, L2, early stopping)

4. OPTIMISER L'ENTRAÎNEMENT
   ├── Activer BatchNorm ou LayerNorm
   ├── Configurer le gradient clipping
   ├── Activer AMP (mixed precision)
   ├── Optimiser le pipeline de données (workers, prefetch, cache)
   └── Configurer un LR schedule (cosine annealing, warm restarts)

5. SURVEILLER
   ├── TensorBoard ou Weights & Biases
   ├── Surveiller loss train vs val, LR schedule, histogrammes de poids
   └── Comparer plusieurs expériences côte à côte

Training created by Harsh. Content based on practical demonstrations with Python, TensorFlow/Keras and PyTorch.

Search Terms

deep · model · evaluation · tuning · optimization · neural · networks · machine · data · science · techniques · precision · error · metrics · rate · regularization · analysis · confusion · drive · hyperparameter · important · mixed · schedules · strategies