Build a Speech Recognition Model

Prerequisite level: Python, command line, machine learning concepts, numerical calculation libraries

Table of Contents

1. Prepare and represent audio data

• 1.1 Introduction to audio data for ASR
• 1.2 Collect and preprocess audio
• 1.3 Feature extraction: Spectrograms and MFCCs

2. Module 2 — Train an ASR deep learning model (25m 16s)

• 2.1 Building an acoustic model
• 2.2 Using SpeechBrain for Modeling
• 2.3 Decode sequences with CTC and Attention
• 2.4 Evaluate model performance

3. Optimize and benchmark ASR models

• 3.1 Fine-tuning and hyperparameter optimization
• 3.2 Benchmarking against Whisper
• 3.3 Integrate models into inference pipelines

1. Prepare and represent audio data

1.1 Introduction to audio data for ASR

Course Overview

Welcome to this course Build a Speech Recognition Model on Pluralsight. In this course you will learn how to build a complete speech recognition pipeline. You will learn to:

• Collect and preprocess audio
• Extract spectrograms and MFCCs
• Train an ASR deep learning model
• Decode sequences
• Evaluate performance against existing platforms like OpenAI’s Whisper

Prerequisites:

• Familiarity with Python and the command line
• Machine learning concepts
• Exposure to Numerical Computing Libraries

What is ASR?

ASR (Automatic Speech Recognition) is the technology that allows computers to convert spoken language into written text. At a fundamental level, ASR systems receive an audio signal — essentially a waveform — and analyze it to determine which sounds correspond to human speech, and ultimately which words are spoken.

You interact with the ASR constantly, often without realizing it. Voice assistants, dictation tools and automatic captions all rely on speech recognition models that operate in the background.

Existing ASR platforms

There are several widely used ASR platforms:

These tools are designed to work out of the box and provide good performance in many languages ​​and environments. Although these tools are extremely useful, they often function as black boxes. In this course, we’ll take a closer look at OpenAI’s Whisper and use it as a benchmarking tool to compare its performance to a model you build yourself, helping you understand where convenience ends and personalization begins.

Speech-to-Text (STT) vs Text-to-Speech (TTS)

When working with audio and language, two fundamental concepts come up regularly:

• STT (Speech-to-Text): Focuses on converting spoken audio into written language. This is the main objective of this course.
• TTS (Text-to-Speech): Works in the opposite direction by generating spoken audio from text. While we’re not building a TTS system here, understanding the distinction helps frame how audio flows in modern AI systems and products.

What is audio data?

Unlike text or images, audio data is continuous, time-based, and highly sensitive to noise and recording conditions. Small audio quality issues can lead to big drops in transcription accuracy.

At its most fundamental level, audio is a signal that represents changes in air pressure over time. When someone speaks, their voice creates vibrations. A microphone captures these vibrations and converts them into an electrical signal, which is then digitized into numbers. These numbers form what is called an audio waveform.

Key components of an audio signal:

• Horizontal axis: represents time
• Vertical axis: represents the amplitude or intensity of the signal
• Each point in the waveform is a numerical value

Sampling Rate (Sampling Rate)

Audio signals are sampled at regular intervals. The sampling rate tells us how many measurements are taken per second.

Common sampling rates:

• 16 kHz — standard for speech recognition
• 44.1 kHz — CD quality (music)

For speech recognition, consistency is critical. Models are usually trained with a fixed sample rate, so all audio must be re-sampled to match. If two audio files have different sample rates, they may sound similar to humans, but they look very different digitally. This lag can cause problems during training if not corrected.

Mono vs Stereo

Audio recordings can be mono or stereo:

• Mono: uses only one channel
• Stereo: uses two channels (each channel represents different parts of the sonic spectrum)

Most speech recognition models expect mono audio. This means that stereo recordings generally need to be converted before they can be used.

Common issues in real audio data

Real audio recordings rarely contain clean audio. Common problems include:

• Background noise
• Room echo or reverb
• Inconsistent volume levels
• Long periods of silence

These issues make it more difficult for a model to learn the relationship between sound and text. Even if two recordings contain the same spoken words, differences in noise and volume can significantly change how they appear digitally. This is why preprocessing is a critical step in ASR pipelines.

Analogy: Neural networks do not understand speech. They only understand numbers. If these numbers are inconsistent, noisy, or poorly structured, the model will have difficulty learning. Our human ear is extremely good at extracting speech from noise — our models must compensate for this limitation.

Preprocessing helps ensure that audio clips have:

• Consistent volume
• Reduced noise
• Silence removed
• Standardized length and format

1.2 Collect and preprocess audio

Tools used

For this module, we will use the following tools:

Reminder: Five minutes reading documentation is better than five hours debugging.

Torchaudio in an ASR pipeline acts as the bridge between raw audio files on disk and the digital representations that deep learning models require, allowing developers to preprocess audio, extract features, and prepare datasets using consistent and repeatable operations.

PyTorch Tensors are multidimensional arrays that can efficiently store and manipulate data such as audio waveforms, spectrograms, and model parameters. In a speech recognition pipeline, PyTorch tensors are used to:

• Represent audio signals as digital values
• Apply mathematical transformations during preprocessing and feature extraction
• Serve as inputs and outputs for deep learning models

Folder structure

Before writing any code, let’s configure the folder structure:

# Créer le dossier de travail
mkdir asr_course
cd asr_course

# Créer le script Python
touch torch_test.py

Download the provided audio file and drag and drop it into the folder. The WAV file must be at the same level as the Python script.

Step 1: Load and inspect an audio file

import torchaudio

# Définir le chemin vers le fichier audio
audio_path = "demos.wav"

# Charger le fichier audio
# torchaudio.load retourne deux valeurs : le waveform et le sample rate
waveform, sample_rate = torchaudio.load(audio_path)

# Inspecter les propriétés
print(f"Waveform shape: {waveform.size()}")
print(f"Sample rate: {sample_rate}")

Expected output:

Waveform shape: torch.Size([2, 499670])
Sample rate: 16000

Interpretation:

• The first number (2) indicates the number of channels — stereo audio with 2 channels
• The second number (499,670) indicates the number of samples per channel

Calculation of duration:

# Durée = nombre d'échantillons / taux d'échantillonnage
duration = waveform.size(1) / sample_rate
print(f"Duration: {duration:.2f} seconds")
# Output: Duration: 31.23 seconds

This corresponds to the length of the JFK speech (~31 seconds). Each sample represents a very thin slice of the audio signal, approximately 1/16,000 of a second.

Understanding the number of samples and sampling rate is important because it allows you to:

• Align audio with transcriptions
• Calculate time windows for feature extraction (MFCCs or spectrograms)
• Properly trim or pad audio for training neural networks

Step 2: Stereo to Mono Conversion

Most ASR models expect mono audio. Here’s how to handle both cases:

import torchaudio
import torch

audio_path = "demos.wav"
waveform, sample_rate = torchaudio.load(audio_path)

# Vérifier si l'audio est stéréo et convertir en mono si nécessaire
if waveform.size(0) > 1:
    # Calculer la moyenne sur tous les canaux
    # keepdim=True préserve la forme attendue : (1, nb_echantillons)
    waveform = waveform.mean(dim=0, keepdim=True)

print(f"Mono waveform shape: {waveform.size()}")
# Output: Mono waveform shape: torch.Size([1, 499670])

Step 3: Amplitude normalization

Audio recordings can have very different volume levels. Neural networks don’t understand volume like humans do.

# Normaliser le waveform
# Diviser par la valeur absolue maximale pour que le point le plus fort ait une valeur de 1
waveform = waveform / waveform.abs().max()

print("Audio normalisé avec succès")

Normalization:

• Does not change spoken words or speech timing
• Only adjusts the overall volume
• Gives an easier to learn standardized signal for a model

Step 4: Noise reduction by energy thresholding

For small datasets, we can apply a practical technique called energy-based thresholding. The idea is simple: Very quiet parts of the audio are more likely to be background noise than speech.

# Seuillage énergétique simple pour réduire le bruit de fond
noise_threshold = 0.005

# Les échantillons sous le seuil sont remplacés par zéro
waveform = torch.where(waveform.abs() > noise_threshold, waveform, torch.zeros_like(waveform))

print("Réduction du bruit appliquée")

This technique does not remove all noise, but reduces low-level background noise and makes the speech signal clearer and more coherent.

Step 5: Voice Activity Detection (VAD) — Removing Silence

We use Torchaudio to apply a form of Voice Activity Detection (VAD) to identify where speech begins and ends.

import torchaudio.transforms as T
import torchaudio.functional as F

# Afficher la forme originale
print(f"Forme originale: {waveform.size()}")

# Appliquer le VAD pour supprimer le silence en début et fin
# torchaudio.functional.vad supprime les régions de silence
waveform, _ = torchaudio.functional.vad(waveform, sample_rate=sample_rate)

print(f"Forme après suppression du silence: {waveform.size()}")
# La réduction du nombre d'échantillons confirme que le silence a été supprimé
# Environ 4800 échantillons supprimés = ~0.3 secondes de silence

Step 6: Resampling

Modern speech recognition models expect audio sampled at 16 kHz. Real datasets often contain files at different sample rates.

# Taux cible standard pour la reconnaissance vocale
target_sample_rate = 16000

# Ré-échantillonner si nécessaire
if sample_rate != target_sample_rate:
    resampler = torchaudio.transforms.Resample(
        orig_freq=sample_rate,
        new_freq=target_sample_rate
    )
    waveform = resampler(waveform)
    sample_rate = target_sample_rate
    print(f"Ré-échantillonné à {target_sample_rate} Hz")
else:
    print(f"Sample rate déjà à {sample_rate} Hz, aucune action requise")

Complete preprocessing pipeline

At this point we have applied a full preprocessing pipeline. Here is the full script with comments:

import torch
import torchaudio
import torchaudio.functional as F_audio

def preprocess_audio(audio_path, target_sample_rate=16000, noise_threshold=0.005):
    """
    Pipeline de prétraitement audio complet pour l'ASR.
    
    Étapes :
    1. Charger l'audio
    2. Convertir en mono
    3. Normaliser l'amplitude
    4. Réduire le bruit (seuillage énergétique)
    5. Supprimer le silence (VAD)
    6. Ré-échantillonner à 16 kHz
    """
    # 1. Charger le fichier audio
    waveform, sample_rate = torchaudio.load(audio_path)
    print(f"Chargé: shape={waveform.size()}, sample_rate={sample_rate}")

    # 2. Convertir en mono
    if waveform.size(0) > 1:
        waveform = waveform.mean(dim=0, keepdim=True)
    print(f"Après conversion mono: {waveform.size()}")

    # 3. Normaliser l'amplitude
    waveform = waveform / waveform.abs().max()
    print("Normalisé")

    # 4. Réduction du bruit par seuillage énergétique
    waveform = torch.where(
        waveform.abs() > noise_threshold,
        waveform,
        torch.zeros_like(waveform)
    )
    print("Débruité")

    # 5. Supprimer le silence avec VAD
    waveform, _ = F_audio.vad(waveform, sample_rate=sample_rate)
    print(f"Après suppression du silence: {waveform.size()}")

    # 6. Ré-échantillonner si nécessaire
    if sample_rate != target_sample_rate:
        resampler = torchaudio.transforms.Resample(
            orig_freq=sample_rate,
            new_freq=target_sample_rate
        )
        waveform = resampler(waveform)
        sample_rate = target_sample_rate
        print(f"Ré-échantillonné à {target_sample_rate} Hz")

    print(f"Prétraitement terminé: shape={waveform.size()}, sample_rate={sample_rate}")
    return waveform, sample_rate

# Point d'entrée principal
if __name__ == "__main__":
    waveform, sr = preprocess_audio("demos.wav")

Pipeline Summary:

• Loaded into memory → Converted to mono → Normalized for consistent volume → Denoised to reduce background noise → Silence removed → Re-sampled to 16 kHz

1.3 Feature extraction: Spectrograms and MFCCs

Why extract features?

Until now, we have worked directly with raw waveforms — long sequences of amplitude values ​​over time. Although waveforms are great for recording and basic preprocessing, they are not ideal for neural networks.

Feature extraction solves this problem by transforming raw audio into structured digital representations that highlight speech-relevant patterns. Features like spectrograms and MFCCs highlight how energy is distributed across frequencies and how these frequencies evolve over time.

Advantages:

1. Much easier for a model to learn the distinctions between phonemes, syllables and words
2. Enforces consistency — instead of raw variable-length signals, we produce tensors with predictable dimensions (required for batching)

From time domain to frequency domain

We represented the audio purely in the time domain (amplitude values ​​changing over time). But speech is best understood in the frequency domain where we can see what frequencies are present at each moment.

• Different speech sounds activate different frequency ranges
• These patterns are what speech models learn to recognize

Spectrograms

A spectrogram captures the idea by dividing the audio into short time segments and calculating the frequency content in each window. The result is a two-dimensional representation:

This transformation gives us a much richer and more informative view of speech than a single waveform line.

# Fichier : torch_feature_extraction.py
import torch
import torchaudio
import torchaudio.transforms as T

# Charger et prétraiter l'audio (depuis notre pipeline précédent)
waveform, sample_rate = torchaudio.load("demos.wav")
if waveform.size(0) > 1:
    waveform = waveform.mean(dim=0, keepdim=True)

# Définir la transformation spectrogramme
# n_fft : taille de la FFT (Fast Fourier Transform)
# win_length : taille de la fenêtre d'analyse
# hop_length : pas entre les fenêtres
spectrogram_transform = T.Spectrogram(
    n_fft=400,
    win_length=400,
    hop_length=160
)

# Appliquer la transformation au waveform prétraité
spectrogram = spectrogram_transform(waveform)

# Vérifier la forme du spectrogramme
print(f"Spectrogram shape: {spectrogram.size()}")

Expected output:

Spectrogram shape: torch.Size([2, 201, 3123])

Interpretation:

• 2: number of audio channels
• 201: number of frequency bins
• 3,123: number of temporal frames

The original waveform was split into 3,123 overlapping time windows. For each window, the spectrogram records how much energy is present at each frequency bin. This is exactly what we want for speech recognition — instead of feeding raw samples to the model, we give it a structured view of how speech energy evolves over time and frequency.

MFCCs (Mel-Frequency Cepstral Coefficients)

Although spectrograms capture raw frequency information, MFCCs go further by modeling how humans perceive sound. Human hearing is more sensitive to certain frequency ranges, particularly those relevant to speech.

How MFCCs are calculated:

1. Apply a Mel-scale filter bank to a spectrogram
2. Taking logarithms
3. Summarize the result using cepstral coefficients

This process reduces dimensionality while preserving speech-critical information. Thanks to this balance between compactness and expressiveness, MFCCs are one of the most widely used features in speech recognition systems.

# Ajout au fichier torch_feature_extraction.py

# Définir la transformation MFCC
# n_mfcc=13 : choix standard qui équilibre puissance expressive et efficacité computationnelle
mfcc_transform = T.MFCC(
    sample_rate=sample_rate,
    n_mfcc=13,           # Nombre de coefficients MFCC
    melkwargs={
        "n_fft": 400,
        "n_mels": 40,    # Nombre de filtres Mel
        "hop_length": 160,
    }
)

# Appliquer la transformation
mfccs = mfcc_transform(waveform)

# Vérifier la forme
print(f"MFCC shape: {mfccs.size()}")

Expected output:

MFCC shape: torch.Size([2, 13, 2499])

Interpretation:

• 2: audio channels
• 13: MFCC coefficients per frame (compact representation)
• 2,499: timestamps on the record

This confirms that the raw audio has been successfully transformed into a temporally aligned representation that can be fed directly into speech recognition models.

Choice of number of coefficients:

• Fewer coefficients → more compact representation
• More coefficients → preserves finer spectral details
• 13 is the standard that balances the two

Audio segmentation in fixed windows

Voice recordings are often long, but models train more effectively on shorter, coherent segments. Segmentation helps to:

• Check input size
• Improve drive stability
• Increase dataset size by creating more examples

def segment_features(feature_tensor, segment_frames=100):
    """
    Divise un tenseur de features basé sur le temps en segments de longueur fixe.
    
    Args:
        feature_tensor: Tenseur de features (ex: MFCCs) de forme (canaux, features, temps)
        segment_frames: Nombre de frames par segment
    
    Returns:
        Liste de tenseurs, chaque segment ayant `segment_frames` frames
    """
    total_frames = feature_tensor.size(-1)  # Dimension temporelle
    segments = []
    
    for start in range(0, total_frames, segment_frames):
        end = start + segment_frames
        if end <= total_frames:
            segment = feature_tensor[..., start:end]
            segments.append(segment)
    
    return segments

# Appliquer la segmentation aux MFCCs
segments = segment_features(mfccs, segment_frames=100)
print(f"Nombre de segments: {len(segments)}")

if segments:
    print(f"Forme d'un segment: {segments[0].size()}")

Features-transcription alignment

For supervised speech recognition, each audio segment must be paired with the correct transcription. This alignment teaches the model how sound patterns correspond to language.

Key requirements:

• All feature tensors must share consistent dimensions
• Each feature tensor must match the correct text label
• Errors at this step can severely impact model performance

Rule of thumb: After feature extraction, segmentation and alignment, our dataset should follow a predictable structure allowing the data to be loaded in batches, mixed and fed directly into a training loop.

2. Training an ASR deep learning model

2.1 Building an acoustic model

Role of acoustic models

Acoustic models are at the heart of speech recognition systems. They take sequences of audio features (spectrograms or MFCCs) and map them to sequences of textual tokens. By learning patterns in the audio over time, these models can predict what is being said.

RNNs and LSTMs

RNN (Recurrent Neural Network): Type of neural network specifically designed to process sequential data by maintaining a memory of previous inputs in a hidden state.

LSTM (Long Short-Term Memory): Specialized type of RNN that uses gate mechanisms to better capture long-term dependencies in sequences.

Transformers revolutionize sequential modeling by using attention mechanisms instead of step-by-step recurrence. This allows the model to focus on the most relevant parts of the input sequence, capturing long-range dependencies efficiently.

Note: Transformers underlie not only many interpretive models like ours, but also most common vocal and multimodal generative models — from Claude to BERT and GPT, transformers are powerful and ubiquitous.

Choice of architecture

2.2 Using SpeechBrain for modeling

SpeechBrain Overview

SpeechBrain is an open-source toolkit for speech processing built on PyTorch. It provides:

• Pre-built acoustic models
• Feature extraction utilities
• Drive pipelines

This makes it much easier to train or experiment with speech models without implementing all the mechanics of low-level neural networks from scratch.

SpeechBrain supports: – CTC and attention-based decoding

• Patching, masking, and preprocessing
• Variable length input variables

Implementation: speech_brain_model.py

# speech_brain_model.py
# Démonstration de l'utilisation de SpeechBrain pour définir et tester
# un modèle acoustique basé sur LSTM pour la reconnaissance vocale

import torch
from speechbrain.lobes.features import Fbank
from speechbrain.nnet.linear import Linear
from speechbrain.nnet.RNN import LSTM
from speechbrain.nnet.normalization import LayerNorm
from speechbrain.nnet.containers import Sequential

# ================================================
# ÉTAPE 1 : Extraction de features avec Fbank
# ================================================
# Le module Fbank convertit des formes d'onde audio brutes en log-Mel spectrogrammes
# Chaque segment de ~25ms est transformé en vecteur 80-dimensionnel
# représentant son contenu fréquentiel sur l'échelle Mel
feature_extractor = Fbank(
    n_mels=80,      # 80 bins de fréquence Mel
    deltas=False    # Pas de features de vélocité/accélération
)

# ================================================
# ÉTAPE 2 : Modèle acoustique LSTM bidirectionnel
# ================================================
# Ce modèle mappe des séquences de features audio vers des logits de tokens
# Architecture : LayerNorm → BiLSTM → Linear

acoustic_model = Sequential(
    # Couche 1 : Normalisation des features d'entrée (stabilise l'entraînement)
    LayerNorm(input_size=80),
    
    # Couche 2 : Cœur du modèle — LSTM bidirectionnel
    # input_size=80 correspond à la sortie du feature extractor
    # hidden_size=256 : dimensionnalité de l'état interne du LSTM
    # num_layers=2 : deux couches LSTM empilées pour une représentation plus profonde
    # dropout=0.1 : régularisation pour éviter l'overfitting
    # bidirectional=True : traite la séquence dans les deux directions
    LSTM(
        input_size=80,
        hidden_size=256,
        num_layers=2,
        dropout=0.1,
        bidirectional=True
    ),
    
    # Couche 3 : Projection linéaire vers l'espace du vocabulaire
    # input_size=512 car bidirectionnel double la dimensionnalité (256 * 2)
    # n_neurons=29 : vocabulaire de 29 tokens (26 lettres + blank CTC + espace + apostrophe)
    Linear(input_size=512, n_neurons=29, bias=True)
)

# ================================================
# ÉTAPE 3 : Afficher et vérifier l'architecture
# ================================================
print(acoustic_model)

# ================================================
# ÉTAPE 4 : Test avec une entrée fictive
# ================================================
# Créer un tenseur représentant 2 échantillons audio, chacun de 3 secondes à 16 kHz
# Forme : (batch_size=2, samples=48000)
dummy_audio = torch.randn(2, 48000)

# Extraire les features Mel
# Les 48000 échantillons deviennent ~300 frames temporelles
features = feature_extractor(dummy_audio)
print(f"Features shape: {features.shape}")  # (2, ~300, 80)

# Passer les features dans le modèle acoustique
# Sortie : (batch_size, time_frames, vocab_size)
output = acoustic_model(features)
print(f"Output shape: {output.shape}")  # (2, ~301, 29)

Expected output:

Sequential(
  (0): LayerNorm(...)
  (1): LSTM(input_size=80, hidden_size=256, num_layers=2, ...)
  (2): Linear(in_features=512, out_features=29, bias=True)
)

Features shape: torch.Size([2, 301, 80])
Output shape: torch.Size([2, 301, 29])

Detailed architecture analysis

Layer 1 — LayerNorm:

• Normalize 80-dimensional Mel features to have consistent ranges
• Ensures all features have similar ranges → faster and more stable training
• element_wise_affine=True means it learns scale and offset parameters

Layer 2 — LSTM Bidirectional:

• Core model that captures how speech sounds change over time
• Takes 80-dimensional features and maintains a 256-dimensional hidden state
• Bidirectional means that the LSTM processes the audio sequence in both directions:
• LSTM front: bed from left to right (past → future)
• Rear LSTM: bed from right to left (future → past)
• Each frame sees the context of before AND after

Why bidirectional? The pronunciation of “read” depends on the context (present or past). With bidirectional LSTM, each frame can use information from subsequent frames to resolve these ambiguities.

Layer 3 — Linear:

• Converts 512-dimensional LSTM outputs into predictions for 29 different tokens
• At each time frame, the model produces 29 values (logits) — one per token

Data flow:

Audio brut (48000 échantillons)
         ↓
Feature Extraction → (301, 80) : 301 frames, 80 features Mel chacune
         ↓
LayerNorm → (301, 80) : même forme, valeurs normalisées
         ↓
BiLSTM → (301, 512) : chaque frame enrichie avec le contexte bidirectionnel
         ↓
Linear → (301, 29) : scores de tokens à chaque frame

Interpretation of logits: At each time frame, the model produces 29 numbers called logits. For example, at frame 150: [1.2, -2.1, 3.5, ...] where each number corresponds to a token (A, B, C, …). Higher values ​​indicate a higher probability for that token. These raw scores are not yet probabilities — they are fed into a decoder like CTC beam search.

2.3 Decode sequences with CTC and Attention

The decoding problem

We have a model that produces raw logits every time frame, but these outputs are not yet readable text. Decoding transforms network outputs into human-readable transcripts.

CTC (Connectionist Temporal Classification)

CTC is a decoding method that allows our acoustic model to handle variable-length input sequences and map them to shorter sequences of textual tokens. This is critical in ASR because audio clips rarely match the length of the transcript exactly.

CTC Characteristics:

• Introduces a special token blank (empty)
• Consider all valid alignments between input frames and output tokens
• Allows the model to predict sequences without labels at the frame level

CTC alignment example:

Audio frames :  A A A _ C _ _ A _ T
                ↓  Collapse ↓
Transcription : A C A T

(Repeated tokens and blanks are merged into the final transcription)

CTC loss implementation: ctc_test.py

# ctc_test.py
# Démonstration du calcul de la perte CTC pour un modèle acoustique

import torch
import torch.nn as nn

# Créer une instance de la perte CTC
# blank=28 : index du token spécial "blank" dans notre vocabulaire de 29 tokens
ctc_loss = nn.CTCLoss(blank=28)

# ================================================
# Simuler des sorties de modèle (logits)
# Dimensions : (time_steps, batch_size, output_dim)
# ================================================
# 50 frames temporelles, 1 batch, 29 tokens possibles
time_steps = 50
batch_size = 1
output_dim = 29  # 26 lettres + blank + espace + apostrophe

logits = torch.randn(time_steps, batch_size, output_dim)

# Convertir en log-probabilités (requis par CTCLoss)
log_probs = torch.log_softmax(logits, dim=2)

# ================================================
# Cibles (transcriptions de référence)
# ================================================
# Séquence de 10 indices de tokens cibles
# (en pratique, ce sont les indices des caractères de la transcription)
targets = torch.randint(0, 28, (batch_size * 10,))  # 10 tokens cibles

# Longueurs valides des entrées et des cibles
input_lengths = torch.full((batch_size,), time_steps, dtype=torch.long)
target_lengths = torch.full((batch_size,), 10, dtype=torch.long)

# ================================================
# Calculer la perte CTC
# ================================================
# PyTorch calcule la vraisemblance log négative sur tous les alignements valides
loss = ctc_loss(log_probs, targets, input_lengths, target_lengths)

print(f"CTC Loss: {loss.item():.4f}")

Expected output:

CTC Loss: 14.2345

Interpretation:

• With random data the loss is high (~14) because the model has no idea of the correct transcription
• With a trained model: loss around 2 → good, 0.5 → excellent

How CTC loss works internally:

1. The log_probs represent the model output on temporal frames
2. log_softmax(dim=2) converts logits into log-probabilities on tokens
3. The targets are the reference sequences of tokens
4. input_lengths and target_lengths tell CTC which portions are valid
5. CTC considers all valid alignments (ex: cat can align like C-CA-T or CCAAT-)
6. The loss is the negative log likelihood summed over all these alignments

Decoding by attention mechanism

Attention mechanisms are a key component of modern sequence-to-sequence models. They allow the model to dynamically focus on different parts of the input sequence when predicting each output token.

Differences vs CTC:

# Démonstration d'un décodeur avec attention (SpeechBrain)
import torch
from speechbrain.nnet.attention import LocationAwareAttention
from speechbrain.nnet.RNN import AttentionalRNNDecoder

# Simuler des sorties du encodeur
# Forme : (batch_size, time_steps, hidden_size)
encoder_outputs = torch.randn(1, 50, 256)
encoder_lengths = torch.tensor([50])

# Entrées du décodeur (embeddings des tokens)
# Forme : (batch_size, decoder_time, embedding_dim)
decoder_inputs = torch.randn(1, 10, 128)

# Le décodeur utilise une attention location-aware
# Spécialement conçue pour la reconnaissance vocale
# (considère les poids d'attention précédents pour encourager un alignement monotonique)
decoder = AttentionalRNNDecoder(
    rnn_type="gru",
    attn_type="location",
    hidden_size=256,
    attn_dim=256,
    num_layers=2,
    input_size=128,
    channels=10,
    kernel_size=100
)

# Projection vers l'espace du vocabulaire
output_projection = torch.nn.Linear(256, 29)

# Passe avant
decoder_output, attention_weights = decoder(
    encoder_outputs,
    encoder_lengths,
    decoder_inputs
)

# Projeter vers les prédictions de tokens
token_predictions = output_projection(decoder_output)

print(f"Decoder hidden states: {decoder_output.shape}")   # (1, 10, 256)
print(f"Token predictions: {token_predictions.shape}")    # (1, 10, 29)

Expected output:

Decoder hidden states: torch.Size([1, 10, 256])
Token predictions: torch.Size([1, 10, 29])

Interpretation:

• 10 hidden states: the decoder has processed the encoder outputs carefully
• 10 × 29: for each position in the output sequence, 29 token scores
• By applying softmax you obtain a probability distribution over the 29 possible characters

2.4 Evaluate model performance

ASR Evaluation Metrics

To quantify the accuracy of our model, we use two main metrics:

WER (Word Error Rate) — Word error rate:

• Evaluates the accuracy of predicted words against the reference transcription
• Counts word-level substitutions, insertions and deletions

CER (Character Error Rate) — Character error rate:

• Similar comparison at character level
• Particularly useful for languages with highly variable word structures

$$WER = \frac{S + D + I}{N}$$

Where:

• $S$ = Substitutions
• $D$ = Deletions (Deletions)
• $I$ = Insertions
• $N$ = Total number of words in the reference

Performance thresholds:

Implementation: evaluation_test.py

# evaluation_test.py
# Mesurer la précision d'un modèle ASR avec jiwer

from jiwer import wer, cer

# ================================================
# Données de test
# ================================================
# Transcriptions correctes (vérité terrain)
ground_truths = [
    "we choose to go to the moon",
    "not because it is easy but because it is hard",
    "that goal will serve to organize and measure the best of our energies"
]

# Sorties du modèle (prédictions)
# Note : fautes de frappe intentionnelles pour la démonstration
predictions_list = [
    "we choos to go to the moon",              # "choose" → "choos" (1 erreur)
    "not becaus it is easy but because it is hard",  # "because" → "becaus" (1 erreur)
    "that goal will serve to organize and mesure the best of our energies"  # "measure" → "mesure"
]

# ================================================
# Calculer les métriques
# ================================================
# WER : proportion d'erreurs au niveau des mots
word_error_rate = wer(ground_truths, predictions_list)

# CER : mesure plus fine au niveau des caractères
character_error_rate = cer(ground_truths, predictions_list)

print(f"Word Error Rate (WER):      {word_error_rate * 100:.2f}%")
print(f"Character Error Rate (CER): {character_error_rate * 100:.2f}%")

Expected output:

Word Error Rate (WER):      10.53%
Character Error Rate (CER):  3.21%

These metrics help you:

• Quantify how well your model matches the reference text
• Identify error patterns
• Compare different ASR decoding models or methods

Improved LSTM model

After an initial evaluation and error analysis, the next phase is iterative improvement:

# improved_model_test.py
# Modèle acoustique LSTM amélioré avec une plus grande capacité

import torch
import torch.nn as nn

class ImprovedLSTMAcousticModel(nn.Module):
    """
    Modèle acoustique LSTM amélioré pour l'ASR.
    
    Améliorations par rapport au modèle de base :
    - 4 couches LSTM (au lieu de 2) pour des représentations hiérarchiques
    - hidden_dim=512 (au lieu de 256) pour plus de capacité
    - dropout=0.2 pour une meilleure régularisation
    """
    
    def __init__(self, input_dim=80, hidden_dim=512, output_dim=29, num_layers=4):
        super(ImprovedLSTMAcousticModel, self).__init__()
        
        # Core LSTM bidirectionnel avec grande capacité
        self.lstm = nn.LSTM(
            input_size=input_dim,
            hidden_size=hidden_dim,        # Plus grande mémoire interne
            num_layers=num_layers,          # 4 couches pour représentations hiérarchiques
            batch_first=True,               # (batch, time, features)
            dropout=0.2,                    # Régularisation entre les couches
            bidirectional=True              # Contexte passé et futur
        )
        
        # Couche de sortie
        # in_features = hidden_dim * 2 car bidirectionnel concatène les sorties
        self.fc = nn.Linear(hidden_dim * 2, output_dim, bias=True)
    
    def forward(self, x):
        """
        Passe avant.
        
        Args:
            x: Tenseur d'entrée de forme (batch_size, time_steps, input_features)
        
        Returns:
            Logits de forme (batch_size, time_steps, output_dim)
        """
        # Traitement LSTM
        lstm_output, _ = self.lstm(x)
        
        # Projection vers le vocabulaire
        logits = self.fc(lstm_output)
        
        return logits

# ================================================
# Initialiser et analyser le modèle
# ================================================
model = ImprovedLSTMAcousticModel(
    input_dim=80,
    hidden_dim=512,
    output_dim=29,
    num_layers=4
)

# Compter les paramètres
total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"Paramètres totaux du modèle: {total_params:,}")
# Output: Paramètres totaux du modèle: 21,087,261

print("\nArchitecture du modèle:")
print(model)

Parameter analysis:

Paramètres totaux du modèle: 21,087,261

Architecture du modèle:
ImprovedLSTMAcousticModel(
  (lstm): LSTM(80, 512, num_layers=4, batch_first=True, dropout=0.2, bidirectional=True)
  (fc): Linear(in_features=1024, out_features=29, bias=True)
)

Where do the 21 million parameters come from?

The vast majority comes from LSTM layers. Each LSTM cell contains several weight matrices for:

• The entrance gate (input gate)
• Forget gate
• The output gate (output gate)
• Calculating cell state

With 4 layers, 512 hidden units and bidirectional processing, the number of parameters grows quickly.

Warning: Greater model capacity requires more training data and computational resources. ASR engineering, like all engineering, is fundamentally iterative. You will hypothesize about potential improvements, implement those changes, evaluate the results, and refine your approach.

3. Optimize and benchmark ASR models

3.1 Fine-tuning and hyperparameter optimization

Complete ASR production pipeline

In this final module, we build a complete SpeechBrainASR class that encapsulates four critical steps:

1. Loading and preprocessing audio
2. Feature extraction
3. Acoustic modeling
4. Transcription

# speechbrain_asr.py
# Pipeline ASR complet de production

import torch
import torchaudio
import torchaudio.functional as F_audio
from speechbrain.lobes.features import Fbank
from speechbrain.nnet.linear import Linear
from speechbrain.nnet.RNN import LSTM
from speechbrain.nnet.normalization import LayerNorm
from speechbrain.nnet.containers import Sequential

class SpeechBrainASR(torch.nn.Module):
    """
    Système ASR complet et prêt pour la production.
    
    Encapsule :
    1. Prétraitement audio (mono, normalisation, débruitage, VAD, rééchantillonnage)
    2. Extraction de features (log-Mel spectrogramme à 80 bandes)
    3. Modèle acoustique (BiLSTM + normalisation + projection)
    4. Transcription (greedy decode)
    """
    
    def __init__(self, vocab_size=29, sample_rate=16000):
        super(SpeechBrainASR, self).__init__()
        
        self.vocab_size = vocab_size
        self.sample_rate = sample_rate
        
        # Extracteur de features : 80 bandes mel (standard industriel)
        self.feature_extractor = Fbank(
            n_mels=80,
            deltas=False
        )
        
        # Modèle acoustique : 3 couches empilées
        self.acoustic_model = Sequential(
            # 1. Normalisation pour stabiliser l'entraînement
            LayerNorm(input_size=80),
            
            # 2. LSTM bidirectionnel : 256 unités cachées, 2 couches
            LSTM(
                input_size=80,
                hidden_size=256,
                num_layers=2,
                dropout=0.1,
                bidirectional=True
            ),
            
            # 3. Projection linéaire vers le vocabulaire (29 caractères)
            Linear(input_size=512, n_neurons=vocab_size, bias=True)
        )
        
        # Vocabulaire de 29 tokens : 26 lettres + blank CTC + espace + apostrophe
        self.chars = list("abcdefghijklmnopqrstuvwxyz") + ["'", " ", "_"]
        self.char_to_idx = {c: i for i, c in enumerate(self.chars)}
        self.idx_to_char = {i: c for i, c in enumerate(self.chars)}
    
    def load_audio(self, audio_path):
        """
        Charger et prétraiter un fichier audio.
        
        Applique :
        - Conversion en mono
        - Normalisation d'amplitude
        - Réduction du bruit (seuillage énergétique)
        - Suppression du silence (VAD)
        - Ré-échantillonnage à 16 kHz
        """
        waveform, sr = torchaudio.load(audio_path)
        
        # Convertir en mono
        if waveform.size(0) > 1:
            waveform = waveform.mean(dim=0, keepdim=True)
        
        # Normaliser l'amplitude
        waveform = waveform / (waveform.abs().max() + 1e-8)
        
        # Réduire le bruit (seuillage énergétique)
        waveform = torch.where(
            waveform.abs() > 0.005,
            waveform,
            torch.zeros_like(waveform)
        )
        
        # Supprimer le silence avec VAD
        waveform, _ = F_audio.vad(waveform, sample_rate=sr)
        
        # Ré-échantillonner si nécessaire
        if sr != self.sample_rate:
            resampler = torchaudio.transforms.Resample(
                orig_freq=sr,
                new_freq=self.sample_rate
            )
            waveform = resampler(waveform)
        
        return waveform
    
    def extract_features(self, waveform):
        """Convertir le waveform en log-Mel features."""
        return self.feature_extractor(waveform)
    
    def forward(self, features):
        """Exécuter le réseau de neurones."""
        return self.acoustic_model(features)
    
    def transcribe(self, audio_path):
        """
        Pipeline complet : du fichier audio à la sortie du modèle.
        
        Args:
            audio_path: Chemin vers le fichier audio
        
        Returns:
            Logits du modèle (tenseur 3D)
        """
        self.eval()
        with torch.no_grad():
            waveform = self.load_audio(audio_path)
            features = self.extract_features(waveform)
            logits = self.forward(features)
        return logits
    
    def train_model(self, train_loader, val_loader, epochs=20,
                    learning_rate=0.001, grad_clip=5.0):
        """
        Boucle d'entraînement complète.
        
        Args:
            train_loader: DataLoader de données d'entraînement
            val_loader: DataLoader de données de validation
            epochs: Nombre d'époques
            learning_rate: Taux d'apprentissage initial
            grad_clip: Valeur de clipping du gradient
        """
        ctc_loss_fn = torch.nn.CTCLoss(blank=28, zero_infinity=True)
        
        # Adam est le choix standard pour l'ASR
        # Il adapte le taux d'apprentissage pour chaque paramètre individuellement
        optimizer = torch.optim.Adam(self.parameters(), lr=learning_rate)
        
        # Réduction du learning rate sur plateau
        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
            optimizer, mode='min', factor=0.5, patience=3, verbose=True
        )
        
        for epoch in range(epochs):
            # ─── Phase d'entraînement ───────────────────────────────
            self.train()
            train_loss = 0.0
            
            for batch_features, batch_targets, input_lengths, target_lengths in train_loader:
                optimizer.zero_grad()
                
                # Passe avant
                logits = self.forward(batch_features)
                
                # CTC loss nécessite (time, batch, vocab) au lieu de (batch, time, vocab)
                log_probs = torch.log_softmax(logits, dim=-1).permute(1, 0, 2)
                
                # Calculer la perte
                loss = ctc_loss_fn(log_probs, batch_targets, input_lengths, target_lengths)
                
                # Rétropropagation
                loss.backward()
                
                # Gradient clipping (CRUCIAL pour les LSTMs)
                # Empêche les gradients de trop grandir
                torch.nn.utils.clip_grad_norm_(self.parameters(), grad_clip)
                
                optimizer.step()
                train_loss += loss.item()
            
            avg_train_loss = train_loss / len(train_loader)
            
            # ─── Phase de validation ────────────────────────────────
            self.eval()
            val_loss = 0.0
            
            with torch.no_grad():
                for batch_features, batch_targets, input_lengths, target_lengths in val_loader:
                    logits = self.forward(batch_features)
                    log_probs = torch.log_softmax(logits, dim=-1).permute(1, 0, 2)
                    loss = ctc_loss_fn(log_probs, batch_targets, input_lengths, target_lengths)
                    val_loss += loss.item()
            
            avg_val_loss = val_loss / len(val_loader)
            
            print(f"Epoch {epoch+1}/{epochs} | "
                  f"Train Loss: {avg_train_loss:.4f} | "
                  f"Val Loss: {avg_val_loss:.4f}")
            
            scheduler.step(avg_val_loss)
        
        return self

Key architectural parameters

Training hyperparameters

Debugging and fine-tuning strategies

The golden rule of fine-tuning: Change one thing at a time and measure the impact, then iterate until you achieve your goals.

Data increase for ASR

To improve the robustness of the model:

• Add noise: Expose the model to different types of background noise
• Speed Disruption: People speak at different speeds
• Tone disturbance: People speak with different tones

These techniques allow the model to generalize to new audio conditions not seen during training.

Use a SpeechBrain pre-trained model

To get fast results without thousands of hours of training data, SpeechBrain provides pre-trained models:

# Utiliser un modèle ASR pré-entraîné de SpeechBrain
# Ce modèle a été entraîné sur 960 heures d'anglais (dataset LibriSpeech)
# Il combine un CRDNN (Convolutional Recurrent Deep Neural Network)
# avec un modèle de langage RNN pour une précision état-de-l'art

from speechbrain.pretrained import EncoderDecoderASR

# Télécharge et met en cache automatiquement le modèle au premier usage
asr_model = EncoderDecoderASR.from_hparams(
    source="speechbrain/asr-crdnn-rnnlm-librispeech",
    savedir="pretrained_models/asr-crdnn-rnnlm-librispeech"
)

# Transcrire un fichier audio
# Le modèle gère automatiquement tout le pipeline :
# - Chargement et ré-échantillonnage
# - Extraction de features acoustiques
# - Décodage avec le modèle de langage
transcription = asr_model.transcribe_file("demos.wav")
print(f"Transcription : {transcription}")

Expected training progress:

• Epoch 1-5: High loss (~50-100), model learns basic patterns. Normal.
• Epoch 5-10: Training loss decreases steadily, validation loss follows closely → good sign
• Epoch 20: Training loss ~20-40%, commit loss within 10-20% of training loss

3.2 Benchmarking against Whisper

Comparison with OpenAI Whisper

Whisper is a large speech recognition model developed by OpenAI and trained on massive amounts of data. Testing our custom model against Whisper gives us a clear reference point.

# benchmark_whisper.py
# Comparer notre modèle personnalisé avec OpenAI Whisper

import whisper
from jiwer import wer, cer

# ================================================
# Transcrire avec Whisper
# ================================================
# Charger le modèle Whisper (disponible en : tiny, base, small, medium, large)
whisper_model = whisper.load_model("base")

# Transcrire l'audio
result = whisper_model.transcribe("demos.wav")
whisper_transcription = result["text"]
print(f"Transcription Whisper : {whisper_transcription}")

# ================================================
# Comparer avec notre modèle (après entraînement)
# ================================================
# Transcription de référence (vérité terrain)
reference_text = "we choose to go to the moon not because it is easy but because it is hard"

# Calculer WER/CER pour Whisper
whisper_wer = wer(reference_text, whisper_transcription.lower())
whisper_cer = cer(reference_text, whisper_transcription.lower())

print(f"\n=== Comparaison des performances ===")
print(f"Whisper WER:      {whisper_wer * 100:.1f}%")
print(f"Whisper CER:      {whisper_cer * 100:.1f}%")

# Pour notre modèle entraîné (exemple de valeurs attendues)
our_model_wer = 0.20  # ~20% avec un entraînement limité
our_model_cer = 0.12  # ~12%

print(f"\nNotre modèle WER: {our_model_wer * 100:.1f}%")
print(f"Notre modèle CER: {our_model_cer * 100:.1f}%")

Expected results:

=== Comparaison des performances ===
Whisper WER:       5.2%
Whisper CER:       2.1%

Notre modèle WER: 20.0%
Notre modèle CER: 12.0%

Tradeoff Analysis

A lower error rate does not automatically make Whisper the better choice. Here are the trade-offs to consider:

Key Insight: For real-time applications or edge devices, a model that is 10% less accurate but 5x faster and uses 1/10 of the memory is often the best choice.

3.3 Integrate models into inference pipelines

Test pipeline before training

The inference pipeline allows you to test your model architecture immediately without waiting for training to complete. Even an untrained model can process audio through the full pipeline.

# Ajout à la fin de speechbrain_asr.py

# ================================================
# Test du pipeline avec un modèle non entraîné
# ================================================
def greedy_decode(logits, idx_to_char):
    """
    Décodage greedy : sélectionner le token le plus probable à chaque frame.
    
    Args:
        logits: Tenseur de forme (1, time_steps, vocab_size)
        idx_to_char: Dictionnaire idx → caractère
    
    Returns:
        Texte décodé (sera du charabia sans entraînement)
    """
    # Prendre le token avec le score le plus élevé à chaque frame
    predictions = torch.argmax(logits, dim=-1)  # (1, time_steps)
    predictions = predictions.squeeze(0)         # (time_steps,)
    
    # Décoder CTC : supprimer les blanks et les répétitions
    blank_idx = len(idx_to_char) - 1  # Dernier index = blank
    decoded = []
    prev_token = None
    
    for token_idx in predictions:
        token_idx = token_idx.item()
        if token_idx != blank_idx and token_idx != prev_token:
            if token_idx in idx_to_char:
                decoded.append(idx_to_char[token_idx])
        prev_token = token_idx
    
    return "".join(decoded)

# Initialiser le modèle avec des poids aléatoires
asr_model = SpeechBrainASR(vocab_size=29, sample_rate=16000)

# Mettre en mode évaluation pour un comportement cohérent
asr_model.eval()

print("Modèle initialisé avec des poids aléatoires")
print("En mode eval : comportement cohérent assuré")

# Exécuter le pipeline d'inférence
logits = asr_model.transcribe("demos.wav")
print(f"Forme des logits : {logits.shape}")  # (1, time_steps, 29)

# Décoder (produira du charabia sans entraînement — c'est attendu !)
text = greedy_decode(logits, asr_model.idx_to_char)
print(f"Sortie (non entraîné) : '{text}'")
# Output: 'Sortie (non entraîné) : 'xkqpjmzwlrtyuvb...'  ← charabia, normal !

Why is this gibberish? With random weights, the model doesn’t yet know which sounds correspond to which characters. This is exactly what you need to see — it proves that the architecture is correct and ready for training.

Complete training script: train_asr.py

# train_asr.py
# Script d'entraînement complet pour le modèle ASR

import os
import torch
import torchaudio
from torch.utils.data import Dataset, DataLoader
from speechbrain.lobes.features import Fbank

# ================================================
# ÉTAPE 1 : Télécharger LibriSpeech
# ================================================
# Exécuter dans le terminal :
# wget https://www.openslr.org/resources/12/train-clean-100.tar.gz
# wget https://www.openslr.org/resources/12/dev-clean.tar.gz
# tar -xzf train-clean-100.tar.gz
# tar -xzf dev-clean.tar.gz
# Cela crée un répertoire LibriSpeech avec tous les fichiers audio et transcriptions

# ================================================
# ÉTAPE 2 : Classe Dataset ASR
# ================================================
class ASRDataset(Dataset):
    """
    Dataset PyTorch qui charge les fichiers audio LibriSpeech
    et les convertit au format attendu par le modèle.
    """
    
    def __init__(self, data_list, sample_rate=16000):
        self.data_list = data_list
        self.sample_rate = sample_rate
        
        # Extracteur de features (mêmes paramètres que dans le modèle)
        self.feature_extractor = Fbank(n_mels=80, deltas=False)
        
        # Dictionnaire caractère → index
        # a-z : indices 1-26, ' : 27, espace : 28, blank : 0
        chars = list("abcdefghijklmnopqrstuvwxyz") + ["'", " "]
        self.char_to_idx = {c: i+1 for i, c in enumerate(chars)}
        self.char_to_idx["_"] = 0  # blank CTC
    
    def text_to_indices(self, text):
        """Convertir une chaîne de texte en liste d'indices."""
        text = text.lower()  # Tout en minuscules
        indices = []
        for char in text:
            if char in self.char_to_idx:
                indices.append(self.char_to_idx[char])
            # Caractères inconnus sont ignorés
        return indices
    
    def __len__(self):
        return len(self.data_list)
    
    def __getitem__(self, idx):
        """
        Charger un échantillon audio et sa transcription.
        
        Returns:
            Tuple (features, target_indices)
        """
        audio_path, transcript = self.data_list[idx]
        
        # Charger l'audio
        waveform, sr = torchaudio.load(audio_path)
        
        # Convertir en mono
        if waveform.size(0) > 1:
            waveform = waveform.mean(dim=0, keepdim=True)
        
        # Ré-échantillonner si nécessaire
        if sr != self.sample_rate:
            resampler = torchaudio.transforms.Resample(orig_freq=sr, new_freq=self.sample_rate)
            waveform = resampler(waveform)
        
        # Normaliser
        waveform = waveform / (waveform.abs().max() + 1e-8)
        
        # Extraire les features mel
        features = self.feature_extractor(waveform)  # (1, time, 80)
        features = features.squeeze(0)                # (time, 80)
        
        # Convertir la transcription en indices
        target = torch.tensor(self.text_to_indices(transcript), dtype=torch.long)
        
        return features, target

# ================================================
# ÉTAPE 3 : Charger les données LibriSpeech
# ================================================
def load_librispeech_data(root_dir):
    """
    Parcourir la structure LibriSpeech et retourner les paires (audio, transcription).
    
    Structure LibriSpeech :
    LibriSpeech/
    └── speaker_id/
        └── chapter_id/
            ├── speaker_id-chapter_id-utterance_id.flac
            └── speaker_id-chapter_id.trans.txt  (contient les transcriptions)
    
    Returns:
        Liste de tuples (chemin_audio, transcription)
    """
    data_list = []
    
    for speaker in os.listdir(root_dir):
        speaker_dir = os.path.join(root_dir, speaker)
        if not os.path.isdir(speaker_dir):
            continue
        
        for chapter in os.listdir(speaker_dir):
            chapter_dir = os.path.join(speaker_dir, chapter)
            if not os.path.isdir(chapter_dir):
                continue
            
            # Trouver le fichier de transcription
            trans_file = os.path.join(chapter_dir, f"{speaker}-{chapter}.trans.txt")
            if not os.path.exists(trans_file):
                continue
            
            # Lire les transcriptions
            with open(trans_file, "r") as f:
                for line in f:
                    parts = line.strip().split(" ", 1)
                    if len(parts) == 2:
                        utterance_id, transcript = parts
                        
                        # Construire le chemin du fichier audio
                        audio_file = os.path.join(chapter_dir, f"{utterance_id}.flac")
                        
                        if os.path.exists(audio_file):
                            data_list.append((audio_file, transcript))
    
    return data_list

# ================================================
# ÉTAPE 4 : Fonction de collate pour le batching
# ================================================
def collate_fn(batch):
    """
    Gérer les séquences de longueur variable dans un batch.
    
    Les fichiers audio ont des longueurs différentes, mais les tenseurs
    nécessitent que toutes les dimensions correspondent.
    Cette fonction remboure les séquences avec des zéros.
    
    Returns:
        Tuple (padded_features, padded_targets, input_lengths, target_lengths)
    """
    features, targets = zip(*batch)
    
    # ─── Rembourrer les features ──────────────────────────────────
    input_lengths = torch.tensor([f.size(0) for f in features], dtype=torch.long)
    max_feat_len = input_lengths.max().item()
    feat_dim = features[0].size(1)
    
    padded_features = torch.zeros(len(features), max_feat_len, feat_dim)
    for i, feat in enumerate(features):
        padded_features[i, :feat.size(0), :] = feat
    
    # ─── Rembourrer les cibles (transcriptions) ───────────────────
    target_lengths = torch.tensor([t.size(0) for t in targets], dtype=torch.long)
    max_target_len = target_lengths.max().item()
    
    padded_targets = torch.zeros(len(targets), max_target_len, dtype=torch.long)
    for i, target in enumerate(targets):
        padded_targets[i, :target.size(0)] = target
    
    return padded_features, padded_targets, input_lengths, target_lengths

# ================================================
# ÉTAPE 5 : Script d'entraînement principal
# ================================================
if __name__ == "__main__":
    
    # Charger les données
    print("Chargement des données LibriSpeech...")
    train_data = load_librispeech_data("LibriSpeech/train-clean-100")
    val_data = load_librispeech_data("LibriSpeech/dev-clean")
    
    print(f"Données d'entraînement : {len(train_data)} échantillons")
    print(f"Données de validation  : {len(val_data)} échantillons")
    # Sortie attendue :
    # Données d'entraînement : ~28,000 échantillons
    # Données de validation  : ~2,703 échantillons
    
    # Créer les datasets
    train_dataset = ASRDataset(train_data)
    val_dataset = ASRDataset(val_data)
    
    # Créer les DataLoaders
    train_loader = DataLoader(
        train_dataset,
        batch_size=16,
        shuffle=True,          # Mélanger les données d'entraînement
        collate_fn=collate_fn,
        num_workers=4          # Chargement parallèle des données
    )
    
    val_loader = DataLoader(
        val_dataset,
        batch_size=16,
        shuffle=False,
        collate_fn=collate_fn,
        num_workers=4
    )
    
    # Initialiser le modèle
    asr_model = SpeechBrainASR(vocab_size=29, sample_rate=16000)
    
    # Entraîner le modèle (20 époques)
    # Sur GPU NVIDIA RTX 3090 : ~12-24 heures
    # Sur CPU : plusieurs jours
    print("\nDémarrage de l'entraînement...")
    asr_model.train_model(
        train_loader=train_loader,
        val_loader=val_loader,
        epochs=20,
        learning_rate=0.001,
        grad_clip=5.0
    )
    
    # Sauvegarder les poids entraînés
    torch.save(asr_model.state_dict(), "trained_model.pth")
    print("\nModèle sauvegardé : trained_model.pth")
    
    print("\n=== Comment utiliser le modèle entraîné ===")
    print("""
    # Charger les poids entraînés
    model = SpeechBrainASR(vocab_size=29, sample_rate=16000)
    model.load_state_dict(torch.load('trained_model.pth'))
    model.eval()
    
    # Transcrire de l'audio
    logits = model.transcribe('mon_audio.wav')
    text = greedy_decode(logits, model.idx_to_char)
    print(f'Transcription : {text}')
    """)

Expected training progress

Epoch  1/20 | Train Loss: 89.4321 | Val Loss: 91.2145  ← Poids aléatoires, perte élevée normale
Epoch  2/20 | Train Loss: 74.2100 | Val Loss: 76.8932
Epoch  5/20 | Train Loss: 45.6789 | Val Loss: 48.1234  ← Le modèle apprend les patterns de base
Epoch 10/20 | Train Loss: 28.3456 | Val Loss: 31.4567
Epoch 15/20 | Train Loss: 22.1234 | Val Loss: 25.6789
Epoch 20/20 | Train Loss: 18.7654 | Val Loss: 21.3456  ← Convergence

Diagnostics:

• If train_loss and val_loss stagnate → underfitting (need more capacity)
• If train_loss is low but val_loss remains high → overfitting (need regularization)

4. Summary and conclusion

Congratulations! You have built a complete and functional ASR system. Here is a summary of what you learned and accomplished:

What you learned

Module 1 — Audio data

• Why clean data is crucial for ASR
• The inner workings of waveforms, sample rates and noise
• Hands-on experience with Torchaudio:
• Normalization, denoising, silence removal, resampling
• Converting raw records to spectrograms and MFCCs
• Segmentation and labeling for neural networks

Module 2 — Deep learning ASR models

• Construction of acoustic models based on RNNs, LSTMs
• Mastery of CTC and attention-based decoding
• Converting network output to readable transcripts
• Evaluation with WER and CER
• Comparing results with benchmarks like OpenAI Whisper

Module 3 — Optimization and deployment

• Preparing a fully custom model for training
• Experimentation with architectures and hyperparameters
• Benchmarking tools against Whisper
• Integration of models into inference pipelines usable in production

Final system architecture

Fichier audio (WAV/FLAC)
         │
         ▼
┌─────────────────────────────┐
│     Prétraitement Audio     │
│  • Mono conversion          │
│  • Normalisation amplitude  │
│  • Réduction du bruit       │
│  • Suppression silence VAD  │
│  • Ré-échantillonnage 16kHz │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│    Extraction de Features   │
│  • Fbank (log-Mel 80 bandes)│
│  • MFCCs (13 coefficients)  │
│  • Spectrogrammes           │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│    Modèle Acoustique        │
│  • LayerNorm                │
│  • BiLSTM (256 units, 2L)   │
│  • Linear → vocab (29)      │
└─────────────┬───────────────┘
              │
              ▼
┌─────────────────────────────┐
│         Décodage            │
│  • CTC Beam Search          │
│  • Greedy Decode            │
│  • Attention Decoder        │
└─────────────┬───────────────┘
              │
              ▼
    Transcription textuelle

Recommended next steps

1. Train on LibriSpeech — Download train-clean-100 (6.3 GB, 100 hours) and run train_asr.py
2. Experiment with hyperparameters — Change one parameter at a time and measure the impact
3. Data Augmentation — Add noise, speed and tone disturbances
4. Exploring Transformers — Replace LSTM with a Transformer encoder for better performance on large datasets
5. Deployment — Expose the inference pipeline as a REST API for real applications

“From raw, messy audio clips to fully trained and optimized speech recognition models, we’ve covered an incredible amount of ground together.”

Search Terms

speech · recognition · model · deep · neural · networks · machine · data · science · asr · audio · models · architecture · ctc · pipeline · speechbrain · acoustic · analysis · attention · decoding · fine-tuning · mfccs · mono · optimization