Natural Language Processing for Deep Learning

Documentation language: French (IT terms in English)

Table of Contents

• Classic NLP Methods
• Introduction to Natural Language Processing
• Why language is a challenge for computers
• NLP Applications
• Historical evolution of NLP
• Classic NLP vs Deep Learning
• Text Preprocessing
• Tokenization
• Stop-word Removal
• Normalization
• Stemming
• Lemmatization
• NLTK Demo
• Token Representations
• Vocabulary and token indices
• Word-level Tokenization
• Character-level Tokenization
• Subword Tokenization
• Classic Text Vectorization
• One-hot Encoding
• Bag-of-Words Model
• TF-IDF
• Using Bag-of-Words Templates
• Support ticket demo
• Limitations of Bag-of-Words
• NLP based on Deep Learning
• Static Word Embeddings
• The intuition behind word embeddings
• Word2Vec
• GloVe
• FastText
• Word Vector Semantics
• Geometric relations and vector offsets
• Cosine Similarity
• Gensim Demo
• Use of Embeddings in Neural Networks
• Why embeddings replace one-hot vectors
• The embedding layer
• Demo: SMS spam detector
• Contextual Word Embeddings
• Limitation of static embeddings
• ELMo
• BERT
• Summary: RNN, Transformer, ELMo, BERT
• Contextual Word Embeddings in Action
• GloVe vs BERT comparison
• Results
• Select Best NLP Approach
• Classic NLP vs Deep Learning
• Choose your embedding model
• Practical deployment constraints
• Hybrid approaches
• Final synthesis

1. Classic NLP Methods

Introduction to Natural Language Processing

Why language is a challenge for computers

Every minute, humans send approximately 16 million text messages and 150,000 emails. If you tried to read all the text generated on the Internet in just 60 seconds, it would take you a lifetime. We are drowning in textual data. And if you’re an engineer, you’ve probably been asked to build systems that can extract meaning.

Natural Language Processing, or NLP, is how we can transform human language into something that computers can understand. This is the technology behind voice assistants, chatbots, and systems capable of translating spoken language in real time.

The language presents three fundamental challenges for machines:

1. Ambiguity

Did you know that the word “run” has more definitions than any other word in the English language? According to the Oxford English Dictionary, it has 645 distinct meanings. You can “run a company”, “run a marathon”, “run a computer program”, or even have a “runny nose”. To us the intended meaning is usually obvious, but to a computer it is absolutely not the case. The same word can take on completely different meanings depending on the context in which it appears.

2. The unstructured nature of the text

Unlike the neat rows and columns of a spreadsheet or the fixed grid of pixels that make up an image, the language has no predefined format. A sentence can be 5 words or 50. A document can contain names, dates, emotions, typos or jargon, and can even mix it all up. This makes it difficult to represent language in a systematic, machine-friendly way.

3. Extreme variability

People express the same idea in countless different ways. “One moment, please”, “hold on a sec” and “give me a minute” all convey the same message. But to a computer, they look like unrelated strings of characters. Add in slang, dialects, misspellings, abbreviations and ever-changing usage, and things get even more complex. Deep learning models love consistency, but human language rarely cooperates.

Applications of NLP

Despite all this complexity, NLP has become one of the most powerful technologies used today:

Historical development of NLP

timeline
    title Évolution historique du NLP
    1950s-1980s : Systèmes à base de règles
                : Instructions if-then manuelles
                : Fragiles et non-scalables
    1990s-2000s : Méthodes statistiques
                : Hidden Markov Models
                : Naive Bayes
                : Plus data-driven
    2013 : Word Embeddings (Word2Vec)
         : Capture des relations sémantiques
         : "king" et "queen" sont liés
    2017-2018 : Transformers et BERT
              : Self-attention
              : Contextual embeddings
    Aujourd'hui : Large Language Models
               : ChatGPT, GPT-4
               : State-of-the-art

Rule-based systems (1950s–1980s) NLP was dominated by rule-based systems. Think about if-then statements. A rule might say: If a sentence begins with a verb, it may be a command. These systems worked for narrow tasks, but they were fragile, laborious to build, and could not scale to the complexity of human language.

Statistical methods (late 1980s–1990s) The field has made a transition toward statistical methods. Instead of hand-written rules, algorithms like Hidden Markov Models and Naive Bayes were used to learn patterns directly from the data. This made NLP more data-driven, but these models struggled with long-range dependencies and deep understanding of meaning.

The neural network revolution (2013–present) Around 2013, word embeddings allowed models to capture semantic relationships. Suddenly, computers could understand that words like “king” and “queen” are related. These breakthroughs have paved the way for neural network architectures capable of modeling complex sequences and long-range dependencies.

Large Language Models today Large-scale models have become the backbone of state-of-the-art systems like ChatGPT. These deep neural networks can capture relationships, tone, and nuance in ways that once seemed impossible for machines.

Classic NLP vs Deep Learning

Even with today’s powerful deep learning models, classic NLP methods remain relevant. Deep learning has not replaced them. Traditional techniques like tokenization, preprocessing, and vectorization form the foundation of modern models. And in many real cases, especially with small datasets or tight performance constraints, classical approaches still outperform deep learning.

The main advantage of deep learning is automatic feature learning. Instead of hand-crafted linguistic features, neural networks learn representations directly from data. In practice, the most effective NLP systems combine both approaches: deep learning for rich pattern recognition, and traditional methods for structure, efficiency and control.

Text Preprocessing

Text is one of the messiest data types we work with. If we fed it directly to a machine learning model, we would get unpredictable results. It’s like cooking a meal. You don’t take a potato from the ground and throw it straight into a pan. We wash it, peel it and cut it into pieces so that it cooks properly. In NLP, preprocessing does the same thing for text: it cleans it and prepares it for what comes next.

flowchart LR
    A[Texte brut] --> B[Tokenization]
    B --> C[Stop-word Removal]
    C --> D[Normalisation]
    D --> E[Stemming / Lemmatization]
    E --> F[Tokens propres]

Tokenization

tokenization consists of cutting the text into individual units called tokens. Most often, tokens are words, but depending on the task, they can also be subwords or individual characters.

If you give a computer an entire sentence as a single string, it can’t do anything meaningful with it. But when you divide the text into tokens, the model can begin to analyze relationships, such as the link between the words “API” and “failed”.

At first glance, tokenization seems trivial. Just divide on spaces, right? But natural language is full of special cases:

• How to deal with contractions like “don’t” or “we’ll”?
• What about compound words, or names like “New York”?
• And how to tokenize languages ​​that don’t use spaces (Chinese, Japanese)?

Modern tokenizers handle all these particularities. Their role is to transform messy text into clean, coherent units that models can actually work with.

Stop-word Removal

stop-words are very common words, such as “the”, “a” and “or”, which appear very frequently in texts. In many typical NLP tasks, these words add little value and can introduce noise.

For example, if you are building a search engine for deep learning recipes, the word “the” is not going to help you find useful documents. Removing these words reduces the size of the dataset and helps the model focus on more meaningful terms.

Example of stop-word removal in Python:

from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

sentence = "The API failed to return a valid response"
tokens = sentence.lower().split()
filtered = [t for t in tokens if t not in ENGLISH_STOP_WORDS]
print(filtered)
# ['api', 'failed', 'return', 'valid', 'response']

Important: Removing stop-words is not universal. It may also be necessary to remove dataset-specific stop-words: words that constantly appear in your corpus but do not distinguish one document from another. For example, in medical research articles, words like “study”, “research” or “analysis” can appear in almost every article without providing analytical value.

Caution: Removing stop-words can be harmful if these words carry central meaning for your task. Consider the sentence “The flight was not good.” If we remove the stop-words, we could remove “was” and “not”, leaving us with “flight good.” This completely reverses the feeling. In tasks like sentiment analysis, grammar analysis, or any application where syntax and function words matter, removing stop-words destroys essential information. Never remove stop-words blindly.

Normalization

Normalization is the process of reducing unnecessary variations in text so that similar elements are treated the same way by the model. Common normalization steps include:

• Setting lowercase (lowercasing)
• Removal or standardization of punctuation
• Managing accents and diacritics
• The expansion of contractions (“don’t” → “do not”)
• Standardization of numbers, dates and special symbols

For example, converting “Cats”, “CATS” and “cats” to the same lowercase token helps the model treat them as a single concept instead of three separate ones.

Stemming

Stemming takes a quick and rough approach. It removes common prefixes and suffixes to obtain a simplified root.

Stemming is quick and simple, but not always accurate. It can produce non-words, or group together words that should not be grouped together (e.g. “universe” and “university” → “universe”).

Lemmatization

lemmatization is more linguistically informed. It uses vocabulary knowledge and morphological analysis to return the dictionary form (the lemma) of a word.

Lemmatization takes into account the part of speech and the meaning, which makes it much more precise, but also more computationally expensive.

NLTK Demo

import nltk
from nltk.stem import PorterStemmer, WordNetLemmatizer

# Liste de mots exemple
words = ["running", "ran", "better", "was", "troubled", "universe", "university"]

# Stemming
stemmer = PorterStemmer()
stemmed = [stemmer.stem(w) for w in words]
print("Stemming:", stemmed)
# ['run', 'ran', 'better', 'wa', 'troubl', 'univers', 'univers']

# Lemmatization avec POS (part of speech)
lemmatizer = WordNetLemmatizer()
word_pos_pairs = [
    ("running", "v"),   # verbe
    ("ran", "v"),       # verbe
    ("better", "a"),    # adjectif
    ("was", "v"),       # verbe
    ("troubled", "v"),  # verbe
]
lemmatized = [lemmatizer.lemmatize(w, pos=pos) for w, pos in word_pos_pairs]
print("Lemmatization:", lemmatized)
# ['run', 'run', 'good', 'be', 'trouble']

Important note: Without providing part-of-speech information, the lemmatizer will assume the word is a noun, and the results will likely not be as expected. It is therefore always necessary to provide the POS context for accurate results.

When to use what?

Token Representations

After cleaning and tokenizing our text, we still cannot give it directly to a model. Patterns include numbers, not words. The next step is therefore to represent the tokens numerically using vocabulary and token indices.

Vocabulary and token indices

A vocabulary is simply a mapping of each unique token in your dataset to an integer ID. We essentially build a lookup table that tells the model which number corresponds to which token.

Once this mapping is established, a sentence like “Deep learning is fun” could be represented numerically as:

"Deep" → 142
"learning" → 89
"is" → 23
"fun" → 341
→ [142, 89, 23, 341]

This step is critical because numeric indices form the foundation of all text representation techniques used in machine learning. They convert language into discrete symbols that a model can reference, compare, and process consistently across many examples.

The way we define these tokens has a major impact on the vocabulary size and model performance. This brings us to the three main types of token representations.

Word-level Tokenization

The most intuitive approach: each distinct word becomes a token, and the vocabulary is simply the set of all words present in the training data.

Advantages:

• The model works directly with full words
• Easy to interpret representations

Disadvantages:

Character-level Tokenization

In this approach, each individual character becomes a token in the vocabulary. Instead of a vocabulary of thousands of words, we have a vocabulary of approximately 100 symbols: the 26 letters of the alphabet, numbers, punctuation, spaces, and a few special symbols.

Advantages:

• Completely eliminate out-of-vocabulary issues
• Any text can be represented without predefined vocabulary
• Significantly reduced memory footprint

Disadvantages:

• Semantics typically emerge at the word level or beyond, so the model must learn to combine individual characters into meaningful structures
• Sequences become much longer: “Can’t log into my account” goes from 5 word-level tokens to 25 character-level tokens
• Longer sequences require more computation and make it harder to capture long-range dependencies
• These models often require more complex architectures

Ideal use cases: messy text (spell checking, log analysis, biomedical jargon, creative text generation, domains with rare or unusual vocabulary).

Subword Tokenization

Subword tokenization provides a middle ground between word-level and character-level approaches. Instead of treating whole words as tokens or breaking everything down into characters, it breaks words into smaller, meaningful pieces: prefixes, suffixes, or learned units based on patterns in the data.

Algorithms like Byte-Pair Encoding (BPE), WordPiece, and SentencePiece analyze the training data and learn a vocabulary of subword units that represent the dataset efficiently.

Example:

"unhappiness" → ["un", "##happi", "##ness"]
               ou ["un", "happiness"]

This is powerful. If the model understands that “un” means “not”, it can apply this rule to any new word starting with “un”, whether it has seen it or not.

Advantages:

• Keeps vocabulary size manageable
• Almost eliminates out-of-vocabulary problem
• Naturally captures morphological structure (prefixes, suffixes, radicals)
• Allows you to generalize related words
• Better handle morphologically rich languages

This is why subword tokenization is used in virtually all modern Transformer models, including BERT, GPT, and many others.

flowchart TD
    A[Types de Token Representations] --> B[Word-level]
    A --> C[Character-level]
    A --> D[Subword-level]
    B --> E["Vocabulaire: 100k+ mots\nProblème OOV\nRelations morpho perdues"]
    C --> F["Vocabulaire: ~100 symboles\nAucun OOV\nSéquences très longues"]
    D --> G["Vocabulaire: 30k-50k unités\nQuasi-aucun OOV\nCapture morphologie"]
    G --> H["Utilisé dans BERT, GPT\n(BPE, WordPiece, SentencePiece)"]

Classic Text Vectorization

Even after tokenization and mapping to integer IDs, these numbers remain arbitrary. Mathematically, 541 is greater than 540, but is a dog greater than a cat? Of course not. If we feed these raw integers directly to a model, it could misinterpret numerical relationships that have nothing to do with meaning. We must therefore convert these IDs into digital representations that capture useful information without implying false hierarchies.

One-hot Encoding

The simplest way to represent tokens as vectors is one-hot encoding. Let’s imagine a vocabulary of seven unique words. One-hot encoding represents each word as a vector of length 7, where each position is a 0, except a single 1 at the index assigned to that word.

Example:

Vocabulaire: [chat, chien, oiseau, poisson, lapin, hamster, souris]
             [  0,     1,      2,       3,      4,       5,      6]

"chat"  → [1, 0, 0, 0, 0, 0, 0]
"chien" → [0, 1, 0, 0, 0, 0, 0]
"oiseau"→ [0, 0, 1, 0, 0, 0, 0]

This completely solves the artificial order problem. There is no notion of “cat” being superior to “dog” because each word is represented as a vector with exactly one 1 and the remainder 0s. All words are equidistant and no artificial numerical relationships are introduced.

Major limitation: These vectors are sparse and huge. For a vocabulary of 150,000 words, each vector would have 150,000 dimensions, with only 1 non-zero entry.

Bag-of-Words Model

To move from representing individual words to representing whole sentences or documents, a common approach is to count how many times each word appears and store these counts in a vector of length V (vocabulary size).

Bag-of-Words (BoW) is exactly what the name suggests: you take a document, throw all the words into a bag, and simply count how many times each word appears. Word order is completely ignored.

Example:

Phrase 1 : "dog bites man"
Phrase 2 : "man bites dog"

Vocabulaire : [bites, dog, man]

Phrase 1 → [1, 1, 1]
Phrase 2 → [1, 1, 1]  ← vecteurs identiques !

The basic BoW model treats all words as equally important, but in reality some words carry much more meaning than others.

TF-IDF

TF-IDF (Term Frequency–Inverse Document Frequency) is a weighting scheme designed to emphasize the most important words for a specific document. It has two components:

1. Term Frequency (TF)

Measures how many times a word appears in a single document. Intuition: If a word appears often, it is probably important to the content of this document.

$$TF(t, d) = \frac{\text{number of occurrences of } t \text{ in } d}{\text{total number of words in } d}$$

2. Inverse Document Frequency (IDF)

Measures how rare a word is across the entire corpus. Words that appear everywhere (like stop-words) get a low IDF score because they are not useful for distinguishing documents. Words that only appear in a few documents get a high IDF score because they carry more unique contextual information.

$$IDF(t) = \log\left(\frac{N}{df(t)}\right)$$

where $N$ is the total number of documents and $df(t)$ is the number of documents containing the term $t$.

3. Final TF-IDF score

$$\text{TF-IDF}(t, d) = TF(t, d) \times IDF(t)$$

A word gets a high TF-IDF score if it appears frequently in a document, but rarely across the corpus. This combination signals that the term is particularly characteristic of this document.

Concrete example:

Let’s imagine a set of scientific articles. The word “experiment” appears in almost every article, so it will have a low IDF score even if it appears frequently in a document. But a specialized term like “chromatography” only appears in a few chemistry articles. If it appears multiple times in a paper, its TF-IDF score will be high, marking that article as strongly related to chromatography.

TF-IDF applications:

• Document retrieval and classification
• Text classification
• Thematic clustering
• Recommender systems

Using Bag-of-Words Models

Support Ticket Demo

Let’s say you’re the customer support manager for a large technology company. You have a database filled with thousands of entries. You have structured columns like customer age, gender and ticket priority. These are easy to analyze with standard statistics. But the real goldmine here is in the plain text, specifically the ticket description. This is where the customer actually tells you what’s wrong.

Goal: Extract the most important words from ticket descriptions to instantly understand what they’re about.

Loading dataset:

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

# Chargement du dataset de tickets de support (Kaggle)
df = pd.read_csv('customer_support_tickets.csv')

# Combiner sujet et description pour un meilleur signal
df['combined_text'] = df['Ticket Subject'] + ' ' + df['Ticket Description']

# Afficher les 3 premiers tickets
print(df['combined_text'].head(3))

TF-IDF vectorizer configuration:

vectorizer = TfidfVectorizer(
    lowercase=True,        # Mettre en minuscules
    stop_words='english',  # Supprimer les stop-words anglais
    min_df=5,              # Ignorer les mots dans moins de 5 tickets
    max_df=0.8             # Ignorer les mots dans plus de 80% des tickets
)

# Apprendre le vocabulaire et transformer les textes
tfidf_matrix = vectorizer.fit_transform(df['combined_text'])

# Extraire le vocabulaire appris
vocabulary = vectorizer.vocabulary_
feature_names = vectorizer.get_feature_names_out()

print(f"Taille du vocabulaire : {len(vocabulary)}")
print(f"Forme de la matrice TF-IDF : {tfidf_matrix.shape}")

Inspect the top 10 terms of a ticket:

import numpy as np

# Analyser un ticket spécifique
ticket_idx = 0
ticket_vector = tfidf_matrix[ticket_idx].toarray()[0]

# Trier par score TF-IDF décroissant
top_indices = np.argsort(ticket_vector)[::-1][:10]
top_terms = [(feature_names[i], ticket_vector[i]) for i in top_indices]

print(f"Top 10 termes du ticket {ticket_idx} :")
for term, score in top_terms:
    print(f"  {term}: {score:.4f}")

Specific term comparison:

The word “assist” receives a much lower TF-IDF score because it appears in many tickets and is not very distinctive. “wordpress,” on the other hand, appears in a smaller subset of tickets and carries a more specific meaning, leading to a higher score.

L2 Normalization:

By default, TF-IDF normalizes each document using L2 normalization. This makes scores comparable between tickets of different lengths, which is useful for tasks like similarity searching and clustering.

# Désactiver la normalisation pour voir les poids bruts
vectorizer_raw = TfidfVectorizer(
    lowercase=True,
    stop_words='english',
    min_df=5,
    max_df=0.8,
    norm=None  # Désactiver la normalisation L2
)
tfidf_raw = vectorizer_raw.fit_transform(df['combined_text'])

Bag-of-Words Limitations

1. Sparse vectors (Sparse)

Our document vectors are padded with zeros. Our vocabulary contains thousands of unique words, but any individual ticket uses only a tiny fraction of them. Sparsity creates computational inefficiencies and statistical challenges:

• With so many dimensions, we need massive amounts of data to learn reliable patterns
• It is difficult to measure meaningful similarity between documents. Two tickets on similar issues might have very few words in common, leading to low similarity scores despite being semantically related.

2. Ignoring word order

BoW completely ignores word order. “The customer complained about the product” and “The product complained about the customer” would have identical representations, even if one makes sense and the other does not.

3. Independence of words

The BoW treats each word as completely independent. The words “broken” and “damaged” are synonyms — they mean essentially the same thing in the context of support tickets — but in our model, they are as different from each other as “broken” is from “billing.”

These limitations point to the need for more sophisticated representations, most of which are implemented with the help of neural networks.

2. NLP based on Deep Learning

Static Word Embeddings

The intuition behind word embeddings

Imagine that you are given a list of words related to software development, and your job is to arrange these words along a single dimension that measures how much each relates to low-level programming versus high-level abstraction.

You could place “compiler” and “memory” closer to the low-level side, while “API” and “framework” are at the high-level end. But what about the word “database”? It’s clearly high-level, but it also introduces an entirely different concern: data persistence, which suggests an additional dimension that measures how well a word relates to data storage.

Now consider “concurrency”. This word introduces yet another dimension that measures how well a concept relates to parallel execution. As we continue to add terms, we continually discover new dimensions. We end up having a high-dimensional space where each word is positioned based on many properties at once.

This is exactly the intuition behind word embedding.

Before word embeddings, we relied on discrete symbols and sparse representations like Bag-of-Words and TF-IDF. These methods were simple and effective, but there was no notion of similarity or relationship. The words “function” and “method” were as loosely related as “function” and “filesystem.”

The static word embeddings changed everything. Instead of assigning each word its own isolated dimension, embeddings represent words using shared dimensions. Each dimension captures a latent semantic or syntactic property learned from the data. Words are represented as dense vectors trained on large amounts of text, and these vectors naturally encode relationships.

In this vector space, words like “linux” and “windows”, “bug” and “error”, or “REST” and “API” are found close together. It was a major breakthrough. It allowed models to reason about meaning geometrically, using math rather than hand-crafted rules.

“A person is known by the company they keep.” This same idea applies remarkably well to language. The meaning of a word can be inferred from the context in which it appears. Words that frequently co-occur near the same surrounding words tend to have related meanings.

Word2Vec

Word2Vec was introduced by Google researchers in 2013. It learns word embeddings by training a simple neural network on a predictive task. It comes in two main variants:

Variant 1: CBOW (Continuous Bag-of-Words)

The model is trained to predict a target word based on the words around it.

Exemple : "The server completed the _____ successfully"
Mots contexte → [The, server, completed, the, successfully]
Mot cible     → "request"

Surrounding words are provided as input to the network, usually as one-hot vectors. They pass through a hidden layer, and the output layer produces a probability distribution over the possible words. The hidden layer of this network becomes the word embeddings.

When training on millions of sentences, words that appear in similar contexts learn similar embeddings.

Variant 2: Skip-gram

Skip-gram works in the opposite direction: instead of predicting a target word from its context, skip-gram predicts surrounding context words given a target word.

Mot cible → "request"
Prédire   → [The, server, completed, the, successfully]

The architecture is similar, but the input is now the target word and the output is a probability distribution over the surrounding context words.

flowchart LR
    subgraph CBOW
        C1[mots contexte] --> H1[couche cachée\nembeddings]
        H1 --> T1[mot cible]
    end
    subgraph SkipGram["Skip-gram"]
        T2[mot cible] --> H2[couche cachée\nembeddings]
        H2 --> C2[mots contexte]
    end

GloVe

GloVe (Global Vectors for Word Representation) takes a more global approach. Rather than predicting local context in a sliding window, it starts by constructing a co-occurrence matrix that counts how many times pairs of words appear together across the entire corpus.

The model then learns embeddings by factoring the matrix in a way that preserves meaningful ratios between word co-occurrences. This allows GloVe to capture both local context and broader global statistical patterns across the dataset.

FastText

FastText (developed by Facebook AI Research) addresses a problem that Word2Vec and GloVe do not address: rare words, spelling mistakes, or words never seen during training.

Word2Vec and GloVe treat each word as a single atomic unit. FastText solves this problem by incorporating subword level information. Each word is represented as a collection of character n-grams.

Example with n-grams of size 3:

"where" → <wh, whe, her, ere, re>
  + marqueurs de frontière : <where>

Embedding is calculated by summing character n-gram embeddings. Under the hood, FastText uses the same training architecture as Word2Vec (CBOW or skip-gram).

Once trained, these three methods assign a unique fixed vector to each word. This is what makes static embeddings effective and widely useful.

Word Vector Semantics

Geometric relations and vector offsets

One of the fascinating things about word embeddings is that relationships between words translate into geometric patterns in vector space.

Example of vector arithmetic:

$$\vec{linux} - \vec{kernel} + \vec{gui} \approx \vec{unix}$$

Why does this arithmetic work? Because word embeddings learn to encode coherent relationships like directions in vector space. These directions are called vector offsets.

Linux is often discussed in contexts that emphasize its low-level, kernel-centric architecture, while Unix systems are more commonly discussed in environments that include user-oriented tools, shells, and graphical interfaces. Through large amounts of training text, the model sees these repeating patterns and learns to represent this contrast as a shared dimension in the embedding space.

Other relationships captured by embeddings:

The relationships that word embeddings capture reflect how words are used in the language, not strict technical definitions.

Cosine Similarity

The most commonly used metric to measure how similar two word vectors are is cosine similarity.

Cosine similarity measures the angle between two vectors. Its values ​​range from -1 to +1:

$$\cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{|\vec{a}| \cdot |\vec{b}|}$$

Why angle rather than Euclidean distance? In word embeddings, we care much more about direction than magnitude. Two words can have very different vector lengths, but still represent similar meanings if their vectors point in the same direction.

Gensim Demo

import gensim.downloader as api
import numpy as np
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt

# Charger le modèle Word2Vec pré-entraîné (Google News, 300 dimensions)
# (déjà chargé hors-écran pour gagner du temps)
model = api.load('word2vec-google-news-300')

# --- Trouver des mots similaires ---
words_to_check = ['linux', 'kernel', 'API', 'database', 'bug']

for word in words_to_check:
    if word in model:
        similar = model.most_similar(word, topn=5)
        print(f"\nMots similaires à '{word}':")
        for w, score in similar:
            print(f"  {w}: {score:.4f}")

Measure similarity explicitly:

# Comparer des paires de mots
pairs = [
    ('linux', 'unix'),         # liés
    ('bug', 'error'),          # liés
    ('server', 'database'),    # modérément liés
    ('linux', 'banana'),       # non liés
]

for w1, w2 in pairs:
    if w1 in model and w2 in model:
        sim = model.similarity(w1, w2)
        print(f"similarity('{w1}', '{w2}') = {sim:.4f}")

Visualization with PCA:

# Liste de mots du domaine software/systèmes
words = ['request', 'response', 'client', 'server', 'API',
         'schema', 'database', 'HTTP', 'latency', 'cache']

# Récupérer les vecteurs (300D)
vectors = np.array([model[w] for w in words if w in model])

# Réduire à 2D avec PCA
pca = PCA(n_components=2)
reduced = pca.fit_transform(vectors)

# Tracer le scatter plot
plt.figure(figsize=(10, 8))
for i, word in enumerate(words):
    plt.scatter(reduced[i, 0], reduced[i, 1])
    plt.annotate(word, (reduced[i, 0], reduced[i, 1]))
plt.title('Word2Vec embeddings - projection PCA 2D')
plt.show()
# Les concepts liés forment des clusters naturels

Vector arithmetic (vector offsets):

# linux - kernel + gui ≈ ?
result = model.most_similar(
    positive=['linux', 'gui'],
    negative=['kernel'],
    topn=3
)
print("linux - kernel + gui ≈", result)
# → [('unix', ...)]

# apple_juice - apple + lemon ≈ ?
result2 = model.most_similar(
    positive=['apple_juice', 'lemon'],
    negative=['apple'],
    topn=3
)
print("apple_juice - apple + lemon ≈", result2)
# → [('lemon_juice', ...)]

Use of Embeddings in Neural Networks

Why embeddings replace one-hot vectors

Word embeddings aren’t just nice tricks for finding analogies and similar words. They are also the fundamental building blocks of modern deep learning models for NLP.

When training a neural network to work with text, the first challenge is converting the text into a format it can process.

The problem with one-hot vectors:

• For a vocabulary of 150,000 words, each one-hot vector has 150,000 dimensions
• Huge waste of space: only one dimension is active (= 1), the rest is 0
• No information on meaning: vectors carry no semantic information
• Extremely inefficient learning

The solution: embeddings

Embeddings solve this problem by mapping each word to a dense vector of smaller dimension.

Advantage of generalization:

Suppose you are training a neural network to sort customer support emails. If the network learns that the word “crash” is associated with high-severity problems, it can often make reasonable predictions when it encounters words like “failure” or “error.” With one-hot vectors this sort of thing is not possible. Each word is independent, and the model must relearn the patterns separately for each token.

The embedding layer

At the high level, an embedding layer is simply a lookup table. It takes a token index as input and returns the corresponding embedding vector. These vectors are then transmitted to the rest of the network.

During training, embeddings can either:

• Stay fixed if they are pre-trained (transfer learning)
• Be fine-tuned with the rest of the model

In both cases, the embedding layer provides a meaningful digital representation on which the network can rely.

flowchart LR
    A["Token IDs\n[142, 89, 23, 341]"] --> B["Embedding Layer\n(lookup table)"]
    B --> C["Vecteurs denses\n[[0.2, -0.1, 0.5, ...],\n [0.8, 0.3, -0.2, ...],\n ...]"]
    C --> D["Réseau de neurones\n(couches cachées)"]
    D --> E["Prédiction"]

Demo: SMS spam detector

Objective: Build a spam detector for SMS messages using pre-trained GloVe embeddings.

Dataset: UCI Machine Learning Repository (SMS messages labeled spam/ham)

Loading and preparation:

import pandas as pd
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
from sklearn.model_selection import train_test_split
import re

# Charger le dataset
df = pd.read_csv('spam_sms.csv', encoding='latin-1')[['v1', 'v2']]
df.columns = ['label', 'message']

# Convertir les labels en numérique
df['label_num'] = (df['label'] == 'spam').astype(int)
# 1 = spam, 0 = ham (non-spam)

Simple tokenization:

def tokenize(text):
    """Lowercase et extraction de mots alphabétiques via regex"""
    return re.findall(r'[a-z]+', text.lower())

df['tokens'] = df['message'].apply(tokenize)

Loading GloVe embeddings (100 dimensions):

# Embeddings GloVe entraînés sur Wikipedia + Gigaword
glove = {}
with open('glove.6B.100d.txt', encoding='utf-8') as f:
    for line in f:
        parts = line.split()
        word = parts[0]
        vector = np.array(parts[1:], dtype=np.float32)
        glove[word] = vector

print(f"Vocabulaire GloVe : {len(glove)} mots")
print(f"Dimensions : {len(list(glove.values())[0])}")

Conversion of messages into vectors (averaging):

def message_to_vector(tokens, glove_dict, dim=100):
    """
    Lookup des vecteurs GloVe pour chaque token et 
    calcul de la moyenne → embedding au niveau du document
    """
    vectors = [glove_dict[t] for t in tokens if t in glove_dict]
    if vectors:
        return np.mean(vectors, axis=0)
    else:
        return np.zeros(dim)

X = np.array([message_to_vector(t, glove) for t in df['tokens']])
y = df['label_num'].values

Split train/stratified test (to handle class imbalance):

# Déséquilibre de classes : beaucoup plus de ham que de spam
print(df['label'].value_counts())

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,
    stratify=y   # Préserver la distribution des classes originale
)

PyTorch Dataset and DataLoaders:

class SMSDataset(Dataset):
    def __init__(self, X, y):
        self.X = torch.FloatTensor(X)
        self.y = torch.FloatTensor(y)
    
    def __len__(self):
        return len(self.y)
    
    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

train_dataset = SMSDataset(X_train, y_train)
test_dataset = SMSDataset(X_test, y_test)

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)

Neural network architecture:

class SpamDetector(nn.Module):
    def __init__(self, input_dim=100):
        super().__init__()
        self.network = nn.Sequential(
            nn.Linear(input_dim, 64),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(64, 32),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(32, 1)   # Logit unique → BCEWithLogitsLoss
        )
    
    def forward(self, x):
        return self.network(x).squeeze(1)

model = SpamDetector(input_dim=100)

Class imbalance management:

# Calcul du poids pour la classe positive (spam)
n_neg = (y_train == 0).sum()  # nombre de ham
n_pos = (y_train == 1).sum()  # nombre de spam
pos_weight = torch.tensor([n_neg / n_pos], dtype=torch.float)

# Pénaliser davantage les erreurs sur les spams
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Drive loop:

EPOCHS = 20

for epoch in range(EPOCHS):
    model.train()
    train_loss = 0.0
    correct = 0
    
    for X_batch, y_batch in train_loader:
        optimizer.zero_grad()
        outputs = model(X_batch)
        loss = criterion(outputs, y_batch)
        loss.backward()
        optimizer.step()
        
        train_loss += loss.item()
        preds = (torch.sigmoid(outputs) > 0.5).float()
        correct += (preds == y_batch).sum().item()
    
    train_acc = correct / len(train_dataset)
    
    # Évaluation sur le test set
    model.eval()
    with torch.no_grad():
        test_correct = 0
        for X_batch, y_batch in test_loader:
            outputs = model(X_batch)
            preds = (torch.sigmoid(outputs) > 0.5).float()
            test_correct += (preds == y_batch).sum().item()
    
    test_acc = test_correct / len(test_dataset)
    
    if (epoch + 1) % 5 == 0:
        print(f"Epoch {epoch+1}/{EPOCHS} | "
              f"Train Loss: {train_loss/len(train_loader):.4f} | "
              f"Train Acc: {train_acc:.4f} | "
              f"Test Acc: {test_acc:.4f}")

Final evaluation:

from sklearn.metrics import classification_report

model.eval()
all_preds = []
all_labels = []

with torch.no_grad():
    for X_batch, y_batch in test_loader:
        outputs = model(X_batch)
        preds = (torch.sigmoid(outputs) > 0.5).float()
        all_preds.extend(preds.numpy())
        all_labels.extend(y_batch.numpy())

print(classification_report(
    all_labels, all_preds,
    target_names=['ham', 'spam']
))

Contextual Word Embeddings

Limitation of static embeddings

Static word embeddings like Word2Vec, GloVe, and FastText allow us to build applications that understand semantic relationships between words, but they have a major limitation.

Consider the word “drive” in these two sentences:

1. “The computer’s drive was replaced.” (hard disk)
2. “She decided to drive to work.” (drive)

The word is the same, but their meanings are completely different. Static word embeddings are, as their name suggests, static. Every word has exactly one fixed vector, regardless of how that word is used in a sentence. For a word like “drive”, the model tries to compress multiple meanings into a single representation. The result is a garbled average that does not fully represent any of the cases.

ELMo

ELMo (Embeddings from Language Models) was developed by researchers at the Allen Institute for AI in 2018. ELMo is built on a deep bidirectional language model that uses stacked LSTMs.

How ​​does a language model work?

A language model learns to predict the next word in a sequence. In the case of the input sequence “The server received the HTTP ___”, the model can predict that the next word is “error”.

ELMo architecture:

flowchart LR
    subgraph Forward["LM gauche → droite"]
        A1[The] --> B1[server] --> C1[received] --> D1[the] --> E1[request]
    end
    subgraph Backward["LM droite → gauche"]
        E2[request] --> D2[the] --> C2[received] --> B2[server] --> A2[The]
    end
    Forward --> F[États cachés LSTM\n= Embeddings contextuels]
    Backward --> F

ELMo trains two language models simultaneously:

1. A model processes text from left to right and predicts each word based on everything that comes before
2. The other processes the text from right to left and predicts each word based on what comes after

This configuration allows the model to capture context from both directions.

ELMo layers:

ELMo typically uses 2 or 3 stacked bidirectional layers:

• Lower layers capture syntactic information (grammar, local structure)
• Higher layers capture more abstract semantic meaning and long-range dependencies

When using ELMo in practice, you pass your input text through the network and extract these internal representations. You can choose embeddings from a specific layer or combine multiple layers depending on your task.

BERT

Researchers at Google have found an even more powerful way to encode text: BERT (Bidirectional Encoder Representations from Transformers).

BERT replaces ELMo’s LSTM-based approach with Transformers.

LSTMs vs Transformers:

Self-attention:

Self-attention allows the model to look at all the words in a sentence at once and learn how much each word should “pay attention to” every other word.

Example:

Phrase : "The server received the request, but it crashed."
                                               ↑
                     "it" se réfère à "server" (pas "request")

Self-attention helps the model understand that “it” refers to “server”, not “request”. Each token embedding is influenced by its relationships to all other words in the sentence.

BERT architecture:

BERT stacks several Transformer layers on top of each other (typically 12 or 24). Each layer refines the token representations by paying attention to the patterns captured in previous layers. This results in deeply contextualized embeddings formed by the complete input sequence.

BERT training goals:

1. Masked Language Modeling (MLM)

During training, random words in a sentence are masked and the model learns to predict them using both left and right context. This allows BERT to construct fully bidirectional representations.

Input  : "The server received the [MASK] successfully"
Output : → "request" (prédit à partir du contexte des deux côtés)

2. Next Sentence Prediction (NSP)

The model learns whether one sentence logically follows another. This helps BERT understand relationships between sentences, which is useful for tasks like question answering and natural language inference.

Phrase A : "The API is down."
Phrase B : "Users cannot log in."     → IsNextSentence = True
Phrase B : "The weather is sunny."    → IsNextSentence = False

Summary: RNN, Transform, ELMo, BERT

flowchart TD
    A[Représentation de texte] --> B[RNNs]
    A --> C[Transformers]
    B --> D[Embeddings traités\nun token à la fois]
    D --> E[État caché porte\nle contexte séquentiellement]
    C --> F[Self-attention\nParallèle sur toute la séquence]
    F --> G[Relations modélisées\nentre chaque mot]
    G --> H[Plus rapide, meilleur\npour longue portée]
    
    I[ELMo] --> J[Embeddings calculés d'abord\npuis passés au modèle de tâche]
    K[BERT] --> L[Le Transformer fait à la fois\nembedding et modélisation de séquence]
    L --> M[Ajouter une couche de sortie\nspécifique à la tâche au-dessus de BERT]

Contextual Word Embeddings in Action

GloVe vs BERT comparison

Dataset: Banking77 (Hugging Face)

• Customer messages tagged with one of 77 categories (e.g.: “card not working”, “change pin”)
• Short messages, often ambiguous and highly context dependent
• Exactly the type of scenario where the choice of embedding makes a real difference

Data preparation:

from datasets import load_dataset
import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split

# Charger le dataset Banking77
dataset = load_dataset("banking77")
train_df = pd.DataFrame(dataset['train'])
test_df = pd.DataFrame(dataset['test'])

# Encoder les labels
encoder = LabelEncoder()
train_df['label_enc'] = encoder.fit_transform(train_df['label'])
test_df['label_enc'] = encoder.transform(test_df['label'])

# Créer un set de validation
train_data, val_data = train_test_split(
    train_df,
    test_size=0.1,
    random_state=42,
    stratify=train_df['label_enc']
)

n_classes = len(encoder.classes_)
print(f"Nombre de classes : {n_classes}")  # 77

Model 1 — Classification with GloVe (static embeddings):

import torch
import torch.nn as nn
import numpy as np

# Chargement des embeddings GloVe pré-entraînés
def load_glove(path, dim=100):
    glove = {}
    with open(path, encoding='utf-8') as f:
        for line in f:
            parts = line.split()
            glove[parts[0]] = np.array(parts[1:], dtype=np.float32)
    return glove

glove = load_glove('glove.6B.100d.txt')

def text_to_glove_vector(text, glove_dict, dim=100):
    """Moyenne des vecteurs GloVe pour chaque token"""
    tokens = text.lower().split()
    vectors = [glove_dict[t] for t in tokens if t in glove_dict]
    return np.mean(vectors, axis=0) if vectors else np.zeros(dim)

# Convertir les messages en vecteurs
X_train = np.array([text_to_glove_vector(t, glove) for t in train_data['text']])
X_val   = np.array([text_to_glove_vector(t, glove) for t in val_data['text']])
X_test  = np.array([text_to_glove_vector(t, glove) for t in test_df['text']])

# Architecture du réseau
class GloVeClassifier(nn.Module):
    def __init__(self, input_dim=100, n_classes=77):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Dropout(0.3),
            nn.Linear(128, n_classes)
        )
    
    def forward(self, x):
        return self.net(x)

Model 2 — BERT Fine-tuning:

from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

# Charger le tokenizer BERT pré-entraîné
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')

def tokenize_function(examples):
    """Tokenisation BERT : subword tokens, padding, conversion en IDs"""
    return tokenizer(
        examples['text'],
        padding='max_length',
        truncation=True,
        max_length=128
    )

# Tokeniser les datasets
train_encoded = train_data.apply(lambda x: tokenize_function({'text': x['text']}), axis=1)
# (version simplifiée - en pratique on utilise dataset.map())

# Dataset PyTorch pour BERT
class BERTDataset(torch.utils.data.Dataset):
    def __init__(self, texts, labels, tokenizer, max_len=128):
        self.encodings = tokenizer(
            texts.tolist(),
            padding='max_length',
            truncation=True,
            max_length=max_len,
            return_tensors='pt'
        )
        self.labels = torch.tensor(labels.values)
    
    def __len__(self):
        return len(self.labels)
    
    def __getitem__(self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item['labels'] = self.labels[idx]
        return item

# Fine-tuning avec Hugging Face AutoModelForSequenceClassification
bert_model = AutoModelForSequenceClassification.from_pretrained(
    'bert-base-uncased',
    num_labels=77
)

training_args = TrainingArguments(
    output_dir='./bert_banking77',
    num_train_epochs=3,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    evaluation_strategy='epoch',
    save_strategy='epoch',
    load_best_model_at_end=True,
    metric_for_best_model='eval_accuracy',
)

trainer = Trainer(
    model=bert_model,
    args=training_args,
    train_dataset=BERTDataset(train_data['text'], train_data['label_enc'], tokenizer),
    eval_dataset=BERTDataset(val_data['text'], val_data['label_enc'], tokenizer),
)

trainer.train()

Results

This is a dramatic improvement: +16 points of accuracy.

Why does BERT outperform GloVe?

Let’s look at some example predictions:

In both cases, the GloVe model predicts a related but incorrect intent, while BERT correctly identifies the intent. This highlights the key limitation of static embeddings: they struggle when the intent depends on subtle wording or context. BERT, on the other hand, weighs each word in relation to every other word, allowing it to disambiguate meaning much more effectively.

Select the Best NLP Approach

Classic NLP vs Deep Learning

At this point in the training, you have seen a wide range of techniques for working with text. The natural question is: which approach should you actually use?

It’s tempting to reach for the biggest, most powerful model every time. But in the world of production machine learning, more complex doesn’t always mean better. This is compromise.

Factor 1: Dataset size

Classic methods like TF-IDF coupled with Random Forest or Naive Bayes work incredibly well with small to medium datasets. They are quick to train, easy to interpret, and very effective for early prototyping. Significant results can often be obtained in hours rather than days of fine-tuning a deep learning model.

If you only have a few hundred training examples, a deep learning model will likely overfit and perform poorly.

Factor 2: Explainability

Deep learning models are treated as black boxes. If your application operates in a regulated environment or requires clear explanations for its predictions (finance, medicine, law), classic models may be a better choice.

Factor 3: Task complexity

Rule of thumb: Use the simplest approach that meets the performance requirements.

Choose your embedding model

If you decide to go with deep learning, your next decision is which embedding model to use.

flowchart TD
    A[Choisir un embedding] --> B{Priorité principale ?}
    B --> |Rapidité + similarité générale| C[Static Embeddings\nWord2Vec / GloVe]
    B --> |Mots rares / fautes orthographe| D[FastText]
    B --> |Précision maximale + contexte| E[Contextual Embeddings\nBERT / variantes]
    E --> F{Contraintes ?}
    F --> |Mémoire limitée| G["DistilBERT\n(66% taille BERT, ~97% performance)"]
    F --> |Latence critique| H[Quantization\nou cache d'embeddings]
    F --> |GPU disponible| I[BERT base / large\nRoBERTa, ALBERT...]

Static Embeddings (Word2Vec, GloVe):

• Excellent when general semantic similarity is needed but fast calculation is important
• Work well for classification or clustering, especially when latency matters
• FastText is particularly useful when the data includes misspellings, rare words, or domain-specific terminology

Contextual Embeddings (BERT and variants):

• When accuracy and contextual understanding are top priority
• These models generate dynamic representations and produce state-of-the-art results
• But there is a cost: significantly larger and with higher computational requirements

Practical deployment constraints

In practice, the choice of embedding comes down to three constraints:

1. Model size

Large models like BERT can be hundreds of megabytes in size. In environments with memory constraints, smaller embeddings, distilled models like DistilBERT, or static embeddings with lightweight networks are often better choices.

2. Latency

Real-time systems need millisecond responses. Static embeddings are extremely fast, while contextual templates are slower.

Policies if latency is critical:

• Use smaller transformers
• Apply quantization (reduce weight precision from float32 to int8)
• Hide embeddings for frequently seen entries

3. Availability of calculation and memory

Contextual models often require GPUs and lots of RAM, while static embeddings and smaller models can run comfortably on CPUs.

Your deployment environment should always guide your choice.

Hybrid approaches

Selecting an NLP approach should not be an all-or-nothing decision. Many real systems, in fact, combine classic NLP and deep learning.

Hybrid architecture example:

flowchart LR
    A[Requête utilisateur] --> B[TF-IDF\nfiltrage rapide]
    B --> C{Top N\ndocuments\ncandidat}
    C --> D[BERT\ncompréhension sémantique profonde]
    D --> E[Résultat final\nranking]

For example, you could use TF-IDF to rapidly reduce a large set of candidate documents, then apply a transformer-based model to the remaining most relevant ones for deeper semantic understanding. This approach provides a very practical balance between speed and accuracy.

Final summary

Throughout this training, you’ve seen how NLP transforms plain text into something machines can understand, and how to make informed choices along the way.

flowchart TD
    A["Texte brut"] --> B["Preprocessing\n(tokenization, stop-words, normalisation)"]
    B --> C["Token Representations\n(word, subword, character)"]
    C --> D["Vectorisation classique\n(One-hot, BoW, TF-IDF)"]
    D --> E["Static Embeddings\n(Word2Vec, GloVe, FastText)"]
    E --> F["Contextual Embeddings\n(ELMo, BERT)"]
    F --> G["Application NLP\n(classification, NER, QA, etc.)"]

Learning path summary:

1. Text Preprocessing: Clean, tokenize and normalize raw text
2. Token Representations: Understanding word-level, character-level and subword
3. Classic Vectorization: One-hot encoding, Bag-of-Words, TF-IDF
4. Static Word Embeddings: Word2Vec, GloVe, FastText — dense representations that encode meaning
5. Contextual Embeddings: ELMo and BERT — dynamic representations that capture context
6. Approach selection: Know the trade-offs and choose intelligently

You now have the tools to evaluate an NLP problem and decide when a simple, interpretable solution is sufficient, and when the problem truly requires deep learning and contextual understanding.

End of training — Natural Language Processing for Deep Learning

Search Terms

natural · language · processing · deep · neural · networks · machine · data · science · embeddings · nlp · word · classic · tokenization · bag-of-words · bert · contextual · elmo · embedding · glove · model · one-hot · static · text