Python 3 Regex Playbook

Table of Contents

1. Course Overview
2. Working with regular expressions in Python

• 2.1 Module Introduction
• 2.2 Introduction to regex
• 2.3 Special characters
• 2.4 The repetition
• 2.5 Python functions of the re module
• 2.6 The match object
• 2.7 Les flags

3. Find and match patterns in files

• 3.1 Module Introduction
• 3.2 Find a word in a file
• 3.3 Find filenames
• 3.4 Specific patterns in files
• 3.5 Search product codes

4. Validate user inputs and data

• 4.1 Module Introduction
• 4.2 Validate user inputs
• 4.3 Validate digital data
• 4.4 Validate email addresses
• 4.5 Sanitize input against code injection

5. Analyze log files

• 5.1 Module Introduction
• 5.2 Automate the processing of logs (IP addresses)
• 5.3 Identify logging levels
• 5.4 Search for a specific error code
• 5.5 Process logs in real time
• 5.6 Cut log file entries

6. Replace, transform and clean data

• 6.1 Module Introduction
• 6.2 Clean user entries (phone numbers)
• 6.3 Correct spelling mistakes
• 6.4 Apply correct formatting (lookahead / lookbehind)
• 6.5 Standardize case and remove punctuation
• 6.6 Clean data in a CSV file
• 6.7 Handle Unicode and special characters
• 6.8 Hide sensitive data

1. Course Overview

Hello everyone. My name is Maaike van Putten, I am a software developer and trainer at Bright Boost. Welcome to my course, Python 3 Regex Playbook.

Did you know that you can speed up your word processing tasks and automate various manual word processing actions with Python regex? In this course, we will give you the knowledge and skills to tackle exactly this challenge.

Topics covered in this course

Here are the main topics we will cover:

• Working with regex expressions in Python
• Find and match patterns in files
• Validate data
• Analyze log files
• Replace, transform and clean data

What you will learn

By the end of this course, you will have the skills to apply Python regex to automate all kinds of word processing tasks, saving you valuable time.

Prerequisites

Before starting this course, you should be familiar with the basics of Python programming, such as variables, loops, and functions.

Next steps after this course

From here, you should feel comfortable diving into advanced word processing and data automation with courses on:

• Extracting and manipulating data using Python libraries
• Text analysis and natural language processing (NLP)
• Web scraping

2. Working with regular expressions in Python

2.1 Module introduction

In this module, we will review the basics of regex and how to use it in Python. We will start by defining what regex is and what it is used for. Next, we’ll move on to how to interpret regex, and we’ll discuss the basic blocks. We’ll also review the special characters that are part of the regex, and we’ll cover repetition and grouping.

We will do this in a very superficial way. If you want to learn regex in depth, you can take one of the excellent regex courses in the Pluralsight library. Whenever I do something that hasn’t already been covered, I’ll make sure to briefly explain the code and the regex pattern.

Finally, we will look at the different ways to use regex in Python and how to work with the match object.

Note: If you already know how to read and write regex and are familiar with the Python functions you can use with regex, you might want to skip this module and look at the later Practical Use Case modules.

2.2 Introduction to regex

In this section, we’ll start by defining what regex is and when to use it. Next, we’ll look at how to read regex basics.

What is regex?

regex is short for regular expression. It is a sequence of characters that represents a text pattern. This pattern can then be used for all kinds of purposes, but it all comes down to one thing: checking if a string matches the pattern.

Regex use cases

Here are examples of situations where you want to check if a string matches a pattern:

1. Search in a text file: Find matches of a certain word, perhaps even with different ways of writing it.
2. Validation of a string: When we ask for an email address, we know that it must have a certain format. We can check this format with the regex. The same goes for phone numbers, zip codes, and many other patterns.
3. Word replacement: For example, when trying to rename all variables in a code file from an old term to a new term.

Read regex

Reading regex can be quite intimidating. At first glance, it looks a bit like a cat has stepped on the keyboard. That’s why it’s important to know how to break it down.

First major point: the regex is read from left to right.

Let’s start with a simple example: matching a word in a sentence, then move on to more complex patterns.

• The simplest regex consists of a single literal character, for example, e. It matches the first e in a string.
• The string we will use here is Python Regex Playbook on Pluralsight. It contains an e, which means there is a match.
• It actually contains multiple e, but it will only match the first one when searching the pattern.

2.3 Special characters

In this section we are going to talk about the different special characters that we can use in the regex.

Overview

We will examine:

1. wildcards (wildcards)
2. anchors to indicate that a certain pattern should occur at the beginning or end of a string
3. The escape character (\) which can give a non-special character a special meaning, and make special characters non-special (literal)
4. Character classes and grouping

Often we want to perform more complex tasks than simply searching for a literal string. Fortunately, we have special characters. Unlike literals, they do not simply correspond to their literal correspondence, but have a special meaning. We can use these special characters to specify a pattern instead of just literal text, which means that different strings can match the pattern.

The wildcard (.)

period (.) matches any character except a newline. It’s called wildcard because it matches almost everything.

Example:

Anchors

• ^ (caret): corresponds to the start of the string.
• ^sight will not match Pluralsight because “sight” is not at the start.
• $ (dollar): matches the end of the string.
• sight$ will match Pluralsight because “sight” is at the end.
• ^sight$ means that the whole string must be exactly “sight”.

The escape character (\)

The backslash \ is used for:

• Give a special meaning to a non-special character: \d for a number, \w for a word character, etc.
• Treat a special character as a literal: \. matches a literal period (not the wildcard).

Example:

\.txt   # Correspond au texte littéral ".txt"
.txt    # Correspond à n'importe quel caractère suivi de "txt" (wildcard)

Character classes ([...])

A character class matches one of the characters specified in square brackets.

The alternation (|)

The pipe | means or: cat|chien corresponds to “cat” or “dog”.

Groups ((...))

Parentheses allow you to group parts of a pattern. A group can be capturing (returns the corresponding content) or non-capturing ((?:...)).

2.4 Repetition

Repeat is a very important regex feature that allows you to repeat a certain pattern snippet.

Repeat quantifiers

Detailed examples

Example 1 — .+\.txt (file with .txt extension):

• . matches any character
• + means one or more times
• \. matches a literal period
• txt matches the letters “txt”

Thus, this pattern will correspond to: data.txt, a.txt, hello.txt, 1234.txt.

Example 2 — .*\.txt (zero or more characters before .txt):

• * means zero or more times, so the pattern will also match .txt alone (without characters before the period).

Example 3 — colored (British AND American spelling):

• The ? makes the u optional.
• Matches colour (British spelling) and color (US spelling).

Example 4 — {n,m}:

\d{2,4}   # Entre 2 et 4 chiffres
[a-z]{3}  # Exactement 3 lettres minuscules

2.5 Python functions of the re module

Python comes with re library which helps in working with regex. There are also third-party libraries, but re is often enough. Here are the functions we will cover: search, match, fullmatch, findall, sub, subn, split, escape, and compile.

re.search(pattern, string, flags=0)

The search function scans the provided string looking for the first match. It accepts three parameters:

• pattern (required): the regex pattern
• string (required): the string to search for
• flags (optional): behavior flags

It returns None if no match is found, and the match object containing the match if a match is found.

import re

text = "This is a sample text."
pattern = r"sample"
match = re.search(pattern, text)

if match:
    print("Found a match!")
else:
    print("No match found.")

Output: Found a match!

If we search for smple (instead of sample), the output will be: No match found.

re.match(pattern, string, flags=0)

The match function checks if the start of the string matches the pattern. Even in multiline mode, it only checks the beginning of the string. It accepts the same parameters as search.

Key difference: re.match only checks the beginning of the string, while re.search searches the entire string.

import re

# re.match — vérifie seulement le début
result = re.match(r"Python", "Python is great")
print(result)  # Objet match

result2 = re.match(r"great", "Python is great")
print(result2)  # None — "great" n'est pas au début

re.fullmatch(pattern, string, flags=0)

The fullmatch function checks if the entire string matches the pattern. Useful for validating that the entire string follows the expected format.

import re

result = re.fullmatch(r"\d{5}", "12345")
print(result)  # Objet match — toute la chaîne est 5 chiffres

result2 = re.fullmatch(r"\d{5}", "12345abc")
print(result2)  # None — la chaîne contient autre chose

re.findall(pattern, string, flags=0)

The findall function returns a list of all matches (not just the first).

import re

text = "My emails are user@example.com and admin@test.org"
pattern = r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
matches = re.findall(pattern, text)
print(matches)
# ['user@example.com', 'admin@test.org']

re.sub(pattern, repl, string, count=0, flags=0)

The sub (substitute) function replaces all occurrences of the pattern in the string with the replacement string repl. It returns the new channel.

import re

text = "Hello World"
result = re.sub(r"World", "Python", text)
print(result)  # "Hello Python"

re.subn(pattern, repl, string, count=0, flags=0)

Same as sub, but returns a tuple (new_string, number_of_replacements).

import re

text = "cat and cat and cat"
result, count = re.subn(r"cat", "dog", text)
print(result)  # "dog and dog and dog"
print(count)   # 3

re.split(pattern, string, maxsplit=0, flags=0)

The split function splits the string at each place where the pattern matches, and returns a list.

import re

text = "one   two\tthree\nfour"
parts = re.split(r"\s+", text)
print(parts)  # ['one', 'two', 'three', 'four']

re.escape(pattern)

The escape function escapes all non-alphanumeric characters in the string, which is useful when you want to treat a string as a literal in a regex pattern.

import re

pattern = re.escape("prix: 10.99$")
print(pattern)  # "prix\:\ 10\.99\$"

re.compile(pattern, flags=0)

The compile function compiles a regex pattern into a reusable pattern object. This is most effective when the same pattern is used several times.

import re

pattern = re.compile(r"\d{4}-\d{2}-\d{2}")  # Format de date YYYY-MM-DD

dates = ["2023-01-15", "2024-06-20", "pas une date"]
for item in dates:
    if pattern.search(item):
        print(f"Date valide : {item}")
    else:
        print(f"Pas une date : {item}")

2.6 The match object

We’ve seen that some Python functions return a match object, but we haven’t really paid attention to it yet.

Description

The match object is a special object that Python returns when it finds a match. It contains several information about the correspondence, such as:

• The substring that was found
• The position of the substring in the original string
• Captured groups, if any

Properties of the match object

• The match object always has a boolean value of True.
• If no match is found, the function returns None.

Important methods of the match object

Example — matching a phone number

import re

# Pattern pour un numéro de téléphone au format (XXX) XXX-XXXX
phone_pattern = r"\((\d{3})\)\s(\d{3})-(\d{4})"

text = "Mon numéro est (514) 555-1234."

match = re.search(phone_pattern, text)

if match:
    print("Chaîne originale :", match.string)
    print("Correspondance complète :", match.group())
    print("Indicatif régional :", match.group(1))
    print("Préfixe :", match.group(2))
    print("Numéro :", match.group(3))
    print("Tous les groupes :", match.groups())
    print("Position de début :", match.start())
    print("Position de fin :", match.end())
    print("Span :", match.span())

Expected output:

Chaîne originale : Mon numéro est (514) 555-1234.
Correspondance complète : (514) 555-1234
Indicatif régional : 514
Préfixe : 555
Numéro : 1234
Tous les groupes : ('514', '555', '1234')
Position de début : 15
Position de fin : 29
Span : (15, 29)

2.7 Flags

There is an important topic in regex that we haven’t covered yet: regex flags. We use them when we want to change the way the regex is compared.

re.I / re.IGNORECASE — Case insensitivity

This flag makes the pattern case insensitive: uppercase and lowercase letters are treated as equivalent.

import re

pattern = r"hello"
text = "Hello, world!"

# Sans flag — pas de correspondance (H majuscule vs h minuscule)
match_no_flag = re.search(pattern, text)
print(match_no_flag)  # None

# Avec IGNORECASE — correspondance
match_with_flag = re.search(pattern, text, re.IGNORECASE)
print(match_with_flag)  # Objet match

re.S / re.DOTALL — Dot matches line breaks

This flag makes the character . match any character, including newlines, whereas normally it matches everything except newlines.

import re

pattern = r"hello.*world"
text = "hello\nworld"

# Sans flag — pas de correspondance (\n n'est pas capturé par .)
match_no_flag = re.search(pattern, text)
print(match_no_flag)  # None

# Avec DOTALL — correspondance
match_with_flag = re.search(pattern, text, re.DOTALL)
print(match_with_flag)  # Objet match
print(match_with_flag.group())  # "hello\nworld"

re.M / re.MULTILINE — ^ and $ on each line

This flag causes the characters ^ and $ to match the start and end of each line in a multiline string, rather than just the start and end of the entire string.

import re

# Pattern pour un ou plusieurs chiffres en début de ligne
pattern = r"^\d+"
text = "123 première ligne\n456 deuxième ligne\ntroisième ligne sans chiffre"

# Sans MULTILINE — trouve seulement le début de toute la chaîne
matches_no_flag = re.findall(pattern, text)
print(matches_no_flag)  # ['123']

# Avec MULTILINE — trouve le début de chaque ligne
matches_with_flag = re.findall(pattern, text, re.MULTILINE)
print(matches_with_flag)  # ['123', '456']

3. Find and match patterns in files

3.1 Module Introduction

Welcome to this new module. We will find and match patterns in files using Python Regex.

This is actually a very common use case: searching for patterns in a file with Python. We can, for example, use it to identify the type of text document we are working with, but we can also search text files for email addresses and telephone numbers, or automatically retrieve the order number from an invoice.

In this module, we will introduce the concept of searching and matching patterns in files using regular expressions in Python. We will see examples of how we can use regular expressions to perform the following different tasks:

1. Search for a single word in a file
2. Search for file names in a file
3. Search for multiple patterns in a file
4. Search for product codes in one or more files

3.2 Find a word in a file

Let’s start with a relatively simple example: searching a file for the first match using Regex and Python. This means that we will stop at the first occurrence.

Steps

1. Open a file
2. Use the re.search function
3. Show result

Code

import re

# Étape 1 : Ouvrir le fichier
with open("sample1.txt", "r") as file:
    # Étape 2 : Boucler sur toutes les lignes du fichier
    for line in file:
        # Rechercher le mot "sample" dans la ligne
        match = re.search(r"sample", line)
        if match:
            # Étape 3 : Afficher la ligne et arrêter
            print("found a match in line:", line.strip())
            break

Output: found a match in line: This is a simple sample text

Why read line by line?

The reason for reading the file line by line is that this approach is more memory efficient than reading the entire file into memory at once. Additionally, reading line by line allows us to process the file incrementally, rather than waiting for the entire file to be read before performing any analysis or processing.

When using the open function to read a file, the entire contents of the file are not loaded into memory immediately. Instead, the file is read one line at a time, allowing large files to be processed without running out of memory. This is especially important when working with large files that might not fit in memory. Therefore, reading all at once is even a little dangerous.

3.3 Find file names

One can use regex to search through a file and find all document names for common file extensions.

Steps

1. Create a list of common extensions
2. Read a file
3. Build a regex pattern from the extension list
4. Find all matches with re.findall

Code

import re

# Étape 1 : Créer la liste des extensions de fichiers
extensions = ["pdf", "doc", "docx", "txt", "xlsx", "csv", "pptx"]

# Étape 2 : Ouvrir et lire le fichier en une seule fois
with open("sample2.txt", "r") as file:
    text = file.read()

# Étape 3 : Construire le pattern regex
# \b       — word boundary (début du mot)
# \w+      — un ou plusieurs caractères de mot
# \.       — un point littéral
# (?:...)  — groupe non capturant
# |.join   — extensions séparées par | (OU)
# \b       — word boundary (fin du mot)
pattern = r"\b\w+\.(?:" + "|".join(extensions) + r")\b"

# Étape 4 : Trouver toutes les correspondances
matches = re.findall(pattern, text)
print("Fichiers trouvés :", matches)

Explanation of the pattern:

• r in front of the string: raw string, avoids having to escape backslashes
• \b: word boundary — ensures you are at the start of a word
• \w+: matches one or more word characters (letters, numbers, _)
• \.: matches a literal point (not the wildcard)
• (?:pdf|doc|docx|txt|...): non-capturing group with alternating extensions

Non-capturing group ((?:...)): Indicated by ?: at the start of the group. Text inside the group will not be captured and will not be returned as a separate group in the result. This is useful when you want to apply a quantizer or alternation to part of the pattern without capturing that part.

3.4 Specific patterns in files

Regular expressions can be used to search for specific patterns in a file. Here are types of patterns commonly checked in regex:

• Phone Numbers
• Email addresses
• Postal Codes
• Order numbers
• Hexadecimal color codes
• And many more…

Steps

1. Define regex patterns
2. Search in a file
3. Show results with match objects

Code

import re

# Étape 1 : Définir les patterns regex
# (versions simplifiées pour l'exemple)
phone_pattern    = r"\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}"
email_pattern    = r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}"
zip_pattern      = r"\b\d{5}(?:-\d{4})?\b"
order_pattern    = r"\bORD-\d{6}\b"
hex_pattern      = r"#[0-9A-Fa-f]{6}"  # Simplifié — 6 caractères hex

# Étape 2 : Ouvrir et lire le fichier
with open("sample3.txt", "r") as file:
    content = file.read()

# Étape 3 : Rechercher chaque pattern
phone_match  = re.search(phone_pattern, content)
email_match  = re.search(email_pattern, content)
zip_match    = re.search(zip_pattern, content)
order_match  = re.search(order_pattern, content)
hex_match    = re.search(hex_pattern, content)

# Affichage des résultats
if phone_match:
    print("Phone number found:", phone_match.group())
if email_match:
    print("Email address found:", email_match.group())
if zip_match:
    print("Zip code found:", zip_match.group())
if order_match:
    print("Order number found:", order_match.group())
if hex_match:
    print("Hex color code found:", hex_match.group())

Note: Some of the regex patterns are a bit simplified — for example, hex_pattern can only specify a six-character hex. In a real application, we might need a more robust pattern.

3.5 Search product codes

Sometimes we want to scan files to find product codes. For example, when you have received a lot of invoices with product codes and you need to collect all these codes for later analysis of product performance.

Steps (single file)

1. Read file
2. Create a pattern and use the findall function
3. Show matches

Explanation of the pattern

The regex pattern uses the \b metacharacter to match word boundaries. This ensures that the pattern only matches the entire product code and not a part of it.

The pipe character | is used to match either of two product code formats:

Code

import re

with open("product_list.txt", "r") as file:
    content = file.read()

# Pattern pour les codes produit
product_pattern = r"\b(?:[A-Z]{2}\d{2}-[A-Z]{2}\d{2}|\d{3}-\d{3})\b"

matches = re.findall(product_pattern, content)
print("Codes produit trouvés :", matches)

Extension — Processing multiple files

You can also put the product code search code in a function and loop through an array of files, storing the product codes in a different file.

import re

def find_product_codes(filename, output_file):
    """Trouve tous les codes produit dans un fichier et les écrit dans un fichier de sortie."""
    product_pattern = r"\b(?:[A-Z]{2}\d{2}-[A-Z]{2}\d{2}|\d{3}-\d{3})\b"
    
    with open(filename, "r") as file:
        content = file.read()
    
    matches = re.findall(product_pattern, content)
    
    with open(output_file, "a") as out:
        for code in matches:
            out.write(f"{code}\n")
    
    return matches

# Liste des fichiers à traiter
files_to_process = [
    "invoice_001.txt",
    "invoice_002.txt",
    "invoice_003.txt",
]

output_file = "all_product_codes.txt"

for filename in files_to_process:
    codes = find_product_codes(filename, output_file)
    print(f"Codes trouvés dans {filename} : {codes}")

4. Validate user inputs and data

4.1 Module Introduction

In this module, we will use Python Regex to validate user inputs and data. We’ll look at different examples of user input:

• Validate a username
• Validate digital inputs
• Validate email addresses and report invalid entries
• Use Python Regex to avoid code injection

4.2 Validate user inputs

Often, when depending on external inputs, validation must take place. The regex can be used for this validation. We have already seen that regex is a powerful tool for pattern matching and text searching, and it can also be used to ensure data integrity and prevent code injection.

Validating a username

Here are the rules for the username:

• Must start with a lowercase or uppercase letter
• Can then contain 0 or more alphanumeric characters or underscores

Steps

1. Create a function that contains the pattern and uses the fullmatch function
2. Request user input
3. Call the function and check if the input is valid

Code

import re

def validate_username(username):
    """Valide un nom d'utilisateur.
    
    Règles :
    - Doit commencer par une lettre (majuscule ou minuscule)
    - Peut contenir des caractères alphanumériques et des underscores
    """
    # ^ commence par une lettre
    # [A-Za-z]    — première lettre
    # [A-Za-z0-9_]* — zéro ou plusieurs alphanumériques/underscores
    pattern = r"^[A-Za-z][A-Za-z0-9_]*$"
    match = re.fullmatch(pattern, username)
    return match

user_input = input("Entrez un nom d'utilisateur : ")

if validate_username(user_input):
    print("Valid username!")
else:
    print("Invalid username.")

Tests:

• Mary123 → Valid username! (starts with a letter, followed by alphanumeric characters)
• 123Mary → Invalid username. (starts with a number, which is invalid)

4.3 Validate digital data

Sometimes we want to ensure that the user only enters numbers or that the data is strictly numeric. For example, when we need data that will be used for analysis and calculations.

Simple version — numbers only

import re

def validate_strictly_numeric(value):
    """Valide que la valeur ne contient que des chiffres."""
    pattern = r"^\d+$"
    return bool(re.search(pattern, value))

user_input = input("Entrez un nombre : ")

if validate_strictly_numeric(user_input):
    print("Valid number!")
else:
    print("Invalid number.")

Limitation: This simple version does not allow negative numbers or decimals. So -6 and 3.14 would be considered invalid.

Improved version — negative numbers and decimals

import re

def validate_numeric_extended(value):
    """Valide un nombre, y compris les négatifs et les décimaux."""
    # -?       — signe négatif optionnel (0 ou 1 fois)
    # \d+      — un ou plusieurs chiffres
    # (?:\.\d+)? — point décimal suivi de chiffres (optionnel)
    # $        — fin de la chaîne
    pattern = r"^-?\d+(?:\.\d+)?$"
    return bool(re.fullmatch(pattern, value))

# Tests
tests = ["42", "-6", "3.14", "-2.5", "abc", "12.34.56"]
for test in tests:
    result = validate_numeric_extended(test)
    print(f"'{test}' — {'Valide' if result else 'Invalide'}")

Output:

'42' — Valide
'-6' — Valide
'3.14' — Valide
'-2.5' — Valide
'abc' — Invalide
'12.34.56' — Invalide

4.4 Validate email addresses

Let’s talk about an essential topic in data validation and processing: working with email addresses. Email addresses are a common part of many data sets and ensuring they are valid is crucial to maintaining data quality and integrity. For example, when working with mailing lists.

Structure of a valid email address

A valid email address consists of:

1. One or more characters: a-z (lowercase or uppercase), 0-9, periods, underscores, percentages, plus and minus signs
2. Followed by an @ symbol
3. Followed by the same characters A-Z, 0-9, periods, or dashes
4. Tracking a point
5. Followed by letters, at least two

Steps

1. Define a regex pattern for a valid email address
2. Read the list of email addresses from a file
3. Filter invalid email addresses
4. Save invalid email addresses in a separate file

Code

import re

# Étape 1 : Pattern pour les adresses email (version simplifiée)
email_pattern = r"^[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}$"

# Étape 2 : Lire le fichier contenant les adresses email
with open("email_list.txt", "r") as file:
    emails = file.readlines()

# Étape 3 : Filtrer les adresses email invalides
invalid_emails = [
    email.strip()
    for email in emails
    if not re.match(email_pattern, email.strip())
]

# Étape 4 : Sauvegarder les adresses invalides dans un fichier
with open("invalid_emails.txt", "w") as out_file:
    for email in invalid_emails:
        out_file.write(email + "\n")

print(f"Adresses email invalides trouvées : {len(invalid_emails)}")
print("Sauvegardées dans 'invalid_emails.txt'")

4.5 Sanitize inputs against code injection

Code injection via inputs is a common problem. This happens when someone, usually with bad intentions, tries to execute code on your server through input fields. Sanitizing this user input is crucial to making this impossible.

Regex can be used to avoid code injection by spotting dangerous instructions and sanitizing them before further processing.

Steps

1. Create a function to sanitize user input
2. Create a function to check for dangerous characters
3. Request user input, sanitize it and check it for code injection

Code

import re

def sanitize_input(user_input):
    """Assainit l'entrée utilisateur en supprimant les caractères potentiellement dangereux."""
    # Supprimer les caractères dangereux : ; ' " < > etc.
    sanitized = re.sub(r"[;'\"\<\>\(\)\{\}\[\]\\]", "", user_input)
    return sanitized

def validate_string_no_code_injection(user_input):
    """Vérifie si l'entrée contient des caractères potentiellement dangereux."""
    # Chercher des caractères utilisés dans les injections SQL, JavaScript, etc.
    dangerous_pattern = r"[;'\"\<\>\(\)\{\}\[\]\\]"
    match = re.search(dangerous_pattern, user_input)
    return match is None  # True si aucun caractère dangereux trouvé

# Étapes 3 : Demander, assainir et valider
user_input = input("Entrez votre texte : ")

# Assainir l'entrée
sanitized = sanitize_input(user_input)
print(f"Entrée assainie : {sanitized}")

# Vérifier l'entrée assainie
if validate_string_no_code_injection(sanitized):
    print("Valid input.")
else:
    print("Invalid input: Detected potentially harmful characters.")

Tests:

• Input: hi there → Sanitation: hi there → Valid input.
• Input: hello; DROP TABLE users → Sanitation: hello DROP TABLE users → Valid input. (the ; has been removed)
• If we disable sanitation and test the same input directly → Invalid input: Detected potentially harmful characters.

Safety lesson: We don’t just check — we also sanitize. Sanitation is the first line of defense, validation is the second. Together, they protect against code injection attempts.

5. Analyze log files

5.1 Module Introduction

Log files play a crucial role in understanding application behavior and troubleshooting problems. However, as they grow in size and complexity, it becomes increasingly difficult to extract meaningful information from them. This is where the Python regex comes into play: it allows us to parse and analyze log files with ease.

In this module, we will explore various techniques to get the most out of your log files. We will see examples of the following operations:

1. Analyze log files for data types and learn how to extract specific information, such as IP addresses
2. Check for error codes in log files
3. Filter a live log to detect and resolve issues in real time
4. Cut log entries — extract essential data such as date, log entry type, and message

5.2 Automate log processing (IP addresses)

Log files become really large and complex, and often we need to automate their processing. We can use regex to parse log files in different ways.

Steps

1. Define a function that takes the log line as input
2. Define a regex pattern for IP addresses
3. Use the findall function to find all IP addresses
4. Open the log file and process it line by line

Pattern for IP addresses

A typical IP address pattern (IPv4): a three-digit one, followed by a period, repeated three times, then another three-digit one.

Attention — capturing vs. non-capturing groups: If we write (\d{1,3}\.){3}, the result will only contain the last captured group. To obtain the full IP address, you must use a non-capturing group (?:\d{1,3}\.){3}.

Code

import re

def extract_ip_addresses(log_line):
    """Extrait et affiche toutes les adresses IP d'une ligne de log."""
    # Pattern avec groupe non capturant (?:...) pour éviter la capture partielle
    ip_pattern = r"(?:\d{1,3}\.){3}\d{1,3}"
    
    ip_addresses = re.findall(ip_pattern, log_line)
    
    if ip_addresses:
        for ip in ip_addresses:
            print(f"Adresse IP trouvée : {ip}")

# Ouvrir et traiter le fichier de log ligne par ligne
with open("server.log", "r") as log_file:
    for line in log_file:
        extract_ip_addresses(line)

Explanation of the non-capturing group: We use these non-capturing groups whenever we want to apply some sort of regex constraint, like a quantifier in this example, to a part of the pattern without actually capturing this part.

5.3 Identify logging levels

Logging is done at different levels:

The regex can be used to identify the type of log entries, such as INFO, WARNING, and ERROR.

Steps

1. Define a function that takes a log line as input
2. In this function, define a pattern that corresponds to the type of log entry
3. Use the search function to find the type of log entry in the line
4. Extract log entry type from match
5. Call the function for each line of the log file

Code

import re

def identify_log_entry_type(log_line):
    """Identifie et affiche le type d'entrée de log (INFO, WARNING, ERROR)."""
    # Pattern qui correspond à l'un des trois niveaux
    pattern = r"\b(INFO|WARNING|ERROR)\b"
    
    match = re.search(pattern, log_line)
    
    if match:
        log_type = match.group(1)
        print(f"Log entry type: {log_type}")

# Exemple de fichier de log :
# 2023-01-15 10:30:01 INFO Application started
# 2023-01-15 10:30:02 INFO Database connected
# 2023-01-15 10:31:00 ERROR Connection timeout
# 2023-01-15 10:31:05 INFO Retrying...
# 2023-01-15 10:32:00 WARNING Disk space low

with open("app.log", "r") as log_file:
    for line in log_file:
        identify_log_entry_type(line)

We can then use this as a starting point to treat each log level differently — for example, notify for ERROR, ignore INFO, etc.

5.4 Search for a specific error code

Sometimes you want to search the log for a specific error code.

Steps

1. Define a function that takes a log line and an error code as input
2. Define a pattern to find this error
3. Use the search function to find the error in the log line
4. Call the function for each row and display the corresponding entries

Code

import re

def search_for_error_code(log_line, error_code):
    """Recherche un code d'erreur spécifique dans une ligne de log."""
    # Pattern pour trouver les lignes contenant "ERROR"
    pattern = r"\bERROR\b"
    
    match = re.search(pattern, log_line)
    
    if match and error_code in log_line:
        print(f"Error code found: {error_code}")
        print(f"  Ligne : {log_line.strip()}")

# Ouvrir et traiter le fichier de log
with open("app.log", "r") as log_file:
    for line in log_file:
        search_for_error_code(line, "404")

Output (example):

Error code found: 404
  Ligne : 2023-01-15 10:35:00 ERROR 404 Page not found: /missing-page
Error code found: 404
  Ligne : 2023-01-15 10:40:00 ERROR 404 Resource not found: /api/data

Note: If there are three errors in the file but only two with code 404, only the two 404 entries will be displayed. If we need another type of log level or another code, we can easily adjust the pattern.

5.5 Process logs in real time

You can also filter live logs with the regex. This can be particularly useful for detecting issues like errors in real time.

Steps

1. Define a function that generates a log message with a random log level (to simulate a live log)
2. Define a function that takes a log line as input and filters for the error pattern
3. In this function, use the search function to match the pattern in the log line
4. In an infinite loop, simulate a live logging system and filter with this function

Code

import re
import time
import random

def get_live_logging_data():
    """Simule l'obtention de données de logging en direct avec des niveaux aléatoires."""
    levels = ["INFO", "WARNING", "ERROR"]
    level = random.choice(levels)
    return f"2023-01-15 10:{random.randint(0,59):02d}:{random.randint(0,59):02d} {level} Some log message"

def filter_errors(log_line):
    """Filtre les lignes de log pour trouver les erreurs."""
    error_pattern = r"\bERROR\b"
    match = re.search(error_pattern, log_line)
    if match:
        print(f"Error found: {log_line.strip()}")

# Simuler un système de logging en direct (boucle infinie)
print("Surveillance du log en direct... (Ctrl+C pour arrêter)")
while True:
    line = get_live_logging_data()
    filter_errors(line)
    time.sleep(1)  # Pause d'une seconde entre chaque message

Note: In this example, we use a very simple pattern (ERROR). In a real application, we would have a much more complex live log and more sophisticated patterns.

5.6 Splitting entries from a log file

Sometimes, when processing a text log, we must structure it and transform it into objects. You can use the split function of the regex to split the entries in a log file and extract specific information such as the date, type of log entry, and message.

Steps

1. Create a split_log_entry function that takes the log line as input
2. Define a pattern that matches one or more whitespace characters
3. Use the split function to divide the log line into parts according to the pattern
4. Use the parts to extract the necessary components

Example of log file

2023-01-15 INFO User logged in successfully
2023-01-15 ERROR Database connection failed: timeout after 30s
2023-01-15 WARNING High memory usage detected: 85%

Code

import re

def split_log_entry(log_line):
    """Divise une entrée de log et extrait la date, le type et le message."""
    # Pattern pour un ou plusieurs caractères d'espacement
    whitespace_pattern = r"\s+"
    
    # Diviser la ligne sur les espaces
    parts = re.split(whitespace_pattern, log_line.strip())
    
    if len(parts) >= 3:
        date = parts[0]
        log_type = parts[1]
        message = " ".join(parts[2:])  # Le reste est le message
        
        print(f"Date: {date}")
        print(f"Type: {log_type}")
        print(f"Message: {message}")
        print()

# Traiter chaque ligne du fichier de log
with open("app.log", "r") as log_file:
    for line in log_file:
        if line.strip():  # Ignorer les lignes vides
            split_log_entry(line)

Output:

Date: 2023-01-15
Type: INFO
Message: User logged in successfully

Date: 2023-01-15
Type: ERROR
Message: Database connection failed: timeout after 30s

Date: 2023-01-15
Type: WARNING
Message: High memory usage detected: 85%

Note: This approach can be easily adapted to split log entries with different formats — for example, with different patterns separating them: hyphens, colons, or any other character or pattern one might look for. You can customize as needed to divide the log entry into several parts.

6. Replace, transform and clean data

6.1 Module Introduction

The data is great and gives us a lot of information. In order to obtain this information, one must be able to perform accurate data analysis. To be able to perform this accurate analysis, our data must be clean and consistent. Without this, they cannot be used for accurate data analysis and effective decision-making.

Fortunately, we can use our Python regex skills to replace, transform and clean up data. This is exactly what we are going to do in this module.

In this module, we will explore various techniques and use cases for replacing, transforming and cleansing data. We will see examples of:

1. Cleaning and standardizing user input
2. Find and correct spelling mistakes in a document
3. Reformat dates, prices and numbers (with the use of lookahead and lookbehind assertions)
4. Text preprocessing techniques: remove punctuation, standardize text case, tokenize words
5. Format and clean data in CSV file
6. Handle Unicode and special characters in data
7. Hide sensitive information

6.2 Clean user entries (phone numbers)

In many applications, user input is a crucial source of data. However, user input can often be inconsistent or even contain errors like different date formats or misspelled words.

For example, imagine a web form where users enter their phone numbers. Some users might use hyphens, others spaces or parentheses. Our goal is to standardize the format so that it is consistent across all entries.

Steps

1. Get user input
2. Define a regular expression pattern for the desired format
3. Validate the input against the pattern by calling the function with the input
4. Show result

Code

import re

def clean_phone_number(phone_number):
    """Nettoie et reformate un numéro de téléphone.
    
    Format de sortie : (XXX) XXX-XXXX
    Retourne None si le numéro est invalide.
    """
    # Supprimer tous les caractères non numériques
    digits_only = re.sub(r"\D", "", phone_number)
    
    # Valider la longueur (10 chiffres pour un numéro américain)
    if len(digits_only) != 10:
        return None
    
    # Reformater au format (XXX) XXX-XXXX
    # \1 = groupe 1 (3 premiers chiffres)
    # \2 = groupe 2 (3 chiffres suivants)
    # \3 = groupe 3 (4 derniers chiffres)
    formatted = re.sub(
        r"(\d{3})(\d{3})(\d{4})",
        r"(\1) \2-\3",
        digits_only
    )
    return formatted

# Demander l'entrée utilisateur
user_input = input("Entrez un numéro de téléphone : ")
result = clean_phone_number(user_input)

if result:
    print(f"Cleaned phone number: {result}")
else:
    print("Invalid phone number.")

Tests:

• Entry: 514/555/1234 → Cleaned phone number: (514) 555-1234
• Entry: (514) 555-1234 → Cleaned phone number: (514) 555-1234
• Entry: 514.555.1234 → Cleaned phone number: (514) 555-1234
• Entry: 123 → Invalid phone number.

6.3 Correct spelling mistakes

Spelling mistakes can make documents look unprofessional and even difficult to understand. For example, in a business report, spelling errors can create confusion and lower the overall quality of the document. This is why we will replace misspelled words in a document using Python regex.

Steps

1. Define a replace_misspellings function
2. Create a dictionary for common spelling mistakes and their corrections (you could also read this dictionary from a file)
3. Open the document and send the content to the function to replace the mistakes
4. Write the correct content to a file (new or same file)

Code

import re

def replace_misspellings(text, corrections):
    """Remplace les fautes d'orthographe dans le texte.
    
    Args:
        text: Le texte pouvant contenir des fautes d'orthographe.
        corrections: Un dictionnaire {mot_incorrect: correction}.
    
    Returns:
        Le texte avec les fautes corrigées.
    """
    for misspelling, correction in corrections.items():
        # Compiler le pattern avec word boundaries et insensibilité à la casse
        pattern = re.compile(r"\b" + re.escape(misspelling) + r"\b", re.IGNORECASE)
        # Remplacer chaque faute par la correction
        text = pattern.sub(correction, text)
    return text

# Dictionnaire des fautes communes et leurs corrections
corrections = {
    "recieve":      "receive",
    "accomodate":   "accommodate",
    "seperate":     "separate",
    "definately":   "definitely",
    "occured":      "occurred",
    "untill":       "until",
}

# Lire le document
with open("document.txt", "r", encoding="utf-8") as file:
    content = file.read()

# Appliquer les corrections
corrected_content = replace_misspellings(content, corrections)

# Écrire le document corrigé
with open("corrected_document.txt", "w", encoding="utf-8") as file:
    file.write(corrected_content)

print("Document corrigé sauvegardé dans 'corrected_document.txt'.")

Hint: We use re.escape(misspelling) to ensure that special characters in the misspelled word are treated as literals. We compile the pattern to be more efficient when we have to apply several replacements.

6.4 Apply correct formatting (lookahead / lookbehind)

Proper formatting is important for readability and consistency, especially when dealing with data that needs to be analyzed automatically. For example, in a financial report, it is essential to have consistent date and price formats to avoid confusion.

To do this, we will use two regex concepts that we have not yet discussed: the lookbehind and the lookahead.

Lookahead and lookbehind assertions

These are special assertions in regular expressions that allow you to match a pattern only if it is preceded or followed by another pattern, without including that other pattern in the match itself.

Illustrative example

Suppose we have a text with a mixture of numbers, uppercase and lowercase letters. We want to find all the uppercase letters that are preceded by a lowercase letter AND followed by a number.

• Pattern for uppercase letters: [A-Z]
• Lookbehind to be preceded by a lowercase: (?<=[a-z])
• Lookahead to be followed by a number: (?=\d)
• Complete pattern: (?<=[a-z])[A-Z](?=\d)

import re

text = "abC3 deF7 GHI jkL9 mNO"
pattern = r"(?<=[a-z])[A-Z](?=\d)"

matches = re.findall(pattern, text)
print(matches)  # ['C', 'F', 'L']
# C est précédé par 'b' (minuscule) et suivi par '3' (chiffre)
# F est précédé par 'e' (minuscule) et suivi par '7' (chiffre)
# L est précédé par 'k' (minuscule) et suivi par '9' (chiffre)

Reformat dates

import re

def reformat_date(date_string):
    """Convertit une date du format MM/DD/YYYY vers YYYY-MM-DD."""
    pattern = r"(\d{2})/(\d{2})/(\d{4})"
    # \3 = année (groupe 3), \1 = mois (groupe 1), \2 = jour (groupe 2)
    reformatted = re.sub(pattern, r"\3-\1-\2", date_string)
    return reformatted

# Test
print(reformat_date("01/15/2023"))  # 2023-01-15
print(reformat_date("12/31/2024"))  # 2024-12-31

Reformat prices with lookahead

import re

def reformat_prices(text):
    """Reformate les prix en ajoutant des espaces appropriés."""
    # Lookahead : trouver des chiffres suivis de "," ou "." et deux autres chiffres
    # Pattern simplifié pour illustration
    pattern = r"\$(\d+(?:\.\d{2})?)"
    result = re.sub(pattern, r"\1 $", text)
    return result

6.5 Standardize case and remove punctuation

Sometimes, for tasks like text analysis, natural language processing (NLP), and data cleaning, it is important to know how to remove punctuation, standardize text case, and tokenize words.

We need it, for example, when we want to analyze customer reviews. You may want to remove all punctuation to make the text easier to compare and analyze.

Steps

1. Remove punctuation
2. Convert all to lowercase
3. Tokenize words by splitting them on spaces
4. Process the result

Code

import re

# Texte d'exemple
text = "Hello, World! This is a sample text. Let's tokenize it."

# Étape 1 : Supprimer la ponctuation
# [^\w\s] — correspond à tout ce qui n'est pas un caractère de mot ou un espace
text_without_punctuation = re.sub(r"[^\w\s]", "", text)
print("Sans ponctuation :", text_without_punctuation)
# "Hello World This is a sample text Lets tokenize it"

# Étape 2 : Convertir en minuscules
# Utiliser la méthode Python intégrée .lower() — plus approprié que regex ici
text_lowercase = text_without_punctuation.lower()
print("En minuscules :", text_lowercase)
# "hello world this is a sample text lets tokenize it"

# Étape 3 : Tokeniser les mots en divisant sur un ou plusieurs espaces
tokens = re.split(r"\s+", text_lowercase)
print("Tokens :", tokens)
# ['hello', 'world', 'this', 'is', 'a', 'sample', 'text', 'lets', 'tokenize', 'it']

Output:

Sans ponctuation : Hello World This is a sample text Lets tokenize it
En minuscules : hello world this is a sample text lets tokenize it
Tokens : ['hello', 'world', 'this', 'is', 'a', 'sample', 'text', 'lets', 'tokenize', 'it']

Note: We use the built-in Python method .lower() for lowercase conversion rather than regex, as there is a dedicated Python function that is more appropriate for this purpose. This cleaned and tokenized data is now ready for further analysis and processing — for example, word frequency counting or comparison.

6.6 Clean data in a CSV file

Python and regular expressions can be used to process and clean data in a CSV file. This can be important for maintaining data consistency and ensuring that the data can be used effectively in further processing or analysis. For example, when working with product data files from suppliers or preparing a dataset for machine learning.

Input CSV Example

Date,Price,Product,Quantity
01/15/2023,$19.99,Widget A,100
06/20/2023,$5.49,Gadget B,250
12/31/2023,$99.00,Gizmo C,50
07/04/2023,$34.99,Thingamajig D,75

Desired transformations:

• Reformat dates: MM/DD/YYYY → YYYY-MM-DD
• Remove $ from prices

Expected result

Date,Price,Product,Quantity
2023-01-15,19.99,Widget A,100
2023-06-20,5.49,Gadget B,250
2023-12-31,99.00,Gizmo C,50
2023-07-04,34.99,Thingamajig D,75

Code

import csv
import re

def clean_data(row):
    """Nettoie et reformate une ligne de données CSV."""
    # Reformater la date de MM/DD/YYYY vers YYYY-MM-DD
    row["Date"] = re.sub(
        r"(\d{2})/(\d{2})/(\d{4})",
        r"\3-\1-\2",
        row["Date"]
    )
    
    # Supprimer le symbole $ du prix
    row["Price"] = re.sub(r"\$", "", row["Price"])
    
    return row

input_file = "products.csv"
output_file = "products_clean.csv"

with open(input_file, "r", newline="", encoding="utf-8") as in_f, \
     open(output_file, "w", newline="", encoding="utf-8") as out_f:
    
    reader = csv.DictReader(in_f)
    writer = csv.DictWriter(out_f, fieldnames=reader.fieldnames)
    
    # Écrire l'en-tête sans modifications
    writer.writeheader()
    
    # Traiter et écrire chaque ligne de données
    for row in reader:
        cleaned_row = clean_data(row)
        writer.writerow(cleaned_row)

print(f"Données nettoyées sauvegardées dans '{output_file}'.")

6.7 Manage Unicode and special characters

So far we have only seen examples of ordinary text (ASCII), but we can also handle Unicode and special characters using Regex and Python.

Unicode characters are essential when working with text in various languages ​​or symbols that are not part of the standard ASCII character set. For example, when processing international text data, mathematical symbols or emojis, one must handle Unicode characters appropriately.

Properties of Unicode characters in patterns

Steps

1. Use Unicode character properties to create regex patterns
2. Match and process Unicode and special characters in text
3. Process text (clean, extract, etc.)

Code

import re

# Texte avec des caractères non-ASCII
text = "Hello! 😀 The value of π is approximately 3.14159."

# Étape 1 : Extraire les emojis
emoji_pattern = re.compile(
    "["
    "\U0001F300-\U0001F5FF"  # Symboles et pictogrammes
    "\U0001F600-\U0001F64F"  # Emojis de visages
    "\U0001F680-\U0001F6FF"  # Transport et symboles de carte
    "\U0001F700-\U0001F77F"  # Symboles alchimiques
    "\U0001F780-\U0001F7FF"  # Symboles géométriques
    "\U0001F800-\U0001F8FF"  # Flèches supplémentaires
    "\U0001F900-\U0001F9FF"  # Symboles supplémentaires et pictogrammes
    "\U0001FA00-\U0001FA6F"  # Symboles de jeux d'échecs
    "\U0001FA70-\U0001FAFF"  # Symboles et pictogrammes étendus
    "\U00002702-\U000027B0"  # Dingbats
    "]+",
    flags=re.UNICODE
)

emojis = emoji_pattern.findall(text)
print("Emojis found:", emojis)

# Étape 2 : Extraire les symboles mathématiques et les caractères grecs
# Symboles mathématiques : u2200 à u22FF
# Caractères grecs : u0370 à u03FF
math_greek_pattern = re.compile(r"[\u2200-\u22FF\u0370-\u03FF]+", re.UNICODE)
math_symbols = math_greek_pattern.findall(text)
print("Mathematical symbols found:", math_symbols)

# Étape 3 : Supprimer les caractères Unicode non-ASCII (garder lettres latines, chiffres, espaces)
# re.UNICODE garantit que \w et \s fonctionnent avec les caractères Unicode
clean_pattern = re.compile(r"[^\w\s]", re.UNICODE)
cleaned_text = clean_pattern.sub("", text)
print("Cleaned text:", cleaned_text)

Output:

Emojis found: ['😀']
Mathematical symbols found: ['π']
Cleaned text: Hello  The value of π is approximately 314159

Note: The re.UNICODE flag ensures that \w and \s work correctly with Unicode characters. In this example, the symbol π remains in the cleaned text because it was included in the pattern for mathematical symbols.

6.8 Hide sensitive data

Sometimes you want to hide sensitive data in a document. This is essential when working with private or confidential data, for example email addresses, credit card numbers, phone numbers, etc. By writing this information (redaction), we can preserve the confidentiality of the owners of this information.

Steps

1. Define patterns to recognize sensitive information
2. Use these patterns to replace sensitive information in text data with a placeholder
3. Show result

Code

import re

# Étape 1 : Données d'exemple contenant des informations sensibles
sensitive_data = """
Bonjour,

Voici les informations du client :
- Email : john.doe@example.com
- Téléphone : (514) 555-1234
- Carte de crédit : 4532 1234 5678 9012

Cordialement,
Service client
"""

# Étape 2 : Définir les patterns regex pour les informations sensibles
email_pattern       = r"[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}"
phone_pattern       = r"\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}"
credit_card_pattern = r"\b\d{4}[\s\-]?\d{4}[\s\-]?\d{4}[\s\-]?\d{4}\b"

# Étape 3 : Remplacer par des placeholders
redacted = sensitive_data
redacted = re.sub(email_pattern,       "[EMAIL REDACTED]",       redacted)
redacted = re.sub(phone_pattern,       "[PHONE REDACTED]",       redacted)
redacted = re.sub(credit_card_pattern, "[CREDIT CARD REDACTED]", redacted)

print("Texte masqué :")
print(redacted)

Output:

Texte masqué :

Bonjour,

Voici les informations du client :
- Email : [EMAIL REDACTED]
- Téléphone : [PHONE REDACTED]
- Carte de crédit : [CREDIT CARD REDACTED]

Cordialement,
Service client

Note: This technique can be adapted to hide other types of sensitive information as needed, in all kinds of documents where we would need to replace this type of information.

7. Summary

Functions of module re

Quick reference regex syntax

Flags

Search Terms

python · regex · playbook · foundations · data · analysis · engineering · analytics · pattern · flags · string · log · match · validate · addresses · characters · clean · find · inputs · line · lookahead · numbers · object · patterns