Python 3 Performance

Level: Intermediate

Table of Contents

1. Course Overview
2. Measure performance

• 2.1 Understanding performance
• 2.2 Strategy to improve performance
• 2.3 Basic methods for measuring performance
• 2.4 Why profile?
• 2.5 More profilers
• 2.6 View profiling data

3. Use the right data structures

• 3.1 Which data structure is fastest?
• 3.2 Comparison of lists and arrays
• 3.3 Comparison of sets and tuples
• 3.4 Comparison of queues and deques
• 3.5 Use dictionaries
• 3.6 Comparison: dataclass, dictionary and namedtuple

4. Optimize Python code

• 4.1 Caching
• 4.2 For vs. list comprehension
• 4.3 Efficient iterations with generators
• 4.4 Fast string concatenation
• 4.5 Permission or Forgiveness?
• 4.6 Faster functions
• 4.7 Optimize numerical calculations
• 4.8 Interpreter-based optimizations
• 4.9 Risky Optimizations

5. Use more threads

• 5.1 What is a thread?
• 5.2 Thread-related challenges
• 5.3 When to use multithreading?

6. Use asynchronous code

• 6.1 Asynchronous code
• 6.2 Asyncio-related challenges
• 6.3 When to use asyncio?

7. Use more processes

• 7.1 Process-based parallelism
• 7.2 Inter-process communication
• 7.3 When to use more processes?
• 7.4 Switch from one to multiple machines
• 7.5 Summary

1. Course Overview

Python is a very popular programming language, widely used in practice. Unfortunately, many Python developers are frustrated by the sometimes disappointing performance of their applications. This course aims to help developers improve the performance of their Python applications.

The course outline is as follows:

1. Measure the performance of Python applications in order to establish a baseline before improving anything.
2. Write faster Python code using the right data structures.
3. Apply concrete tactics to improve performance.
4. Speed ​​up applications using threads, asyncio and additional processes.

Prerequisite: Intermediate experience with Python, including hands-on experience writing Python code for applications.

2. Measure performance

2.1 Understanding performance

To improve the performance of a Python application, you must first understand what performance really is and how to identify it. This module covers the following steps:

1. Understand performance to have a clear vision.
2. Strategy to Improve Performance — a high-level plan to guide the work.
3. Basic methods to measure performance and obtain a concrete baseline.
4. Profiling — understand its basics and see demonstrations of the profilers included in Python.
5. Other third-party profilers.
6. Best ways to visualize profiling data.

Context: the Globomantics company

The training uses a fictional company named Globomantics, which develops several Python applications for its business needs. Developers and managers are very satisfied with Python for its rich ecosystem and high productivity. Globomantics uses Python for applications ranging from web apps to data science projects, typically deployed in the cloud.

However, as data volumes increase and code bases grow, a recurring problem arises: applications are too slow. This is the main reason why performance becomes a critical issue.

2.2 Strategy to improve performance

The complexity of improving performance requires a three-part strategy:

The three pillars of strategy

1. Prevent performance issues
2. Fix performance issues
3. Measure performance issues to ensure the first two parts are effective

Prevent performance issues

• Review the architecture of the application to detect potential bottlenecks upstream. For example, if the application needs to retrieve the same data from a database many times, consider a caching mechanism.
• Be aware of performance implications when writing code — for example, choosing the right data structures.
• Use efficient libraries rather than reimplementing them. Use efficient algorithms and data structures.

Fix performance issues

Start by asking: Is the Python code really the bottleneck? It’s possible that the application is slow because of the database, not the Python code.

If the Python code is indeed the bottleneck:

• If you have a tight deadline, consider more powerful hardware as a preliminary step — especially useful for early-stage applications.
• Have good test coverage to avoid breaking existing features when optimizing.
• Consider tradeoffs: faster code may be more difficult for the team to understand.

Measure performance

Measurement is essential to guide optimization efforts and ensure that changes have the expected impact.

2.3 Basic methods for measuring performance

Performance measurement is critical. There are several approaches.

System Tools

• Windows: Task Manager — graphical interface to view processes and resources.
• macOS: Activity Monitor — similar tool.
• Ubuntu Linux: System Monitor — similar functionality.

Dedicated Linux commands

• time: measures the time required to execute a program. Can be used to measure a Python application.
• top: gives details about the running processes, their CPU and memory usage.

These commands measure the performance from outside of the Python code.

Measuring with Python

There are 3 approaches to measuring from Python code:

1. The time function

Returns the current time. To measure the duration of a piece of code, call time() before and after, then calculate the difference.

import time

def heavy_work():
    for _ in range(100_000_000):
        pass

start_time = time.time()
heavy_work()
end_time = time.time()
print(f'Duration: {end_time - start_time}')

2. The timeit module

Excellent for measuring the execution time of pieces of Python code.

3. The pytest-benchmark plugin

Allows you to integrate measurements into test suites to get early warnings if part of the application is too slow.

# test_empty_loop.py
from empty_loop import heavy_work

def test_benchmark_heavy_work(benchmark):
    benchmark(heavy_work)

Demo — sum_loop.py

This example illustrates performance measurement with time on a loop that calls another function:

import time

def heavy_work():
    for _ in range(100_000_000):
        do_stuff()

def do_stuff():
    return 1 + 2

start_time = time.time()
heavy_work()
end_time = time.time()
print(f'Duration: {end_time - start_time: .2f} seconds')

2.4 Why profile?

Profiling is a powerful approach to obtaining detailed performance metrics. The main goal is to find bottlenecks in an application.

Profiling result

A detailed report with execution times:

• How long did each function take
• How many times each function was called

Key Benefit

Profiling **takes the guesswork out of optimization efforts. It’s hard to guess the bottlenecks. Profilers provide detailed metrics to make guesswork unnecessary.

The Pareto principle (80/20 rule)

Most results come from a small number of entries. This principle also applies to performance optimization: small parts of code have the greatest impact on performance. Profilers make it possible to identify these critical portions.

Profiler types

There are two types of profilers depending on their measurement method:

1. Event profilers (event-based / deterministic)

• Collect data about all events that occur during code execution (which functions are called and how many times).
• Produce lots of detailed data → high precision on what is actually happening at runtime.
• Disadvantage: add significant overhead to the application (it runs slower during profiling). Acceptable in development or staging, but not in production.

2. Statistical profilers (sampling)

• Take periodic samples of running code.
• Less precise but add much less overhead.
• Best suited for production environments.

Profilers integrated into Python

Python includes two built-in profilers: profile and cProfile.

• profile: implemented in pure Python, very slow due to high overhead.
• cProfile: implemented in C, significantly faster than profile — recommended.

Demo — profiling with cProfile and large_function.py

import time

@profile
def heavy_work():
    print('Do something')
    print('Do something')
    print('Do something')

    for _ in range(100_000_000):
        pass

    print('Do something')
    print('Do something')

start_time = time.time()
heavy_work()
end_time = time.time()
print(f'Duration: {end_time - start_time}')

The @profile decorator is provided by line_profiler or memory_profiler depending on the use case.

Demo — memory profiling with memory_intensive.py

@profile
def create_big_list():
    return 10_000_000 * [0]

@profile
def create_huge_list():
    return 30_000_000 * [0]

create_big_list()
create_huge_list()

2.5 More profilers

The built-in profile and cProfile profilers are good starting points, but they have several limitations:

Limitations of built-in profilers

1. Performance: They tend to have slow performance and add significant overhead. The profile module is very slow, while cProfile is clearly faster. Workaround: profile only parts of the code.
2. Multithreading: They cannot profile multithreaded applications by default. Workaround: Run cProfile in each thread separately.
3. Internal visibility of functions: They give details about the time a function takes and its number of calls, but not about what happens inside the function. Workaround: Refactor the code into smaller functions.
4. Memory: they do not provide visibility into memory consumption. Workaround: use a dedicated memory profiler.

Popular third-party profilers

line_profiler

• Focuses on profiling functions line by line.
• If cProfile detects a large slow function, line_profiler allows you to gain visibility inside this function.

py-spy

• Statistical profiler (sampling) → much faster than built-in event profilers.
• Can profile multithreaded applications.

Scalene

• Also a statistical profiler.
• Adds only 20% or less overhead to a running Python application — excellent.
• Supports line by line profiling.
• Provides visibility into memory consumption.

memory_profiler

• Profiler dedicated to memory.
• Support row-by-row profiling for memory.
• Can generate graphs to visualize memory consumption of functions over time.

2.6 View profiling data

In the previous demonstrations, we obtain the profiling data directly in the console. This is acceptable for a few lines of code. However, for Python projects with thousands of lines, profiling an application generates a lot of data — navigating this data in console becomes very difficult.

We therefore need user-friendly ways of visualizing this data to extract the right insights.

Visualization with memory_profiler

memory_profiler can generate graphs to visualize the memory consumption of functions over time. Prerequisites: install Matplotlib.

# Étape 1 : profiler et générer les données
mprof run mon_script.py

# Étape 2 : générer le graphique
mprof plot -o output.jpg

The resulting JPEG graph shows the memory consumption of each function over time.

Visualization with Snakeviz

Snakeviz is a Python package dedicated to visualizing profiling data created by the Python community.

• Allows you to view the output of cProfile.
• This is an interactive browser that opens in the web browser.
• Allows you to visually explore and navigate profiling data.

# Générer les données de profilage avec cProfile
python -m cProfile -o output.prof mon_script.py

# Visualiser avec Snakeviz
snakeviz output.prof

Snakeviz displays an interactive “icicle” or “sunburst” diagram allowing you to zoom in on the most expensive functions.

3. Use the right data structures

3.1 Which data structure is fastest?

Data structures are a critical component for implementing applications. Lists, dictionaries and sets have their strengths and weaknesses. Using the right data structures brings significant performance benefits.

Module plan

1. Understand how to compare the performance of Python data structures objectively.
2. Compare lists and arrays.
3. Study sets and tuples.
4. Study queues and deques.
5. Parse dictionaries.
6. Compare dataclasses, dictionaries and namedtuples.

Big O notation

To evaluate the performance of different data structures objectively, we use Big O notation. This notation indicates the complexity of an algorithm in terms of the order of magnitude of the time required.

• Machine independent: abstracts hardware-related variations.
• Simple result: returns concise notation.
• Widely accepted in the IT community.

Common examples:

• O(1) — constant time: the operation always takes the same time, regardless of the size of the data.
• O(n) — linear time: time increases proportionally to the size of the data.
• O(n²) — quadratic time: time grows as the square of the data size.

Demo — big_o.py

import random
import time

def double_first_amount(amounts):
    return amounts[0] * 2

def sum_odd_amounts(amounts):
    sum = 0
    for a in amounts:
        if a % 2:
            sum += a
    return sum

randomAmounts = [random.randint(1, 100) for _ in range(1000)]

start_time = time.time()
double_first_amount(randomAmounts)
double_duration = time.time() - start_time

start_time = time.time()
sum_odd_amounts(randomAmounts)
sum_duration = time.time() - start_time

print(f'Duration double: {double_duration}')
print(f'Duration sum: {sum_duration}')
print(f'Ratio of durations: {sum_duration/double_duration:.2f}')

double_first_amount has complexity O(1) (direct access to the first element). sum_odd_amounts has complexity O(n) (iteration over all elements).

3.2 Comparing lists and arrays

Python Lists

Lists are one of the most popular data structures in Python. A list is an ordered collection of items. “Ordered” means there is a first element, a second element, etc. Elements can have mixed types (integers and strings in the same list). For performance reasons, it’s best to store elements of the same type in a list — this allows Python to make optimizations internally.

Internally, lists are implemented as resizable arrays: a new list is given a block of memory to store its elements side by side efficiently.

Performance of list operations

Quick operations — O(1):

• Get the value of an element: very fast.
• Set a value: very fast.
• append of a new element: very fast — Python pre-allocates extra space for future appends.

Slow operations — O(n):

• Find an element in a list: O(n) — if the code often needs to check whether elements are in a large list, consider another data structure.
• Delete an element: O(n) — Python must first find the value to delete it.

List memory management

Since lists are resizable arrays, their memory allocation mechanism has an impact on performance:

• Lists pre-allocate extra space for future appends.
• For large lists this may cause high memory consumption.
• Mitigation: avoid adding many elements to a large list. Try to build the full list from the start or use arrays.

NumPy Arrays

For numeric operations on large collections, NumPy arrays perform much better than Python lists.

Demonstration — list_array.py

import numpy

def double_list(size):
    initial_list = list(range(size))
    return [2 * i for i in initial_list]

def double_array(size):
    initial_array = numpy.arange(size)
    return 2 * initial_array

double_list(1_000_000)
double_array(1_000_000)

NumPy arrays use memory efficiently (continuous blocks) and are optimized for modern CPU architectures, making them significantly faster than lists for numeric operations.

3.3 Comparison of sets and tuples

Sets

Sets are unordered collections — unlike lists, there is no first or last element. The key difference with other collections is that the sets have unique elements: no duplicates.

• The elements of a set must be immutable (not modifiable). For example, you cannot add a list to a set since lists are mutable.
• However, the sets themselves are mutable: you can add elements to an existing set.

Set performance:

Very fast operations — O(1):

• Add elements to a set.
• Remove elements from a set.
• Membership check: very fast — unlike lists.

Linear operations — O(n):

• Remove duplicates (when creating a set from a list).

Tip: If business logic requires checking frequently whether items are part of a large collection, use a set to store that collection.

Tuples

Tuples are an interesting alternative to lists. Think of tuples as immutable lists or read-only lists.

• Less flexibility than lists (no modification possible).
• Lighter data structure: consume less memory because they do not need to reserve additional space for future appends.
• These features make tuples faster than lists.

Tradeoff: flexibility vs performance.

Comparison table sets vs tuples

Demo — set_tuple.py

import random

def search(items, collection):
    count = 0
    for i in items:
        if i in collection:
            count += 1
    return count

@profile
def main():
    SIZE = 1_000_000

    big_list = list(range(SIZE))
    big_set = set(big_list)
    big_tuple = tuple(big_list)

    items_to_find = [random.randint(0, SIZE) for _ in range(1000)]

    search(items_to_find, big_list)
    search(items_to_find, big_set)
    search(items_to_find, big_tuple)

main()

This code demonstrates that searching in a set is drastically faster than in a list or tuple for large collections.

3.4 Comparison of cues and deques

Python provides several implementations of queues. Let’s compare the two main ones.

Queue (queue module)

• Must be imported before use: from queue import Queue.
• Implements a FIFO (First In, First Out) structure: first in, first out — like a queue of people.
• Particularly used for multithreading to implement scenarios with multiple consumers and producers.
• Offers few operations.

from queue import Queue

orders = Queue()
orders.put({'id': 1, 'item': 'coffee'})
orders.put({'id': 2, 'item': 'tea'})
orders.put({'id': 3, 'item': 'juice'})

first_order = orders.get()  # Retire et retourne le premier élément

Deque (collections module)

• Deque = double-ended queue.
• Optimized for operations at both ends of the queue.
• Supports FIFO (first in, first out) AND LIFO (last in, first out — like a stack).
• Also supports multithreading.
• More operations available than Queue.

from collections import deque

orders = deque()
orders.append('coffee')      # Ajoute à droite
orders.appendleft('urgent')  # Ajoute à gauche
orders.pop()                 # Retire à droite
orders.popleft()             # Retire à gauche

Demo — deque.py

from collections import deque

@profile
def main():
    SIZE = 100_000

    big_list = list(range(SIZE))
    big_queue = deque(big_list)

    while big_list:
        big_list.pop()
    while big_queue:
        big_queue.pop()
    
    big_list = list(range(SIZE))
    big_queue = deque(big_list)

    while big_list:
        big_list.pop(0)    # O(n) — doit décaler tous les éléments
    while big_queue:
        big_queue.popleft()  # O(1) — très rapide

main()

The pop(0) operation on a list (indent at the beginning of the list) is O(n) because Python must shift all elements. popleft() on a deque is O(1).

Summary: when to use what?

• Queue: Simple multithreaded scenarios with only one end.
• deque: when we need fast operations at both ends, or when the list is used as a stack/queue.

3.5 Using dictionaries

The data structures seen so far stored individual items in a collection. Dictionaries are different: they store key-value pairs.

Characteristics of dictionaries

• Dictionaries are mutable: you can add, delete pairs, or update a value.
• Keys play a critical role in accessing values.
• Anything can be stored as value.
• Constraints on keys:
• Keys must be unique (like sets — we can also think of sets as dictionaries having only keys without associated values).
• Keys must be hashable (cacheable), that is, have a hash value that never changes during their lifetime.

Dictionary performance

Dictionaries are typically very fast with constant time complexity:

• Get an element: O(1)
• Define an element: O(1)
• Delete an element: O(1)

In very rare cases, some operations may be O(n) (worst case only).

Dictionaries are highly optimized — they are used extensively in Python itself.

Impact of hashes

To achieve high performance, dictionaries must calculate hashes for keys. This imposes constraints: keys must be hashable (strings, integers, tuples — but not lists).

Demonstration — dictionary.py

import random

def search_list(big_list, items):
    count = 0
    for item in items:
        for order in big_list:
            if item == order[0]:
                count += 1
    return count

def search_dictionary(some_dictionary, items):
    count = 0
    for item in items:
        if item in some_dictionary:
            count += 1
    return count

@profile
def main():
    SIZE = 100_000

    big_list = []
    big_dictionary = {}

    for i in range(SIZE):
        big_list.append([i, 2 * i, i * i])
        big_dictionary[i] = [2 * i, i * i]

    orders_to_search = [random.randint(0, SIZE) for _ in range(1000)]
    search_list(big_list, orders_to_search)
    search_dictionary(big_dictionary, orders_to_search)

main()

Dictionary lookup is O(1) vs O(n × m) for nested list — the dictionary is dramatically faster.

3.6 Comparison: dataclass, dictionary and namedtuple

Named tuples

named tuples are specialized tuples with named fields.

• Ordinary tuples use indices to access their fields.
• Named tuples use names to access their fields → better readability.
• Always immutable (cannot be modified after creation).
• Excellent for storing read-only data (e.g. data read from a database).
• Can be unpacked into function arguments.

from collections import namedtuple

Order = namedtuple('Order', ['id', 'amount', 'customer'])
order = Order(id=1, amount=150.00, customer='Alice')
print(order.id)       # Accès par nom
print(order.amount)

Data classes

data classes are similar to named tuples but offer more flexibility.

• Allows you to store data using classes while avoiding writing boilerplate code.
• To transform an ordinary class into a data class: decorate with @dataclass.
• By default, data classes are mutable. We can make them immutable with frozen=True.
• Support type hints to improve code robustness.

from dataclasses import dataclass

@dataclass
class Order:
    id: int
    amount: float
    customer: str

order = Order(id=1, amount=150.00, customer='Alice')
print(order.id)
order.amount = 200.00  # Mutation possible par défaut

Comparison table: dataclass vs namedtuple

Tip: use namedtuples for lightweight immutable data (e.g.: BD results). Use dataclasses when you need class functionality and/or mutability.

4. Optimize Python code

4.1 Caching

Caching is a particularly effective technique for dealing with two types of bottlenecks:

1. Computation bottlenecks: functions that perform a lot of calculations and take a lot of time. The idea: store the results of these expensive calculations in a cache to reuse them and avoid redoing the calculations.

2. Network bottlenecks: sending many requests to an external API. It is more efficient to cache results and reuse them rather than making costly calls again.

Caching limitations

• The application will require additional memory to store the results. It’s not realistic to cache everything — storing only the most used data.
• The cache may contain outdated data if the source data changes. A cache invalidation strategy must be implemented.
• Caching is not suitable for all situations, for example if inputs always vary or results change frequently.

lru_cache — Python’s built-in LRU cache

Python provides lru_cache in the functools module. LRU stands for Least Recently Used. This cache keeps in memory the results of the last N calls to a function.

from functools import lru_cache

@lru_cache
def get_order_details(order_id):
    # Simule un calcul coûteux
    for i in range(100_000):
        pass
    return 100 * order_id

Demo — caching.py

from functools import lru_cache
import random

@lru_cache
def get_order_details(order_id):
    for i in range(100_000):
        pass
    return 100 * order_id

@profile
def main():
    orders_to_search = [random.randint(0, 100) for _ in range(1000)]
    for order in orders_to_search:
        get_order_details(order)

main()

Thanks to @lru_cache, when the same order_id is requested several times, the result is returned directly from the cache without executing the expensive loop. The performance gain is massive on high repetition data.

4.2 For vs. list comprehension

A typical Python application must often process elements in these three steps:

1. Prepare input data: a collection of elements to process (list, set, etc.).
2. Process elements: simple transformation or sophisticated steps.
3. Return result as a new collection.

There are two popular approaches to implementing these steps: for loops and list comprehensions.

Code comparison

For loop:

def loop(orders):
    result = []
    for amount in orders:
        if amount > 50:
            result.append(2 * amount)
    return result

List comprehension:

def comprehension(orders):
    return [2 * amount for amount in orders if amount > 50]

List comprehension is more concise and faster.

Advantages and disadvantages

Demonstration — comprehension.py

import random

def loop(orders):
    result = []
    for amount in orders:
        if amount > 50:
            result.append(2 * amount)
    return result

def comprehension(orders):
    return [2 * amount for amount in orders if amount > 50]

@profile
def main():
    orders = [random.randint(0, 100) for _ in range(100_000)]
    loop(orders)
    comprehension(orders)

main()

Tip: use for loops for complex business logic, and list comprehensions for simple transformations where performance matters.

4.3 Efficient iterations with generators

generator expressions are another approach to processing a collection of elements. Think of generator expressions as a lazy version of comprehensions.

Why is “lazy” a good thing here?

Generator expressions avoid the upfront cost of creating the list entirely. Instead, values ​​are created just in time, one at a time, when they are needed.

Concrete example: to process a very large file line by line — with a list comprehension, you would have to read the entire file into memory at once. With a generator, we read each line one by one, which is much more memory efficient.

Limitations of generator expressions

1. Single iteration: a generator expression can only be iterated once. A list created with a list comprehension can be iterated as many times as necessary.
2. No random access: we cannot directly access the 10th element then the 5th. Lists provide this random access.

Syntax

# List comprehension — crée toute la liste en mémoire immédiatement
comprehension = [2 * amount for amount in orders if amount > 50]

# Generator expression — crée les valeurs à la demande
generator = (2 * amount for amount in orders if amount > 50)

The only syntactic difference is the use of parentheses () instead of brackets [].

Demo — generator.py

import random

@profile
def main():
    orders = [random.randint(0, 100) for _ in range(100_000)]

    comprehension = [2 * amount for amount in orders if amount > 50]
    generator = (2 * amount for amount in orders if amount > 50)

    sum(comprehension)
    sum(generator)

main()

Result: list comprehension consumes more memory (the entire list is created in memory). The generator consumes much less memory because it generates the values ​​on the fly.

Tip: use generators when you only need to traverse the collection once and memory is a constraint.

4.4 Fast string concatenation

Concatenating strings may seem trivial, but has a significant performance impact.

Two scenarios

1. Small fixed number of strings: the impact on performance is generally limited.
2. Large variable number of strings: the impact on performance is significant.

The three main approaches

1. Operator +

Intuitive and very readable. Main drawback: tends to be slow because Python strings are immutable. Python cannot modify an existing string — it must recreate a new one with each concatenation.

result = item1 + item2

2. f-strings

Excellent for formatting strings. To concatenate a few strings, enclose their variable names in curly brackets.

result = f'{item1}{item2}'

3. join method

Concatenates a set of strings into a larger string. Inserts a custom string between each string — to simply join, use '' as a separator.

items = ['hello', 'world']
result = ''.join(items)  # 'helloworld'
result = ' '.join(items)  # 'hello world'

Performance table

Demo — concatenation.py

import random

@profile
def main():
    orders = [str(random.randint(0, 100)) for _ in range(50_000)]

    # Approche lente avec +
    report = ''
    for o in orders:
        report += o
    
    # Approche rapide avec join
    ''.join(orders)

main()

Advice: to concatenate a large number of strings, always prefer ''.join(liste_of_strings).

4.5 Permission or Forgiveness?

In the real world, problems can arise in a Python application: missing file, missing fields, unexpected types, etc. There are two approaches to managing them.

Approach 1: Permission (LBYL — Look Before You Leap)

The idea is to check if an operation will succeed before attempting it. For example, verify that the file exists in the specified location before reading it.

• Uses a lot of if statements to check that everything is in order.
• Defensive and preventive approach.

def permission(orders):
    result = []
    for amount in orders:
        if type(amount) == int:    # Vérifie le type avant
            if amount > 50:
                result.append(2 * amount)
    return result

Approach 2: Forgiveness (EAFP — Easier to Ask Forgiveness than Permission)

Instead of checking up front, the approach is to handle issues after they occur. We use try/except blocks to catch and handle potential exceptions.

• Popular in Python (Pythonic philosophy).
• Added benefit: Helps prevent race conditions in multithreaded code. For example, with the permission approach, the code checks if the file exists on disk, but the file may be deleted just before opening.

def forgiveness(orders):
    result = []
    for amount in orders:
        try:
            if amount > 50:
                result.append(2 * amount)
        except TypeError:
            pass
    return result

Performance comparison

Demo — permission.py

import random

def permission(orders):
    result = []
    for amount in orders:
        if type(amount) == int:
            if amount > 50:
                result.append(2 * amount)
    return result

def forgiveness(orders):
    result = []
    for amount in orders:
        try:
            if amount > 50:
                result.append(2 * amount)
        except TypeError:
            pass
    return result

@profile
def main():
    orders = [random.randint(0, 100) for _ in range(100_000)]

    # Peu de mauvaises données
    for i in range(10):
        orders[i] = 'bad data'
    permission(orders)
    forgiveness(orders)

    # Beaucoup de mauvaises données
    for i in range(90_000):
        orders[i] = 'bad data'
    permission(orders)
    forgiveness(orders)

main()

Result: when exceptions are rare, the forgiveness approach (EAFP) is faster because it avoids systematic checks. When exceptions are frequent, permission can perform better. In practice, in well-structured code, error cases are rare — so EAFP is generally the recommended approach.

4.6 Faster functions

Functions are used extensively in all Python applications. Understanding their impact on performance is essential.

Classic functions vs lambda

lambda functions are very small anonymous functions (usually one line of code) without their own name.

# Fonction classique
def get_random_integer():
    return random.randint(0, 100)

# Lambda équivalente
lambda: random.randint(0, 100)

The cost of function calls

Calling other functions has a cost. When a function is called thousands of times, this cost accumulates.

Demo — functions.py

import random

def get_random_integer():
    return random.randint(0, 100)

@profile
def main():
    [random.randint(0, 100) for _ in range(100_000)]           # Appel direct
    [get_random_integer() for _ in range(100_000)]             # Via une fonction
    [(lambda: random.randint(0, 100))() for _ in range(100_000)]  # Via lambda

main()

Result: Direct call to random.randint is fastest. Calling an intermediate function (get_random_integer) or a lambda adds overhead, because Python must handle the additional calling context.

Standalone functions vs nested function calls

Approach with call to another function:

def heavy_work():
    for _ in range(100_000_000):
        do_stuff()            # Appel coûteux répété 100 millions de fois

def do_stuff():
    return 1 + 2

Standalone approach (inline):

def heavy_work():
    for _ in range(100_000_000):
        result = 1 + 2        # Logique intégrée directement

Comparison table

Tip: only avoid function calls in very hot loops where performance is critical. Don’t sacrifice maintainability for small gains.

4.7 Optimize numerical calculations

Numerical calculations are a critical component of many Python applications: arithmetic operations, vector operations, statistical calculations, machine learning, etc.

NumPy — Numerical Python

NumPy is a Python package dedicated to numerical calculations for scientific computing.

• Considered the unofficial standard for scientific computing.
• NumPy arrays are very fast and powerful.
• Open source with a large ecosystem.
• Used in data science, machine learning, visualization.
• Prerequisites for other popular packages (Matplotlib, Pandas, scikit-learn, etc.).

NumPy internal optimizations:

• Many functions are implemented in C for maximum performance.
• Arrays use memory efficiently with contiguous blocks.
• Optimized to take advantage of modern CPU architectures.

Pandas

Pandas is another popular open source package for numerical calculations.

• Internally, Pandas relies on NumPy for performance.
• Particularly popular for data analysis and manipulation.
• Provides many functions for working with data (filtering, grouping, pivots, etc.).
• Ideal for working with tabular data (like CSVs or SQL tables).

Demo — numerical.py

import random
import numpy as np

def loop_approach(orders):
    result = 0
    for o in orders:
        result += o * o
    return result

def numpy_approach(orders):
    numpy_orders = np.array(orders)
    return np.sum(numpy_orders * numpy_orders)

@profile
def main():
    orders = [random.randint(0, 100) for _ in range(100_000)]
    loop_approach(orders)
    numpy_approach(orders)    
    
main()

Result: The NumPy approach is significantly faster than the Python loop for numeric operations on large collections.

4.8 Interpreter-based optimizations

Python interpreters offer very interesting possibilities for obtaining surprising performance gains on existing Python applications.

What is a Python interpreter?

A Python interpreter is a tool that performs various transformations under the hood to enable execution of Python code on hardware. These transformations convert Python code into instructions that the underlying hardware can understand and execute.

The official interpreter: CPython

CPython is the most popular interpreter and reference implementation of Python. It is written in C.

Advantages:

• Low risk: this is the official implementation.
• High portability: works on a very wide variety of hardware configurations.
• Wide compatibility with libraries.

Other alternative interpreters

PyPy

• Can run Python code faster than CPython thanks to a JIT (Just-In-Time) compiler that compiles Python code into machine code at runtime.
• Significant gains for CPU-intensive applications.

Cython

• Takes Python code and compiles it into efficient C code.
• Adds additional syntax for calling C functions and declaring C types.
• Increased performance, but code can no longer be run directly with CPython.

Jython

• Implemented in Java.
• Integrates easily with Java code.
• Quite behind the official interpreter in terms of features.

Pyston

• Aims to be a fast and highly compatible Python implementation.
• Good compatibility with the existing Python ecosystem.

Tip: If you’re experimenting with other Python interpreters, make sure you have a good regression test suite to catch potential issues before reaching production.

4.9 Risky optimizations

Optimizations often have tradeoffs. It is important to be aware of this to make the best decisions.

Main risks

1. Code more difficult to read and understand → more difficult to maintain over time. Mitigation: Use comments, documentation and testing.

2. Introduction of new bugs during optimization. Mitigation: have a good regression test suite.

3. Significant effort for negligible performance gain. Mitigation: use metrics and profilers to ensure you focus on the real bottlenecks.

Examples of risky optimizations

1. Large stand-alone functions

Avoiding calls to other functions may give a small performance gain, but these large functions are less maintainable.

2. Use an alternative interpreter

May produce clear performance gains, but may introduce bugs due to compatibility issues with certain modules.

3. Multiple assignments

Can slightly improve performance but tends to make the code more difficult to understand.

Demo — assignments.py

import random

def multiple_assignments(order):
    order_subtotal, order_tax, order_shipping = order   # Déstructuration en une ligne

def individual_assignments(order):
    order_subtotal = order[0]
    order_tax = order[1]
    order_shipping = order[2]

@profile
def main():
    orders = [(random.randint(0, 100),
               random.randint(0, 100),
               random.randint(0, 100)) for _ in range(100_000)]
    
    for o in orders:
        multiple_assignments(o)
        individual_assignments(o)

main()

Result: Multiple assignments (destructuring) are slightly faster, but the difference is small. The real risk is reduced readability for other developers.

5. Use more threads

5.1 What is a thread?

So far, we have improved the performance of Python applications by focusing on single-threaded code. multithreading can also improve performance. Here are the key points.

Thread characteristics

1. Separate execution flows: The same lines of code are executed separately in different threads.
2. One or more threads per process: the programs seen previously used a single thread, but a process can use several.
3. Concurrency: threads allow a Python program to do several tasks concurrently within a single process.
4. Global Interpreter Lock (GIL): Python has a GIL that only allows one thread to run at a time.
5. Despite the GIL, using multiple threads can improve performance in certain scenarios (eg: I/O intensive applications).

Differences between threads and processes

Demo — more_threads.py

import threading
from time import sleep

def process_order(order_id):
    print(f'Processing order with id={order_id}')
    sleep(1)
    print(f'Finished processing order with id={order_id}')

first_thread = threading.Thread(target=process_order, args=(10,))
second_thread = threading.Thread(target=process_order, args=(20,))

first_thread.start()
second_thread.start()

With sleep(1), the two orders are processed in parallel — the total time is ~1 second instead of 2 seconds.

Demo — challenges_threads.py (with CPU loop)

import threading
from time import sleep

def process_order(order_id):
    print(f'Processing order with id={order_id}')
    # sleep(1)
    for _ in range(100_000_000):  # Tâche CPU-intensive
        pass
    print(f'Finished processing order with id={order_id}')

first_thread = threading.Thread(target=process_order, args=(10,))
second_thread = threading.Thread(target=process_order, args=(20,))

first_thread.start()
second_thread.start()

With a CPU-intensive loop, the GIL prevents the two threads from truly executing in parallel — the gain is zero or even negative.

5.2 Thread Challenges

Three main challenges

1. Synchronize threads: coordinate the execution of two or more threads so that they can share resources (common variables). Bad synchronization can freeze, crash or give bad results.

2. Debugging multithreaded code: If a piece of code has a timing bug, finding it can be very difficult.

3. The constraints imposed by the GIL.

The four types of sync bugs

1. Breed conditions

Two or more threads attempt to read and write to the same resource at the same time, producing unpredictable behavior. The value of a variable can be modified by another thread and produce unexpected results.

Solution: Locks — a mechanism to ensure that only one thread acquires a resource at a time and releases it when finished.

2. Deadlocks

Two threads are waiting for the other to release a resource. None releases it → the application appears blocked.

3. Starvation (starvation)

A thread cannot acquire a resource because of other threads. The starved thread cannot complete its execution → the overall performance of the application decreases significantly.

4. Livelocks

Multiple threads acquire and release resources in a way that does not allow any of them to make progress. Results in applications that crash or have very poor performance.

The GIL — Global Interpreter Lock

The GIL is a fundamental limitation of CPython. It prevents multiple threads from running simultaneously on multiple CPU cores. This means:

• For I/O tasks (waiting for network response, reading files): the thread releases the GIL while waiting → threads can alternate efficiently → performance gain.
• For CPU-intensive tasks: the thread does not release the GIL → no real parallelism → no performance gain.

5.3 When to use multithreading?

Three synchronization mechanisms

1. Lock

A synchronization object that only allows one thread to acquire it. Other threads are blocked until the initial thread releases the lock.

import threading
lock = threading.Lock()

with lock:
    # Section critique — un seul thread à la fois
    shared_resource.modify()

2. Semaphore

Allows a predefined number of threads to acquire it simultaneously. Example: limit the number of connections to a database.

semaphore = threading.Semaphore(5)  # Maximum 5 threads simultanés

with semaphore:
    connect_to_database()

3. Condition varies

Allows a thread to wait until a certain condition is met before continuing execution. Used to notify waiting threads that a resource has just been updated.

When to use threads

✅ Use threads for:

• Tasks involving waiting for external events (eg: user input) → makes the application more responsive.
• Blocking I/O operations (network read/write, files) → improves performance.
• Relatively simple logic not requiring sophisticated synchronization.

❌ Avoid threads for:

• CPU-intensive tasks (GIL prevents true parallelism).
• Complex code with elaborate synchronization → high risk of bugs.
• When asyncio or multiprocessing are more suitable.

Demo — download_threads.py

import threading
from urllib import request

def download():
    return request.urlopen('https://google.com').read()

def single_thread():
    for _ in range(5):
        download()

def multiple_threads():
    threads = []
    for _ in range(5):
        threads.append(threading.Thread(target=download))
    for t in threads:
        t.start()
    for t in threads:
        t.join()

@profile
def main():
    single_thread()
    multiple_threads()

main()

Result: The 5 multi-threaded downloads are significantly faster because each download releases the GIL during network waiting.

6. Use asynchronous code

6.1 Asynchronous code

This module focuses on asynchronous code as an alternative to multithreading.

Key points of asynchronous code

1. Inspired by other languages: JavaScript, Go and C# have had success with asynchronous code. This is why it was added relatively recently to Python via the asyncio module.

2. Two key benefits:

• Reduced risk of bugs: asyncio reduces the risk of synchronization bugs compared to threads, allowing large applications to be implemented without encountering difficult-to-debug bugs.
• Maximizing core usage: With threads, context switching between many small threads wastes resources. asyncio doesn’t have this context switching → better performance for many small tasks.

3. Tradeoffs: in exchange for fewer bugs and better performance, asyncio introduces new syntax, new concepts and new tools.

Fundamental concepts of asyncio

• Coroutines: the equivalent of Python functions in asyncio. Defined with async def.
• await: suspends the execution of the current coroutine until the expected coroutine is completed.
• Event loop: the heart of asyncio applications. It takes care of coordinating all calls.

import asyncio

async def process_order():
    await asyncio.sleep(1)
    print('Order complete')

async def main():
    await process_order()
    await process_order()
    print('Finished processing')

asyncio.run(main())

Concurrent execution with asyncio.gather

async def main():
    await asyncio.gather(process_order(), process_order())
    print('Finished processing')

asyncio.gather allows multiple coroutines to be executed concurrently.

Demo — orders_asyncio.py

import asyncio

async def process_order():
    await asyncio.sleep(1)
    print('Order complete')

async def main():
    await process_order()
    await process_order()
    print('Finished processing')

asyncio.run(main())

Demo — challenges_asyncio.py (CPU-intensive task)

import asyncio
import time

async def process_order():
    # await asyncio.sleep(1)   # I/O — permettrait la concurrence
    # time.sleep(3)            # Bloquant — bloquerait tout
    for _ in range(100_000_000):  # CPU-intensif — pas de concurrence
        pass
    print('Order complete')

async def main():
    await asyncio.gather(process_order(), process_order())
    print('Finished processing')

asyncio.run(main())

Demonstration: a CPU-intensive task in a coroutine blocks the event loop — asyncio brings no gain for CPU-intensive tasks.

6.2 Challenges related to asyncio

Three main challenges

1. Learning curve

• New syntax: async and await differentiate traditional synchronous code from new asynchronous code.
• New concepts: coroutines (equivalent of functions), event loop (coordination), etc.
• New libraries: to obtain the best performance with asyncio, use specific libraries:
• aiohttp for HTTP requests
• aiomysql for MySQL
• aiosqlite for SQLite
• etc.

2. Debugging

• Execution order difficult to follow: Multiple tasks are running concurrently. The event loop coordinates their execution.
• Complex application status: for the same reason.
• Mitigation: use logging and debugger to clarify execution order and status.

3. Compatibility

• Difficulty integrating asyncio into synchronous legacy code.
• If a required dependency uses blocking code, performance will be affected.
• Migrating to asyncio may require rewriting significant parts of the code.

6.3 When to use asyncio?

Recommended use cases for asyncio

✅ Use asyncio for:

• Applications with a lot of I/O operations (disk read, external resources).
• Management of many small tasks → better performance than multithreading (no context switching overhead between threads).
• Applications that want to avoid thread synchronization complexity and subtle bugs.
• When dependent libraries have asynchronous versions optimized for asyncio.
• Data processing pipelines processing large volumes of data concurrently.
• Network applications using many I/O operations.

Demo — download_asyncio.py

import asyncio
import aiohttp
from urllib import request

def download():
    return request.urlopen('https://google.com').read()

def synchronous():
    for _ in range(5):
        download()

async def async_download(session, url):
    async with session.get(url) as response:
        return await response.text()

async def asynchronous():
    async with aiohttp.ClientSession() as session:
        coroutines = [async_download(session, 'https://google.com') for _ in range(5)]
        await asyncio.gather(*coroutines)

@profile
def main():
    synchronous()
    asyncio.run(asynchronous())

main()

Result: the asynchronous version with aiohttp is significantly faster than the synchronous version for network downloads.

Cases where to avoid asyncio

❌ Avoid asyncio for:

• CPU-intensive tasks (numerical calculations, etc.): asyncio provides no improvement — use multiprocessing instead.
• Code using blocking code: since asyncio only uses the main thread, any blocking code blocks the entire application.
• Dependencies required with blocking code: many Python libraries do not yet have an asynchronous version.

7. Use more processes

7.1 Process-based parallelism

Multithreading and asyncio are limited to a single CPU core. Multiprocessing allows you to use multiple CPU cores to significantly increase performance.

Limitations of multithreading and asyncio

1. Increased risk of bugs: Multithreaded code is particularly vulnerable to complex timing bugs.
2. High learning curve: asyncio requires learning new syntax and new libraries.
3. Limited to a single CPU core: neither threads nor asyncio handle CPU-intensive tasks well.

Advantages of multiprocessing

• Isolation: Each process has its own separate memory space. If one process crashes, the others are not affected → more stable applications.
• No GIL: unlike threads, there is no Global Interpreter Lock for processes → reduced risk of complex synchronization bugs.
• Using all CPU cores: Production machines have many CPU cores. Multiprocessing makes it possible to exploit them all.

Multiprocessing tradeoffs

• Process creation is slower than thread creation.
• Inter-process communication (IPC) is more complex and has overhead.
• Processes have separate memory → data exchange between processes requires serialization.

Demo — more_processes.py

from multiprocessing import Process

def clean_order():
    for _ in range(500_000_000):
        pass
    print('Finished cleaning')

if __name__ == '__main__':
    p1 = Process(target=clean_order)
    p2 = Process(target=clean_order)

    p1.start()
    p2.start()

    p1.join()
    p2.join()

Demo — clean_processes.py

from multiprocessing import Process

def clean_order(order_id):
    for _ in range(500_000_000):
        pass
    print(f"Finished cleaning order with id={order_id}")

if __name__ == '__main__':
    first = Process(target=clean_order, args=(10,))
    second = Process(target=clean_order, args=(20,))

    first.start()
    second.start()

    first.join()
    second.join()

Demo — clean_threads.py (comparison)

import threading

def clean_order(order_id):
    for _ in range(500_000_000):
        pass
    print(f"Finished cleaning order with id={order_id}")

if __name__ == '__main__':
    first = threading.Thread(target=clean_order, args=(10,))
    second = threading.Thread(target=clean_order, args=(20,))

    first.start()
    second.start()

    first.join()
    second.join()

Result: for a CPU-intensive task, the version with processes is twice as fast (uses 2 cores) while the version with threads brings no gain (GIL blocks true parallelism).

7.2 Communication between processes

CPU-intensive applications benefit from using multiple processes. This creates the challenge of communication between processes, as they often need to coordinate and exchange data.

Challenges of inter-process communication

• Handling exceptions and errors that may occur during communication (can block or crash an application).
• Process synchronization with mechanisms such as locks and semaphores to limit the number of processes accessing a shared resource.
• Using join() to ensure that the main process waits for all processes to finish executing.
• Load balancing: ensure that all processes receive enough work. Avoid situations where one process is overloaded while others are idle.

Two main approaches for inter-process communication

1. Blowjobs

Quick and easy approach to sending data from one process to another.

• A pipe can only have two ends → usable only for exchange between two processes.

from multiprocessing import Pipe

parent_conn, child_conn = Pipe()
# Envoyer depuis le processus enfant
child_conn.send({'order_id': 1, 'amount': 150})
# Recevoir dans le processus parent
data = parent_conn.recv()

2. Tails

Offer more functionality and are better suited to larger amounts of data.

• Allow more than two processes to exchange data (unlike pipes).

Demonstration — Producer/Consumer Pattern

from multiprocessing import Process, Queue
import random

def producer(queue):
    for _ in range(10):
        queue.put(random.randint(0, 100))
    queue.put(None)  # Sentinel pour signaler la fin

def consumer(queue):
    while True:
        item = queue.get()
        if item is None:
            break
        # Traiter l'élément
        print(f'Processing: {item}')

if __name__ == '__main__':
    q = Queue()
    p = Process(target=producer, args=(q,))
    c = Process(target=consumer, args=(q,))

    p.start()
    c.start()

    p.join()
    c.join()

7.3 When to use more processes?

Comparison: multiprocessing vs threading vs asyncio

multiprocessing, threading and asyncio complement each other — their strengths and weaknesses are complementary.

Three main use cases for multiprocessing

1. Basic Data Processing Pipelines: Each stage of the pipeline is a separate process. The output of one process is the input to the next.

2. Producer/Consumer Pattern: one or more processes are producers, others have the role of consumers who take inputs from producers and process them.

3. Parallelizable workloads: If the application needs to process thousands of independent commands, this workload can be split and distributed across multiple processes to take advantage of multiple CPU cores.

7.4 Switching from one to multiple machines

As Python applications process larger and larger volumes of data, scaling becomes critical.

• Threads and asyncio: allow you to extract a lot of performance from a single CPU core (especially I/O).
• Multiprocessing: allows you to use multiple CPU cores for CPU-intensive tasks.
• But: a machine has a limited number of CPU cores.

The next step is to move from one machine to multiple machines to access even more memory and CPU cores. In addition to performance, the application also becomes more reliable (if one machine breaks down, the others take over).

Celery

Celery is an open source Python library that provides a task queue for handling various tasks outside of the main application.

• Example: for a Python web application, send long tasks to the task queue so as not to block the main web application.
• Uses a dedicated message broker (like Redis) for communication between the main program and its workers.
• Allows you to run tasks on several worker machines.
• Just as multiprocessing goes from one core to many cores, Celery goes from one machine to many machines.

from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def process_order(order_id):
    # Traitement long
    return result

Dask

Dask is a free and open source Python library for parallel computing.

• Allows you to scale Python workflows from a single computer to clusters of machines.
• Particularly popular in the data science ecosystem because it integrates well with NumPy, Pandas and scikit-learn.
• Offers Dask DataFrames similar to Pandas DataFrames but distributed.
• Automatically manages the distribution of tasks across multiple machines.

Demo — hello_dask.py

from dask.distributed import Client

def clean_order(order_id):
    for _ in range(500_000_000):
        pass
    print(f"Finished cleaning order with id={order_id}")

if __name__ == '__main__':
    client = Client()
    client.map(clean_order, [10, 20])

With Dask, client.map() automatically distributes work to available workers (locally or on a cluster).

Ray

Ray is another popular library for distributed execution of Python code. Particularly used in machine learning and reinforcement learning projects. Ray provides simple primitives to distribute the computation.

7.5 Summary

This module focused on using multiple processes so that Python applications can use all the CPU cores on a machine and handle CPU-intensive workloads.

Full Course Summary

The course covered the following topics in order:

Module 2 — Measuring performance

• Using basic terminal commands through to dedicated profilers to identify bottlenecks.

Module 3 — Good data structures

• Discussion of the most popular Python data structures to understand their performance and use cases. Sometimes bottlenecks come from using bad data structures.

Module 4 — Optimization tactics

• Robust list of concrete optimization tactics (caching, list comprehensions, generators, concatenation, etc.) to improve the performance of single-threaded code.

Module 5 — Multithreaded code

• How multithreaded code can deliver even more performance gains while avoiding pitfalls.

Module 6 — Asynchronous code

• asyncio to increase performance, especially for applications with many small I/O workloads.

Module 7 — Multiprocessing and scaling

• Using multiple CPU cores on a machine and even multiple machines to make Python applications fast, even with very large amounts of data.

Next steps

Apply the knowledge learned in this course to improve the performance of Python applications in your organization:

1. Measure first — identify real bottlenecks with profilers before optimizing.
2. Use the right data structures — choose list, set, dict, tuple, deque depending on the use case.
3. Apply optimization tactics — caching, list comprehensions, generators, join for strings, etc.
4. Parallelize intelligently — threads for I/O, asyncio for many small I/O tasks, multiprocessing for CPU-intensive tasks.
5. Scale if necessary — Celery or Dask to distribute to multiple machines.

Search Terms

python · performance · foundations · data · analysis · engineering · analytics · comparison · asyncio · challenges · limitations · multiprocessing · processes · profilers · tuples · asynchronous · cases · communication · functions · interpreter · list · optimizations · permission · sets