Python Requests Playbook

Python library requests · HTTP · REST APIs · Authentication · Sessions

Table of Contents

1. Course Overview
2. Introduction and Requests Library

• 2.1 What is a Playbook course?
• 2.2 Use the Requests library to access APIs
• 2.3 Configure demo API server
• 2.4 Module 2 Summary

3. Parse GET requests and Response object

• 3.1 Initiate GET requests
• 3.2 Analyze response status code
• 3.3 Handle HTTP exceptions with raise_for_status
• 3.4 Inspect Response Object
• 3.5 Passing values with query string parameters
• 3.6 Assign event hooks
• 3.7 Module 3 Summary

4. Send data to server

• 4.1 Submit data with POST requests
• 4.2 Working with form data
• 4.3 Working with JSON data
• 4.4 Working with XML and other data types
• 4.5 Upload files with POST
• 4.6 Use other HTTP methods (PUT, PATCH, DELETE)
• 4.7 Module 4 Summary

5. Provide and receive additional data with headers

• 5.1 Analyze response headers
• 5.2 Customize query headers
• 5.3 Module 5 Summary

6. Persist connections with cookies and sessions

• 6.1 Send and receive cookies
• 6.2 Persisting connections with sessions
• 6.3 Retrying connections with Transport Adapters
• 6.4 Module 6 Summary

7. Manage redirects and timeouts

• 7.1 Working with redirects and history
• 7.2 Configure timeouts
• 7.3 Module 7 Summary

8. Authenticate requests for secure APIs

• 8.1 Use HTTP Basic Authentication
• 8.2 OAuth 2.0 and the requests-oauthlib library
• 8.3 Exploit OAuth 2.0 for data recovery
• 8.4 Implement custom authentication methods
• 8.5 Ensuring data privacy with HTTPS
• 8.6 Module 8 Summary

9. Appendix — Demo FastAPI API Server

1. Course Overview

Estimated duration: 1m 36s

Efficient retrieval of data from the Internet is a crucial task for many applications. This course equips you with the skills to use Python’s requests library, which is the de facto standard for making HTTP requests in Python.

Topics covered

• Retrieve data with GET requests
• Submit data to server
• Manage cookies and sessions
• Manage redirects and timeouts
• Authentication and security (HTTP Basic Auth, OAuth 2.0)

Prerequisites

Before starting this course, you should be familiar with:

• Python Language Basics
• How HTTP and the web work (request-response cycle)

Course structure

This course is a Playbook, which means that each lesson represents a separate strategy (a “play”) that you can apply with the requests library. You can, in theory, watch the lessons in any order depending on what you want to learn. However, if you are a beginner in this field, it is recommended to watch them in order.

2. Introduction and the Requests library

Estimated duration: 11m 31s

2.1 What is a Playbook course?

A Playbook is, in the traditional sense, a collection of strategies or plays. In the United States, this term is often associated with American football, where it describes tactics to win the game. In the context of this course, each lesson represents a different strategy that you can apply with the requests library.

The key feature of this course structure is flexibility: you can watch the lessons in any order according to your needs. However, the first module (Module 2) should be watched in full as it explains how to configure the demo API server.

2.2 Use the Requests library to access APIs

The Web and HTTP

The World Wide Web is a huge network of interconnected machines. It is made up of web pages and web applications linked together by hyperlinks and URLs. The rules for how to transfer information between two machines on the web are defined by the Hypertext Transfer Protocol, commonly called HTTP.

HTTP defines the rules for how messages are sent and received between a client and a web server. The heart of this web interaction is the request-response cycle:

• The machine that initiates a request is called client
• The machine that responds to this request is called server

For example, when you type an address in your browser’s URL bar, it makes a request to the server you want to reach. The server processes this request and returns a response, which can be:

• The HTML markup of the site you want to see
• An error message
• Data in another format (JSON, XML, etc.) depending on server type

Application Programming Interfaces (APIs)

APIs (Application Programming Interfaces) are sets of endpoints available on a particular server. Unlike traditional URLs that you access in your browser, these routes are designed for machine-to-machine communication. They provide a structured way for software to request and exchange data.

Servers that serve websites typically return HTML so the browser knows how to render the site on your computer. However, native mobile apps do not use HTML, but still need access to the server. For example, social media apps need information about your profile — and this information is provided to them via JSON APIs.

Why the requests library?

The requests library allows you to make HTTP requests from Python code. The goal of these requests is to retrieve content from a web server programmatically. Since we are not making requests from a browser, the expected response should not be HTML, but another content type, such as JSON.

2.3 Configure Demo API Server

To test the functionality of the library, we need an API. Instead of depending on a third-party server that we cannot control, the instructor created a simple API server with the FastAPI framework.

Installing dependencies

Start by creating a virtual environment:

# Sur Unix/Linux/macOS
python3 -m venv venv
source venv/bin/activate

# Sur Windows
python -m venv venv
venv\Scripts\activate

Then install the dependencies:

pip install -r requirements.txt

File requirements.txt

annotated-types==0.6.0
anyio==3.7.1
certifi==2023.11.17
click==8.1.7
dnspython==2.4.2
email-validator==2.1.0.post1
fastapi==0.104.1
h11==0.14.0
httpcore==1.0.2
httptools==0.6.1
httpx==0.25.1
idna==3.4
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.3
orjson==3.9.10
pydantic==2.5.1
pydantic-extra-types==2.1.0
pydantic-settings==2.1.0
pydantic_core==2.14.3

Start server

python main.py

The server starts at http://127.0.0.1:8000. The interactive Swagger UI documentation is available at http://127.0.0.1:8000/docs.

Installing the requests library

pip install requests

2.4 Summary of Module 2

• Before watching the course, you should be familiar with the basics of Python and how the web works.
• This course is a Playbook course, which means that except for the first module, you do not have to watch the lessons in order.
• Communication rules between machines on the web are defined by HTTP.
• HTTP defines the request-response cycle in which the client machine requests a resource at a URL and the server that contains that resource ideally responds with the desired information.
• APIs provide a structured way for software to retrieve information.
• The requests library allows you to make HTTP requests programmatically from Python code.

3. Parse GET requests and Response object

Estimated duration: 17m 58s

3.1 Initiate GET requests

HTTP Methods

HTTP defines a set of request methods, also called HTTP verbs. You can choose a specific HTTP method depending on the action you want to perform on a resource.

The most common request method is GET. Every time you type an address into your browser, it initiates a GET request to that URL, because the main purpose of browsers is to present information.

Characteristics of GET requests:

• Mainly used to fetch data from a specific URL
• If you make a request to an API, you will usually get the response in JSON
• GET requests are idempotent: doing the same GET request multiple times will always produce the same result
• They are limited in data size: not suitable for sending large volumes of data to the server
• Since they do not modify the server state, they are generally cached and can be bookmarked

First GET script

import requests

response = requests.get("http://127.0.0.1:8000/api/items")
print(response)

The string representation of the response object is not very descriptive. It only tells us that we have received a response and what its status code is.

Output:

<Response [200]>

JSON — JavaScript Object Notation

JSON is a lightweight data interchange format that is easy for humans to read and write, and easy for machines to parse and generate. This is the most commonly used format for API responses.

3.2 Analyze the status code of the response

The status code is very important because it informs us if the request was successful. To get the status code specifically, you can use the status_code property.

HTTP status code categories

Control program flow with status code

import requests

response = requests.get("http://127.0.0.1:8000/something")
print(response.status_code)

# Méthode booléenne simplifiée (True si 200-299)
if response:
    print("Success!")

# Contrôle plus précis
if response.status_code == 200:
    print("Success!")
elif response.status_code == 500:
    print("Server error.")
elif response.status_code == 404:
    print("Page not Found.")

Note: The boolean method if response: will evaluate to True if the code is in the range 200-299, and False for any other code. The instructor prefers to check the exact status code to have more precise control.

3.3 Handle HTTP exceptions with raise_for_status

You can control the flow of the program by checking the status code. However, if you only want to check if the request was successful, you can use the response’s raise_for_status method.

If the status code indicates a client error (4xx) or a server error (5xx), this method will throw an HTTPError exception.

Error types

• Client errors (4xx): 400 Bad Request, 401 Unauthorized, 404 Not Found
• Server errors (5xx): 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable

import requests

try:
    response = requests.get("http://127.0.0.1:8000/something/")
    response.raise_for_status()
except requests.exceptions.HTTPError as http_err:
    print(f"HTTP error occurred: {http_err}")
except Exception as err:
    print(err)
else:
    print(response.status_code)

This method is particularly useful in larger applications where you want to implement consistent error handling across different sections of the code. The exception includes detailed information about the error, which is useful for debugging.

3.4 Inspect the Response object

The most important aspect of the response is the data it contains, also known as the payload. This is especially true for GET requests, because the entire purpose of a GET request is to retrieve data.

The content property — raw bytes

import requests

response = requests.get("http://127.0.0.1:8000/api/items")

# Payload en format bytes bruts
print(response.content)
# b'[{"name":"Foo","price":23.45},{"name":"Bar","price":67.89},...]'

# Bytes bruts en hexadécimal
print(response.content.hex())
# 5b7b226e616d65223a22466f6f222c2270726963...

The letter b at the beginning of the string indicates that what we are seeing is a series of bytes. Reading raw bytes is useful for downloading images, videos, PDF files, or other content that is not text.

The text property — character string

# Vérifier l'en-tête content-type
print(response.headers["content-type"])
# application/json

# Payload converti en chaîne de caractères
# response.encoding = "utf-8"  # optionnel, spécifier l'encodage
print(response.text)
# '[{"name":"Foo","price":23.45},{"name":"Bar","price":67.89},...]'

To convert the series of bytes into a real character string, we use the text property. If you do not specify the encoding, the requests library will first look at the Content-Type header of the response.

The json() method — automatic conversion to Python structures

# Convertir le payload en JSON (structures Python)
print(response.json())
# [{'name': 'Foo', 'price': 23.45}, {'name': 'Bar', 'price': 67.89}, ...]

# Accéder à un élément spécifique
print(response.json()[1]["name"])
# Bar

The json() method automatically converts the JSON string into Python compatible data structures (lists, dictionaries, etc.).

Full demo code

import requests

response = requests.get("http://127.0.0.1:8000/api/items")

# Payload en format bytes brut
# print(response.content)

# Bytes bruts en hexadécimal
# print(response.content.hex())

# Vérifier l'en-tête content-type
# print(response.headers["content-type"])

# Payload converti en chaîne de caractères
# response.encoding = "utf-8"
# print(response.text)

# Conversion du payload en JSON
# print(response.json())
print(response.json()[1]["name"])

3.5 Passing values ​​with query string parameters

GET requests are used to retrieve data, so their size is very limited, unlike other HTTP methods like POST. But you can still provide a small amount of data directly in the URL itself using query string parameters.

Anatomy of a URL with query parameters

http://127.0.0.1:8000/api/items?max_price=40&offset=2&limit=2

• ? separates base URL from parameters
• max_price=40 is a key-value parameter
• & separates parameters from each other
• offset=2 and limit=2 are other parameters

Warning: GET requests are not suitable for connection routes because the password would be visible in the URL.

Passing parameters with requests

Instead of constructing the URL manually, the requests library allows parameters to be passed in dictionary form via the params argument:

import requests

query_params = {"offset": 2, "limit": 2, "max_price": 40}
response = requests.get(
    "http://127.0.0.1:8000/api/items",
    params=query_params,
)
print(response.json())

The library will automatically construct the URL with the correctly encoded parameters. This is cleaner, more readable and avoids manual encoding errors.

3.6 Assign event hooks

The requests library implements a hooks system which can be useful in certain situations, such as setting up a custom logging system.

A hook is simply a callback function triggered by a specific event in the request process. Currently, the requests library only offers the response hook.

Using event hooks

import requests

def log_url(response, *args, **kwargs):
    print(f"Requested URL: {response.url}")

response = requests.get(
    "http://127.0.0.1:8000/api/items",
    hooks={"response": log_url}
)

• Dictionary value hooks is callback function
• The key is the name of the specific hook (“response”`)
• The response hook will pass the response object as the first argument to the callback function

Use case: detect redirects

Suppose you use the URL /api/items/ (with a trailing slash) while the server expects /api/items (without a slash). Thanks to the hook, you will see two URLs in the output, revealing that a redirect has taken place. This can lead to unexpected behavior with POST requests because the redirection could convert POST to GET.

Example output:

Requested URL: http://127.0.0.1:8000/api/items/
Requested URL: http://127.0.0.1:8000/api/items

Hooks are great for debugging purposes because you can access any part of the response and even intercept redirects.

3.7 Module 3 Summary

4. Send data to server

Estimated duration: 19m 28s

4.1 Submit data with POST requests

The original implementation of HTTP in 1999 only defined the GET request to retrieve data. However, as the internet grew and became more complex, there was a need to send more data with the query itself. The next version of HTTP introduced the POST method, which allowed more data to be submitted to the server, usually via web forms.

Characteristics of POST requests

• Allows sending data in the request body (request body)
• Unlike GET requests, data is not exposed in the URL
• More secure than GET requests for sending sensitive information
• Suitable for authentication (password sending)
• Non-idempotent: the same POST request made several times can create several resources
• Cannot be cached or used as favorites
• Can send any type of data: text, files in binary format, etc.

The same URL can have different behaviors depending on the HTTP method used. Handlers on the server are selected based on the HTTP method.

4.2 Working with form data

The most common type of data sent with a POST request is form data. When you submit something via an HTML form, you are actually sending form data to the server.

How HTML forms submit data

<form action="/items/new" method="post">
    <label for="name">Item Name:</label>
    <input type="text" id="name" name="name" required>

    <label for="price">Price:</label>
    <input type="number" id="price" name="price" step="0.01" required>

    <input type="submit" value="Submit">
</form>

• The method="post" attribute indicates that data will be sent in the request body
• The action attribute defines where the POST request will send the data
• The Content-Type header is automatically set to application/x-www-form-urlencoded

Send form data with requests

To send form data, you pass a dictionary to the data argument:

import requests

form_data = {
    "name": "NewItem",
    "price": 29.99
}

response = requests.post(
    "http://127.0.0.1:8000/items/new",
    data=form_data
)
print(response.status_code)

The requests library will automatically format the dictionary as form data with the appropriate Content-Type (application/x-www-form-urlencoded).

Equivalent curl example

curl -X POST "http://127.0.0.1:8000/api/items" \
     -H "Content-Type: application/x-www-form-urlencoded" \
     -d "name=NewItem&price=29.99"

4.3 Working with JSON data

If you want to submit data to an API, there is a good chance that the API will only accept data in JSON format.

Manual approach (not recommended)

import requests
import json

new_item = {"name": "NewItem", "price": 29.99}

# Mauvaise approche : json.dumps + Content-Type manuel
import json
response = requests.post(
    "http://127.0.0.1:8000/api/items",
    data=json.dumps(new_item),
    headers={"Content-Type": "application/json"}
)

Recommended approach with the json argument

Since JSON is so popular, the requests library offers a simpler way: use the json argument instead of data. This automatically converts the dictionary to JSON and sets the correct Content-Type.

import requests

new_item = {"name": "NewItem", "price": 29.99}

response = requests.post(
    "http://127.0.0.1:8000/api/items",
    json=new_item
)

# Vérifier que les headers sont correctement définis
print(response.request.headers)
# {'Content-Type': 'application/json', ...}

print(response.json())

The response.request property is a PreparedRequest object which provides access to the request object used to perform the request in question.

Header output:

{
    'Content-Type': 'application/json',
    'Content-Length': '34',
    'User-Agent': 'python-requests/2.31.0',
    ...
}

4.4 Working with XML and other data types

The raw body of a request can be used to send any data in any format. The requests library has built-in support for form data and JSON, but you can send other popular formats like XML.

Send XML data

import requests
import xml.etree.ElementTree as ET

xml_data = """
<item>
    <name>XMLItem</name>
    <price>19.99</price>
</item>
"""

response = requests.post(
    "http://127.0.0.1:8000/api/items/xml",
    data=xml_data,
    headers={"Content-Type": "application/xml"}
)

print(response.text)

Since the message is a string, it will not automatically be converted to form data. You must, however, manually set the Content-Type to application/xml.

Process XML response

The requests library does not support XML natively, but you can use the standard Python xml.etree.ElementTree library:

import requests
import xml.etree.ElementTree as ET

xml_data = """
<item>
    <name>XMLItem</name>
    <price>19.99</price>
</item>
"""

response = requests.post(
    "http://127.0.0.1:8000/api/items/xml",
    data=xml_data,
    headers={"Content-Type": "application/xml"}
)

# Parser la réponse XML
root = ET.fromstring(response.text)
name = root.find("name").text
price = root.find("price").text

print(f"Name: {name}, Price: {price}")

No matter what format you need to work with, you can use the requests library to get the response content and then process it with the appropriate tools.

4.5 Upload files with POST

The body of a POST request is not just for submitting text. You can also use it to upload files in binary format.

Upload a single file

import requests

with open("report.csv", "rb") as f:
    response = requests.post(
        "http://127.0.0.1:8000/upload-files",
        files={"file": ("report.csv", f, "text/csv")}
    )
print(response.json())

Upload multiple files simultaneously

import requests

# Bonne pratique : ouvrir les fichiers en mode binaire ('rb')
file1 = open("data1.csv", "rb")
file2 = open("data2.csv", "rb")

# Liste de tuples : (field_name, (filename, file_object, content_type))
files = [
    ("files", ("data1.csv", file1, "text/csv")),
    ("files", ("data2.csv", file2, "text/csv")),
]

response = requests.post(
    "http://127.0.0.1:8000/upload-files",
    files=files
)

print(response.json())
# {'uploaded_files': ['data1.csv', 'data2.csv']}

# Fermer les fichiers après utilisation
file1.close()
file2.close()

Use a context manager (recommended approach)

import requests

with open("data1.csv", "rb") as f1, open("data2.csv", "rb") as f2:
    files = [
        ("files", ("data1.csv", f1, "text/csv")),
        ("files", ("data2.csv", f2, "text/csv")),
    ]
    response = requests.post(
        "http://127.0.0.1:8000/upload-files",
        files=files
    )
    print(response.json())

Structure of a file tuple

Each file is represented by a tuple with the following structure:

(field_name, (filename, file_object, content_type))

• field_name: equivalent to the name attribute of an HTML <input> element
• filename: the suggested name of the file you provide to the server
• file_object: the open file object in Python
• content_type: the MIME type of the file (ex: text/csv, image/png)

4.6 Use other HTTP methods (PUT, PATCH, DELETE)

For historical HTML implementation reasons, web forms can only send data with the GET and POST methods. HTTP version 1.1 introduced additional HTTP methods: PATCH, PUT, and DELETE. APIs don’t have this limitation, so we can use any method.

HTTP Method Naming Convention

According to the HTTP specification: PUT should be used to update the entire resource, PATCH to update a specific part of the resource.

Use PUT to update an entire resource

import requests

# Mettre à jour complètement l'item avec id=1
updated_item = {"name": "UpdatedItem", "price": 99.99}
response = requests.put(
    "http://127.0.0.1:8000/api/items/1",
    json=updated_item
)
print(response.json())

Use PATCH to partially update

import requests

# Mettre à jour uniquement le prix de l'item avec id=1
partial_update = {"price": 49.99}
response = requests.patch(
    "http://127.0.0.1:8000/api/items/1",
    json=partial_update
)
print(response.json())

Use DELETE to delete a resource

import requests

# Supprimer l'item avec id=1
response = requests.delete("http://127.0.0.1:8000/api/items/1")
print(response.json())
# {'status': 'Item deleted', 'item': {'name': 'Bar', 'price': 67.89}}

Retrieve a single item with GET

import requests

# Récupérer l'item avec id=1
response = requests.get("http://127.0.0.1:8000/api/items/1")
print(response.json())
# {'name': 'Bar', 'price': 67.89}

4.7 Summary of Module 4

5. Providing and receiving additional data with headers

Estimated duration: 4m 58s

5.1 Analyze response headers

headers are a crucial part of the HTTP request-response cycle because they serve as metadata carrying additional information about the content being transferred. They provide context to the payload and ensure that the sender and receiver understand the type, size, and other characteristics of the data.

Inspect all response headers

import requests

response = requests.get(
    "http://127.0.0.1:8000/api/items",
)

print(response.headers)

Example output:

{
    'content-type': 'application/json',
    'content-length': '432',
    'date': 'Mon, 01 Jan 2024 10:00:00 GMT',
    'server': 'uvicorn'
}

Access a specific header

# Accéder au Content-Type header
print(response.headers['Content-Type'])
# application/json

# Les noms de headers sont insensibles à la casse (case-insensitive)
print(response.headers['content-type'])   # fonctionne
print(response.headers['CONTENT-TYPE'])   # fonctionne aussi
print(response.headers['Content-type'])   # fonctionne aussi

According to the HTTP specification, header names are case-insensitive.

Current headers

5.2 Customize request headers

Request headers serve as a way to provide additional information about the request, such as the type of content the client can handle, authentication information, and other preferences.

Define custom headers

import requests

custom_headers = {
    "Authorization": "Bearer ACCESS_TOKEN",
    "Accept": "application/json",
}

response = requests.get(
    "http://127.0.0.1:8000/api/items",
    headers=custom_headers
)

# Vérifier les headers de la requête envoyée
print(response.request.headers)

Output:

{
    'User-Agent': 'python-requests/2.31.0',
    'Accept-Encoding': 'gzip, deflate',
    'Accept': 'application/json',
    'Connection': 'keep-alive',
    'Authorization': 'Bearer ACCESS_TOKEN'
}

Explanation of custom headers

• Accept: indicates to the server that our client is waiting for the response in JSON format. If the server supports several formats, it will use the one specified in Accept.
• Authorization: contains a bearer token. This is a common pattern for providing JSON Web Tokens (JWT) for authorization purposes, where the token represents a secure way to identify the user or session.

Use cases for custom headers

• Cache-Control
• Cookie management
• Content encoding (Accept-Encoding)
• Preferred language (Accept-Language)
• Analytics and tracking
• Preload headers
• Authentication

5.3 Summary of Module 5

6. Persist connections with cookies and sessions

Estimated duration: 15m 57s

6.1 Send and receive cookies

Cookies play a crucial role in web functionality by allowing web servers to store stateful information, such as items added to a shopping cart or recording user activity. These are small pieces of data sent from a website and stored in the user’s web browser.

How cookies work

HTTP is a stateless protocol — each request is independent of the previous ones. Cookies provide a way to maintain user state between different pages and sessions.

Example of a connection flow with “Remember me”:

1. User makes a POST request to the login route by submitting their username and password
2. In response, the server sets a cookie in the browser via the Set-Cookie header. This cookie may contain a hashed user ID, with attributes like Max-Age (lifespan of the cookie) and Domain (domain to which the cookie belongs)
3. The browser saves this cookie in memory
4. Once the cookie is saved, each subsequent request to the same domain will include this cookie in the Cookie header of the request
5. The server can then identify the user and maintain their connection state using the cookie

Send cookie with request

import requests

custom_cookies = {"user_id": "2"}

response = requests.get(
    "http://127.0.0.1:8000/api/cookies",
    cookies=custom_cookies
)

Read response cookies

# Obtenir tous les cookies sous forme de dictionnaire
print(response.cookies.get_dict())
# {'user_id': '2'}

# Accéder à un cookie spécifique
print(response.cookies["user_id"])
# 2

The cookies property of the response returns a RequestsCookieJar object. The get_dict() method converts this cookie jar into a standard Python dictionary.

6.2 Persisting connections with sessions

sessions in the requests library simplify the process of maintaining state between multiple requests. They remember cookies and other necessary details, and they also use the underlying TCP connection for subsequent requests, which can improve performance.

Connection flow without session (problem)

import requests

credentials = {"username": "some_name", "password": "pass"}

# Login
login_response = requests.post(
    "http://127.0.0.1:8000/api/login",
    data=credentials
)

# Récupérer les cookies
login_cookies = login_response.cookies

# Accéder à la route protégée — vous devez passer les cookies manuellement
response = requests.get(
    "http://127.0.0.1:8000/protected",
    cookies=login_cookies
)
print(response.status_code)

This flow can become difficult to maintain once you start making more requests to different protected routes — you have to pass cookies manually with each request.

Connection flow with session (solution)

import requests

with requests.Session() as session:
    credentials = {"username": "some_name", "password": "pass"}

    # Login — la session stocke automatiquement les cookies retournés
    session.post("http://127.0.0.1:8000/api/login", data=credentials)

    # Accéder à la route protégée — les cookies sont envoyés automatiquement
    response = session.get("http://127.0.0.1:8000/protected")

    print("Protected route:")
    print(response.status_code)
    print(response.text)

Benefits of sessions:

• Automatic cookie management: cookies received in responses are automatically stored and returned with all subsequent requests
• TCP Connection Reuse: TCP connections are kept in a pool and reused, improving performance
• Persistent context: headers, authentication and other parameters can be set once at session level

Set default settings on a session

import requests

session = requests.Session()

# Définir des headers par défaut pour toutes les requêtes de cette session
session.headers.update({
    "Authorization": "Bearer TOKEN",
    "Accept": "application/json"
})

# Définir une authentification par défaut
session.auth = ("username", "password")

6.3 Retrying connections with Transport Adapters

The Transport Adapters in the requests library are a powerful feature for configuring and customizing how HTTP requests are handled. The official documentation states that Transport Adapters provide a mechanism for defining interaction methods for an HTTP service.

The HTTP adapter is always attached to a session object. It determines how the session interacts with the server, including settings like SSL version, retries, and other connection-related configurations.

Configure a retry policy

import logging
import requests
from requests.adapters import HTTPAdapter
from requests.exceptions import RetryError
from urllib3.util.retry import Retry

# Configuration du logging pour voir les tentatives
logging.basicConfig(level=logging.DEBUG)
requests_log = logging.getLogger("urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

session = requests.Session()

# Configurer la stratégie de retry
retries = Retry(
    total=3,                    # Nombre maximum de tentatives
    backoff_factor=0.1,         # Délai entre les tentatives (exponentiel)
    status_forcelist=[500],     # Status codes déclenchant un retry
    allowed_methods={"GET"}     # Méthodes HTTP pour lesquelles retry s'applique
)

# Monter l'adaptateur sur le domaine spécifique
session.mount("http://127.0.0.1", HTTPAdapter(max_retries=retries))

try:
    response = session.get("http://127.0.0.1:8000/flaky")
    print("Final response status:", response.status_code)
except RetryError:
    print("Maximum retries exceeded. Server is not available.")

Retry parameters

How backoff_factor works

With backoff_factor=0.1, the delays between retries will be:

• Attempt 1 → 0 seconds (immediate)
• Attempt 2 → 0.1 second
• Attempt 3 → 0.2 seconds

The formula is: {backoff_factor} * (2 ** (retry_number - 1))

Mount an adapter

# Pour toutes les requêtes HTTP
session.mount("http://", HTTPAdapter(max_retries=retries))

# Pour toutes les requêtes HTTPS
session.mount("https://", HTTPAdapter(max_retries=retries))

# Pour un domaine spécifique seulement
session.mount("http://127.0.0.1", HTTPAdapter(max_retries=retries))

6.4 Summary of module 6

7. Manage redirects and timeouts

Estimated duration: 7m 17s

7.1 Working with redirects and history

Redirects are a natural part of HTTP communication, often used to guide clients to a new resource location or to improve security. However, sometimes we don’t want to allow them because they can lead to unexpected behavior and security issues.

Default behavior

By default, the requests library automatically follows redirects:

import requests

response = requests.get("http://127.0.0.1:8000/old-route")

print(response.history)     # Liste des réponses de redirection
print(response.url)         # URL finale après redirection
print(response.status_code) # 200 (après redirection réussie)
print(response.text)        # Contenu de la destination finale

Output:

[<Response [307]>]
http://127.0.0.1:8000/new-route
200
{"message": "This is the new route!"}

Disable redirects

# Désactiver les redirections pour une requête GET
response = requests.get(
    "http://127.0.0.1:8000/old-route",
    allow_redirects=False
)
print(response.status_code)  # 307 (Temporary Redirect)

The HEAD method and redirects

The HEAD method is the only method that does not allow redirects by default:

# HEAD sans redirection (comportement par défaut)
response = requests.head("http://127.0.0.1:8000/old-route")
print(response.status_code)  # 307

# HEAD avec redirection activée
response = requests.head(
    "http://127.0.0.1:8000/old-route",
    allow_redirects=True
)
print(response.status_code)  # 200

The HEAD method is the same as GET but without the response body. It is useful for checking if a resource exists without downloading its content.

Inspect redirect history

The history property of the response returns a list of all Response objects created to complete the request. The answers are ordered from oldest to most recent.

response = requests.get("http://127.0.0.1:8000/old-route")

print(response.history)
# [<Response [307]>]

for r in response.history:
    print(f"Redirected from: {r.url} (status: {r.status_code})")

print(f"Final URL: {response.url}")

If your application requires increased security, you can always disable redirects and only enable them if you trust the server in question.

7.2 Configure timeouts

If you make a request to an unresponsive server, your code will hang indefinitely — this is the default behavior of the requests library. This is unacceptable for any production code, so it is strongly recommended to always set a timeout on each request.

The two types of timeouts

The requests library distinguishes between two types of timeouts:

Single timeout (both timeouts have the same value)

import requests

try:
    response = requests.get(
        "http://127.0.0.1:8000/slow-response",
        timeout=3  # 3 secondes pour les deux types
    )
    print(response.json())
except requests.exceptions.Timeout:
    print("A timeout error occurred.")

Separate timeout (tuple)

import requests

try:
    # connect_timeout=5s, read_timeout=3s
    response = requests.get(
        "http://127.0.0.1:8000/slow-response",
        timeout=(5, 3)
    )
    print(response.json())
except requests.exceptions.ConnectTimeout:
    print("The request failed to connect in the allotted time.")
except requests.exceptions.ReadTimeout:
    print("The server did not send any data in the allotted amount of time.")
except requests.exceptions.Timeout:
    print("A timeout error occurred.")

Complete demonstration with exception handling

import requests

try:
    # response = requests.get("http://127.0.0.1:8000/slow-response", timeout=(5, 3))
    response = requests.get("http://10.255.255.1", timeout=(5, 6))
    print(response.json())
except requests.exceptions.ConnectTimeout:
    print("The request failed to connect in the allotted time.")
except requests.exceptions.ReadTimeout:
    print("The server did not send any data in the allotted amount of time.")
except requests.exceptions.Timeout:
    print("A timeout error occurred.")

http://10.255.255.1 is a non-routable IP address that drops all incoming connections, useful for demonstrating connect timeout.

Exceptions related to timeouts

7.3 Summary of module 7

8. Authenticate requests for secure APIs

Estimated duration: 23m 26s

8.1 Use HTTP Basic Authentication

Authentication is an essential part of web development because it ensures that only authorized users can access sensitive data while maintaining system security and integrity.

The simplest form of authentication is HTTP Basic Authentication. When you try to access a protected route with this method, the server responds with the WWW-Authenticate header, which means you must provide a username and password encoded in the Authorization header of the request.

How HTTP Basic Auth Works

1. The client makes a request without credentials
2. The server responds with 401 Unauthorized and the header WWW-Authenticate: Basic realm="..."
3. The client encodes username:password in Base64 and sends it in the Authorization: Basic <encoded_credentials> header
4. The server decodes the credentials and checks their validity

Use HTTPBasicAuth

import requests
from requests.auth import HTTPBasicAuth

username = "username"
password = "pass"

response = requests.get(
    "http://127.0.0.1:8000/protected-endpoint",
    auth=HTTPBasicAuth(username, password)
)

print(response.text)
# {"message": "Welcome, authenticated user!"}

Shortcut with a tuple (equivalent)

Since Basic Authentication is the default authentication method, you can simply provide a tuple with the credentials:

import requests

response = requests.get(
    "http://127.0.0.1:8000/protected-endpoint",
    auth=("username", "pass")
)

print(response.text)

Other authentication types available

from requests.auth import HTTPDigestAuth, HTTPProxyAuth

# Digest Authentication
response = requests.get(url, auth=HTTPDigestAuth(username, password))

# Proxy Authentication
response = requests.get(url, auth=HTTPProxyAuth(username, password))

8.2 OAuth 2.0 and the requests-oauthlib library

OAuth is an open standard for access delegation, commonly used to grant websites or applications access to a user’s information on other websites without giving them the passwords. It provides a secure and efficient way for users to allow third-party applications to access their data stored on different services.

To implement it with the requests library, you will need to install the complementary library requests-oauthlib:

pip install requests-oauthlib

OAuth 1 vs OAuth 2

• OAuth 1: original version of the protocol
• OAuth 2: most recent major version, used by most sites and services

The actors of the OAuth 2 flow

The OAuth 2 flow in detail

1. L'utilisateur visite notre application (Event Planner)
2. Notre application redirige l'utilisateur vers le Consent Screen de Google
3. L'utilisateur accorde l'accès à son calendrier
4. Google redirige l'utilisateur vers notre Callback URL avec un Authorization Code
5. Notre application échange cet Authorization Code contre un Access Token
6. Notre application utilise cet Access Token pour accéder aux données du calendrier

The Consent Screen

The Consent Screen is the page displayed to the user asking if they wish to grant our application access to their resources on the third-party service. It clearly lists the requested access scopes.

Scopes

Scopes define exactly which resources our application requests access to. For example:

8.3 Leveraging OAuth 2.0 for Data Recovery

Registering the client application

Before using OAuth 2.0, you must register your client application with the authorization server (e.g. Google Cloud Console):

1. Create a project in Google Cloud Console
2. Go to “APIs & Services” → “Credentials”
3. Create OAuth Client ID credentials
4. Configure the Consent Screen with app name and contact information
5. Add the necessary scopes (eg: Google Calendar API)
6. Add your email address as a test user (for apps in test mode)
7. Retrieve the client_id and the client_secret

Full implementation with requests-oauthlib

from requests_oauthlib import OAuth2Session
import json

# Configuration de l'application
client_id = "your_client_id"
redirect_uri = "https://127.0.0.1:8000/callback"
authorization_base_url = "https://accounts.google.com/o/oauth2/auth"
token_url = "https://oauth2.googleapis.com/token"
scope = ["https://www.googleapis.com/auth/calendar"]

# Étape 1 : Créer une session OAuth2
oauth = OAuth2Session(client_id, redirect_uri=redirect_uri, scope=scope)

# Étape 2 : Générer l'URL d'autorisation et rediriger l'utilisateur
authorization_url, state = oauth.authorization_url(
    authorization_base_url, prompt="consent"
)
print("Please go here and authorize access:", authorization_url)

# Étape 3 : L'utilisateur autorise l'accès et Google le redirige vers
#           notre callback URL avec un authorization code
#           Dans une vraie app, cette étape serait gérée par le serveur web
redirect_response = input("Paste the full redirect URL here: ")

# Étape 4 : Échanger l'authorization code contre un access token
token = oauth.fetch_token(
    token_url,
    authorization_response=redirect_response,
    client_secret="your_client_secret",
)

# Étape 5 : Utiliser l'access token pour accéder aux ressources
response = oauth.get("https://www.googleapis.com/calendar/v3/users/me/calendarList")
data = response.json()

# Sauvegarder les données
filename = "data.json"
with open(filename, "w") as file:
    json.dump(data, file, indent=4)

Explanation of flow in code

1. OAuth2Session: creates an OAuth 2 session with the client application settings
2. oauth.authorization_url(): generates the Consent Screen URL to which to redirect the user
3. redirect_response: the full redirect URL containing the authorization code (provided by Google after the user consents)
4. oauth.fetch_token(): exchanges the authorization code for an access token
5. oauth.get(): performs authenticated requests with the access token

In production, the redirect_response retrieval step would be managed automatically by a web server which would receive the redirection request from Google.

OAuth 1 (for reference)

from requests_oauthlib import OAuth1Session

# OAuth 1 est plus simple à implémenter mais moins utilisé aujourd'hui
oauth = OAuth1Session(
    client_key="your_client_key",
    client_secret="your_client_secret",
    resource_owner_key="your_resource_token",
    resource_owner_secret="your_resource_secret"
)

response = oauth.get("https://api.example.com/resource")

8.4 Implement custom authentication methods

You can also implement your own authentication method. For example, if your company has an internal authentication system not supported by the requests library or its complementary libraries.

How to create a custom authentication class

To implement your own authentication, you must:

1. Import the AuthBase class
2. Create a class that extends AuthBase
3. Implement the __call__ method which modifies the request object before sending it

Example: Authentication using JWT (JSON Web Token)

Once the server sends you the token, you must include it in the Authorization header of the request in the form Bearer <token>.

import requests
from requests.auth import AuthBase

# Authorization: Bearer <token>

class JWTAuth(AuthBase):
    def __init__(self, token):
        self.token = token

    def __call__(self, request):
        request.headers["Authorization"] = f"Bearer {self.token}"
        return request

token = "abcde123"
response = requests.get(
    "http://127.0.0.1:8000/jwt-protected-route",
    auth=JWTAuth(token)
)
print(response.text)
# {"message": "Access to protected route granted"}

Explanation of the JWTAuth class

• __init__: initializes the class with the JWT received from the server. When creating a JWTAuth(token) instance, the token is stored in self.token.
• __call__: The requests library takes this instance and calls it before sending the request. This method has access to the request object and can modify it — here, it adds the Authorization header.

Benefits of a custom authentication class

• Separation of responsibilities: authentication logic is isolated in its own class
• Reusability: a single class can be used for all requests requiring this type of auth
• Testability: easier to test in insulation
• Maintainability: centralized modifications in case of change of auth protocol

8.5 Ensuring data privacy with HTTPS

SSL (Secure Sockets Layer) and its successor TLS (Transport Layer Security) are protocols for establishing secure connections on the Internet. You can see that the server uses TLS by the extra letter S after HTTP in the URL (HTTPS).

How the SSL/TLS certificate works

1. When you visit an HTTPS site, the server sends its SSL certificate to the client
2. The client checks if this certificate exists in the list of Certificate Authorities (CAs)
3. The certificate must be provided by a recognized CA, otherwise someone may be trying to impersonate the server
4. If the certificate is valid, client and server can communicate securely using private and public keys

The certified library

The requests library uses the certifi library to access CAs. This library uses the Mozilla Firefox browser certificate bundles.

Automatic check in requests

import requests

# La vérification SSL est automatique pour les URLs HTTPS
response = requests.get("https://api.weather.com/data")
# requests lève une SSLError si le certificat est invalide

Disable SSL verification (not recommended in production)

import requests

# ATTENTION : désactiver la vérification SSL est dangereux en production
response = requests.get(
    "https://api.example.com/data",
    verify=False  # Ne jamais faire cela en production !
)

Security Warning: Disabling SSL checking makes your application vulnerable to Man-in-the-Middle (MitM) attacks. Only do this in a development/test environment.

Provide your own CA certificate bundle

import requests

# Utile pour les environnements d'entreprise avec des certificats internes
response = requests.get(
    "https://internal.company.com/api",
    verify="/path/to/ca-bundle.crt"
)

Use cases for custom certificates

• Enterprise environments: Internal HTTPS without public certificate
• Self-signed certificates: for testing and development
• Internal Certificate Authorities: Internal Enterprise CA

8.6 Summary of module 8

9. Appendix: Demo FastAPI API Server

The course uses a local API server created with FastAPI. Here is the full server code (main.py) for reference.

Full code of main.py

from fastapi import (
    FastAPI,
    Form,
    Body,
    HTTPException,
    Response,
    UploadFile,
    File,
    Request,
    Depends,
    Cookie,
    Header,
    status
)
from fastapi.security import HTTPBasic, HTTPBasicCredentials
from fastapi.responses import HTMLResponse, RedirectResponse, JSONResponse
from jinja2 import Environment, select_autoescape, BaseLoader
from typing import Optional, List, Annotated
from pydantic import BaseModel
import xml.etree.ElementTree as ET
import uvicorn
import secrets
import random
import time

app = FastAPI()

new_item_template = """
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Create New Item</title>
</head>
<body>
    <h1>Create a New Item</h1>
    {% for message in flash_messages %}
    <div style="background: greenyellow">{{ message }}</div>
    {% endfor %}
    <form action="/items/new" method="post">
        <label for="name">Item Name:</label>
        <input type="text" id="name" name="name" required><br><br>
        <label for="price">Price:</label>
        <input type="number" id="price" name="price" step="0.01" required><br><br>
        <input type="submit" value="Submit">
    </form>
</body>
</html>
"""

flash_messages = []
user_id_hash = "0000"

jinja_env = Environment(loader=BaseLoader, autoescape=select_autoescape(["html"]))

items_db = [
    {"name": "Foo", "price": 23.45},
    {"name": "Bar", "price": 67.89},
    {"name": "Baz", "price": 12.34},
    {"name": "Qux", "price": 56.78},
    {"name": "Quux", "price": 45.67},
    {"name": "Corge", "price": 78.90},
    {"name": "Grault", "price": 90.12},
    {"name": "Garply", "price": 34.56},
    {"name": "Waldo", "price": 89.01},
    {"name": "Fred", "price": 67.23},
    {"name": "Plugh", "price": 45.89},
    {"name": "Xyzzy", "price": 23.78},
    {"name": "Thud", "price": 90.23},
]

class Item(BaseModel):
    name: Optional[str] = None
    price: Optional[float] = None

security = HTTPBasic()
USERNAME = 'username'
PASSWORD = 'pass'

# --- Routes GET ---

@app.get("/api/items")
async def read_items(
    offset: Optional[int] = None,
    limit: Optional[int] = None,
    max_price: Optional[float] = None,
):
    filtered_items = items_db
    if max_price is not None:
        filtered_items = [item for item in items_db if item["price"] <= max_price]
    if offset is None:
        offset = 0
    if limit is None:
        limit = len(filtered_items) - offset
    return filtered_items[offset: offset + limit]

@app.get("/api/items/{item_id}")
async def get_item(item_id: int):
    if 0 <= item_id < len(items_db):
        return items_db[item_id]
    raise HTTPException(status_code=404, detail="Item not found")

# --- Routes POST ---

@app.post("/api/items")
async def create_item(item: Item):
    items_db.append(item.model_dump())
    return item

@app.post("/api/items/xml")
async def create_item_xml(xml_body: str = Body(..., media_type="application/xml")):
    try:
        root = ET.fromstring(xml_body)
        name = root.find("name").text
        price = root.find("price").text
        item = {"name": name, "price": price}
        items_db.append(item)
        xml_response = f"<response><name>{name}</name><price>{price}</price></response>"
        return Response(content=xml_response, media_type="application/xml")
    except ET.ParseError:
        raise HTTPException(status_code=400, detail="Invalid XML")

@app.post("/upload-files")
async def upload_files(files: List[UploadFile] = File(...)):
    if not files:
        raise HTTPException(status_code=400, detail="No files provided")
    filenames = []
    for file in files:
        contents = await file.read()
        filenames.append(file.filename)
    return {"uploaded_files": filenames}

# --- Routes PUT / PATCH / DELETE ---

@app.put("/api/items/{item_id}")
async def update_item(item_id: int, item: Item):
    if 0 <= item_id < len(items_db):
        items_db[item_id] = item.dict()
        return items_db[item_id]
    raise HTTPException(status_code=404, detail="Item not found")

@app.patch("/api/items/{item_id}")
async def patch_item(item_id: int, item: Item):
    if 0 <= item_id < len(items_db):
        if item.name:
            items_db[item_id]["name"] = item.name
        if item.price:
            items_db[item_id]["price"] = item.price
        return items_db[item_id]
    raise HTTPException(status_code=404, detail="Item not found")

@app.delete("/api/items/{item_id}")
async def delete_item(item_id: int):
    if 0 <= item_id < len(items_db):
        deleted_item = items_db.pop(item_id)
        return {"status": "Item deleted", "item": deleted_item}
    raise HTTPException(status_code=404, detail="Item not found")

# --- Routes Cookies ---

@app.get("/api/cookies")
async def get_cookies(request: Request):
    response = JSONResponse(content={"message": "ok"})
    for cookie_name, cookie_value in request.cookies.items():
        response.set_cookie(key=cookie_name, value=cookie_value)
    return response

# --- Routes Authentication ---

@app.post("/api/login")
async def login(username: str = Form(...), password: str = Form(...)):
    if username == "some_name" and password == "pass":
        global user_id_hash
        user_id_hash = secrets.token_hex(16)
        content = {"message": "Login successful"}
        response = JSONResponse(content=content)
        response.set_cookie(key="user_id", value=user_id_hash)
        return response
    else:
        return {"message": "Invalid credentials"}

def verify_user_id(user_id: str = Cookie(None)):
    if user_id_hash != user_id:
        raise HTTPException(status_code=401, detail="Unauthorized")

@app.get("/protected")
async def protected_route(user_id_verified: str = Depends(verify_user_id)):
    return {"message": "You have access to this protected route"}

def get_current_user(credentials: HTTPBasicCredentials = Depends(security)):
    correct_username = secrets.compare_digest(credentials.username, USERNAME)
    correct_password = secrets.compare_digest(credentials.password, PASSWORD)
    if not (correct_username and correct_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect username or password",
            headers={"WWW-Authenticate": "Basic"},
        )
    return credentials

@app.get("/protected-endpoint")
def read_protected_route(user: HTTPBasicCredentials = Depends(get_current_user)):
    return {"message": "Welcome, authenticated user!"}

@app.get("/jwt-protected-route")
async def jwt_protected_route(authorization: str = Header(None)):
    if authorization and authorization.startswith("Bearer "):
        token = authorization.split(" ")[1]
        if token == "abcde123":
            return {"message": "Access to protected route granted"}
        else:
            raise HTTPException(status_code=401, detail="Invalid token")
    else:
        raise HTTPException(status_code=401, detail="Authorization header missing or invalid")

# --- Routes Simulation ---

@app.get("/flaky")
async def flaky_endpoint():
    if random.choice([True, False]):
        raise HTTPException(status_code=500, detail="Server Error")
    else:
        return {"message": "Success"}

@app.get("/slow-response")
async def slow_response():
    time.sleep(5)
    return {"message": "Response after delay"}

# --- Routes Redirection ---

@app.get("/old-route")
async def old_route():
    return RedirectResponse(url="/new-route")

@app.get("/new-route")
async def new_route():
    return {"message": "This is the new route!"}

# --- Formulaire HTML ---

@app.get("/items/new")
async def new_item_form():
    messages = list(flash_messages)
    flash_messages.clear()
    rendered_template = jinja_env.from_string(new_item_template).render(
        flash_messages=messages
    )
    return HTMLResponse(content=rendered_template)

@app.post("/items/new")
async def create_item_from_form(name: str = Form(...), price: float = Form(...)):
    item = {"name": name, "price": price}
    items_db.append(item)
    flash_messages.append(f"Item {name} added successfully!")
    return RedirectResponse(url="/items/new", status_code=303)

if __name__ == "__main__":
    uvicorn.run(app, port=8000)

Demo Server Endpoints

Query Parameters of /api/items

Search Terms

python · requests · playbook · foundations · data · analysis · engineering · analytics · headers · http · custom · response · authentication · flow · library · oauth · json · parameters · redirects · send · server · timeouts · cookies · get