Working with Files in Python 3

This training covers the most common and useful operations for manipulating files and folders in Python 3 from the standard library, without external dependencies.

Table of Contents

1. Course Overview
2. Finding Files

• 2.1 Introduction and overview
• 2.2 What you will learn
• 2.3 Demo: List a directory
• 2.4 Demo: String methods
• 2.5 Demo: Pattern matching with fnmatch
• 2.6 Demo: Advanced pattern matching
• 2.7 Demo: Pattern matching with glob
• 2.8 Module 2 Summary

3. Working with files and folders

• 3.1 Introduction and overview
• 3.2 Demo: Get file attributes
• 3.3 Demo: Traversing a directory
• 3.4 Demo: Copy files
• 3.5 Demo: Moving files
• 3.6 Demo: Renaming files
• 3.7 Demo: Delete files
• 3.8 Module 3 Summary

4. Archiving Files

• 4.1 Introduction and overview
• 4.2 Demo: Create a ZIP archive
• 4.3 Demo: Add files to a ZIP archive
• 4.4 Demo: Read a ZIP archive
• 4.5 Demo: Extract a ZIP archive
• 4.6 Module 4 Summary

5. Reading and Writing Files

• 5.1 Introduction and overview
• 5.2 Demo: Working with text files
• 5.3 Demo: Working with CSV files
• 5.4 Demo: Working with XML files
• 5.5 Demo: Working with JSON files
• 5.6 Demo: Persisting Python Objects (Pickle)
• 5.7 Summary and Final Thoughts

6. Structure of exercise files
7. Python modules used — quick reference

1. Course Overview

Manipulating files and folders is an essential aspect of working with Python. Whether you are a software developer, business automation specialist, or data engineer, you will constantly need to interact with the file system.

This course teaches you how to apply the most useful and common file and folder operations available in the standard Python library.

What you will learn

• Finding files: find files using string methods and pattern matching techniques.
• File and folder operations: copy, move, rename, delete, archive and traverse directories.
• Reading and writing: read and write text, CSV, XML, JSON and pickle files — among the most popular file formats.

Prerequisites

• Have basic programming skills.
• Have some proficiency in Python.
• This course is not a fundamental course: we do not start from scratch.

Python version

This course was created for Python 3.X. The demonstrations use Python 3.10, but the content is applicable to all Python 3 versions since it is entirely based on the standard library.

2. Finding Files

2.1 Introduction and Overview

This module covers all the approaches that Python offers for finding files in the file system:

1. List a directory with os.listdir()
2. String methods: endswith(), startswith()
3. Pattern matching with fnmatch
4. Advanced pattern matching with fnmatch and wildcards
5. Pattern matching with the glob module via pathlib.Path

Each module of this training is independent: you can follow them in the order you wish.

2.2 What you will learn

This course is of the playbook type: it is practical and focused on demos. Its goal is to give you hands-on exposure to the most useful file operations in Python.

This is not a fundamental course: you must already have a foundation in Python to get the most out of it. Each module is independent, giving you the freedom to learn at your own pace, in the order you want.

Here are the topics covered by this course as a whole:

2.3 Demo: List a directory

The first fundamental operation is to list the contents of a directory. Before you can manipulate files and folders, you need to know what a folder contains.

Python provides os.listdir() to get the list of files in a folder. This function returns a list of character strings representing the names of files and subfolders contained in the specified directory.

Working directory structure

demos/
├── files/
│   ├── subfolder/
│   │   ├── 01_file_test.csv
│   │   ├── 01_file_test.txt
│   │   ├── 01_test_file copy.txt
│   │   ├── 01_test_file.csv
│   │   └── 01_test_file.txt
│   ├── 01_file.csv
│   ├── 01_file.txt
│   ├── 01_file_test.csv
│   ├── 01_file_test.txt
│   ├── 01_test.csv
│   ├── 01_test.txt
│   ├── 01_test_file.csv
│   ├── 01_test_file.txt
│   ├── 02_file.csv
│   ├── 02_file.txt
│   ├── 02_file_test.csv
│   ├── 02_file_test.txt
│   ├── 02_test.csv
│   ├── 02_test.txt
│   ├── 02_test_file.csv
│   ├── 02_test_file.txt
│   └── text.txt
├── files_to_read/
│   ├── authors.json
│   ├── backup.py
│   ├── ef_author.xml
│   ├── example.txt
│   ├── names.csv
│   └── names2.csv
├── M2_Finding_Files/
├── M3_Working_with_Files_and_Folders/
├── M4_Archiving_Files/
├── M5_Reading_and_Writing_Files/
└── run.txt

Code — 01_Demo_Listing_a_Directory.py

import os

def list_dir(fld):
    for fn in os.listdir(fld):
        print(fn)

list_dir('./files')

Detailed explanation

• import os: we import the os module from the standard Python library. This module provides a portable interface for using operating system dependent features.
• os.listdir(fld): returns a list of all file names and subfolders contained in the fld path. The names returned are simple strings (without full path).
• The for fn in os.listdir(fld) loop iterates over each returned element and prints it with print(fn).

Note: os.listdir() does not distinguish between files and folders. It returns all elements of the directory indifferently. To filter, you will need to use os.path.isfile() or os.path.isdir().

Run script

py "./M2_Finding_Files/01_Demo_Listing_a_Directory.py"

2.4 Demo: String methods

After listing the contents of a directory, you can use native Python string methods to filter files according to simple criteria: the start or end of their name.

Python has two very useful string methods for this:

• str.endswith(suffix): returns True if the string ends with suffix.
• str.startswith(prefix): returns True if the string starts with prefix.

Code — 02_Demo_Using_String_Methods.py

import os

def ends_with(fld, search):
    for fn in os.listdir(fld):
        if fn.endswith(search):
            print(fn)

def starts_with(fld, search):
    for fn in os.listdir(fld):
        if fn.startswith(search):
            print(fn)

#ends_with('./files', '.txt')
starts_with('./files', '01_test')

Detailed explanation

Function ends_with

• We iterate over the list of files in fld with os.listdir(fld).
• For each fn file, we call fn.endswith(search).
• If the condition is true, the file name is displayed.
• Example: ends_with('./files', '.txt') will return all .txt files in the files directory.

Function starts_with

• Same logic, but with fn.startswith(search).
• Example: starts_with('./files', '01_test') returns all files whose name starts with 01_test.

Limits of string methods

The startswith and endswith methods are simple and effective for basic criteria, but they do not allow you to create complex search criteria involving several substrings simultaneously, or wildcards. This is why Python also offers modules dedicated to pattern matching.

Run script

py "./M2_Finding_Files/02_Demo_Using_String_Methods.py"

2.5 Demo: Pattern matching with fnmatch

The fnmatch module (Unix filename matching) provides a function fnmatch.fnmatch() which allows you to perform pattern matching on file names using Unix-style wildcards.

Wildcards supported by fnmatch

Code — 03_Demo_Pattern_Matching_with_fnmatch.py

import os, fnmatch

def match(fld, search):
    for fn in os.listdir(fld):
        if fnmatch.fnmatch(fn, search):
            print(fn)

match('./files', '*2*_test_.csv')

Detailed explanation

• We import both os and fnmatch.
• The match function takes the folder to inspect (fld) and the search criterion (search).
• For each file in the folder, we call fnmatch.fnmatch(fn, search).
• If the match is True, we display the file.
• The pattern '*2*_test_.csv' means: any name containing 2, followed by _test_, with the extension .csv.

Examples of fnmatch patterns

# Tous les fichiers .csv
match('./files', '*.csv')

# Tous les fichiers contenant '_file' et avec extension .csv
match('./files', '*_file*.csv')

# Tous les fichiers dont le nom commence par '01' et finit par .txt
match('./files', '01*.txt')

Advantage over string methods

With fnmatch, you can search for files by specifying more sophisticated patterns including multiple substrings and wildcards, which is not possible with endswith or startswith alone.

Run script

py "./M2_Finding_Files/03_Demo_Pattern_Matching_with_fnmatch.py"

2.6 Demo: Advanced pattern matching

This demo explores more complex fnmatch patterns, combining several wildcards to precisely refine the searched files.

Code — 04_Demo_Advanced_Pattern_Matching.py

import os, fnmatch

def match(fld, search):
    for fn in os.listdir(fld):
        if fnmatch.fnmatch(fn, search):
            print(fn)

#match('./files', '*_file*.*')
#match('./files', '*_file_*.*')
match('./files', '*2_*_*.*')

Detailed explanation of patterns

Pattern 1: '*_file*.*'

• *: any prefix.
• _file: must contain the substring _file.
• *: any suffix after _file.
• .*: any extension.
• Result: all files whose name contains _file, regardless of their extension (.txt, .csv, etc.).

Pattern 2: '*_file_*.*'

• Same, but with _file_ (with an underscore after file).
• There will be fewer results because the criterion is stricter.
• Only files containing _file_ (with underscore on either side) will be returned.

Pattern 3: '*2_*_*.*'

• Name must contain 2_, followed by any segment, followed by another _, then any extension.
• Example match: 02_file_test.csv, 02_test_file.txt.

Pattern progress

# Plus permissif → Plus restrictif
match('./files', '*_file*.*')    # Contient '_file'
match('./files', '*_file_*.*')   # Contient '_file_' (underscore des deux côtés)
match('./files', '*2_*_*.*')     # Contient '2_', puis '_', puis n'importe quelle extension

This progression illustrates how we can progressively refine a search criterion to isolate exactly the desired files.

Run script

py "./M2_Finding_Files/04_Demo_Advanced_Pattern_Matching.py"

2.7 Demo: Pattern matching with glob

The glob module (via pathlib.Path) offers a different approach to pattern matching. Rather than first listing all files and then filtering, it directly applies the search criteria to the path of the folder to be inspected.

pathlib is an object-oriented library for manipulating file paths. The Path.glob() method returns a generator of all paths matching the pattern.

Code — 05_Demo_Pattern_Matching_with_glob.py

from pathlib import Path

def glob_match(fld, search):
    p = Path(fld)
    for n in p.glob(search):
        print(n)

#glob_match('./files', '*2*.t*')
glob_match('./files/subfolder', '*1_t*file_*c*.t*')

Detailed explanation

• from pathlib import Path: we import the Path class from the pathlib module.
• p = Path(fld): we create a Path object representing the folder to inspect.
• p.glob(search): returns a generator (iterable) of all paths matching the search pattern.
• Unlike fnmatch, the results returned are full Path objects (with folder path included), not just file names.

Key differences between glob and fnmatch

Recursive glob with **

pathlib.Path.glob() also supports the ** pattern for recursive search in subfolders:

# Trouver tous les fichiers .csv dans le dossier et tous ses sous-dossiers
glob_match('./files', '**/*.csv')

Advanced pattern example

The pattern '*1_t*file_*c*.t*' means:

• *1_t: any prefix, containing 1_t.
• *file_: followed by file_ with any prefix.
• *c*: containing c.
• .t*: extension starting with t (.txt, .tsv, etc.).

Run script

py "./M2_Finding_Files/05_Demo_Pattern_Matching_with_glob.py"

2.8 Module 2 Summary

In this module, we covered five approaches to finding files with Python:

In the next module, we will explore in detail how to work with files and folders and perform common operations.

3. Working with files and folders

3.1 Introduction and Overview

This module covers the most important operations on files and folders, which constitute the heart of daily business operations:

1. Get file attributes: read metadata (modification date, size, etc.)
2. Cross a directory: retrieve all subfolders and files in a tree
3. Copy files: copy individual files or entire folders
4. Move files: move or rename via move
5. Renaming files: rename with os.rename() or pathlib.Path.rename()
6. Delete Files: Safely Delete with Precheck

3.2 Demo: Get file attributes

Python allows you to read the attributes (metadata) of any file: creation date, modification date, size, and more. This is very useful for inventory, backup or audit scripts.

Code — 01_Demo_Getting_File_Attributes.py

import os
from datetime import datetime

def get_date(timestmp):
    return datetime.utcfromtimestamp(timestmp).strftime('%d %b %Y')

def get_file_attrs(fld):
    with os.scandir(fld) as dir:
        for f in dir:
            if f.is_file():
                inf = f.stat()
                print(f'Modified {get_date(inf.st_mtime)} {f.name}')

get_file_attrs('./files/subfolder')

Detailed explanation

Function get_date

• datetime.utcfromtimestamp(timestmp): converts a Unix timestamp (number of seconds since January 1, 1970 UTC) to a datetime object.
• .strftime('%d %b %Y'): formats the date into a readable string (e.g.: 15 Jun 2024).
• This function is used to display the modification date in a human-readable format.

Function get_file_attrs

• os.scandir(fld): similar to os.listdir(), but returns DirEntry objects that contain both the file name and its attributes. This is more efficient because the stat information can be obtained directly from the DirEntry object without additional system calls.
• f.is_file(): checks that the current entry is indeed a file (and not a folder).
• f.stat(): returns an os.stat_result object containing all the attributes of the file.
• inf.st_mtime: timestamp of the last modification of the file (in seconds since the Unix epoch).
• f.name: file name without path.

Attributes available via stat()

Run script

py "./M3_Working_with_Files_and_Folders/01_Demo_Getting_File_Attributes.py"

3.3 Demo: Traversing a directory

Traversing (or traversing) a directory means recovering the entire tree: all subfolders and all the files they contain, regardless of the nesting depth.

Code — 02_Demo_Traversing_a_Directory.py

import os

def traverse(fld):
    for fldpath, dirs, fls in os.walk(fld):
        print(f'Folder: {fldpath}')
        for fn in fls:
            print(f'\t{fn}')

traverse('./files')

Detailed explanation

• os.walk(fld): This is the key function here. It recursively traverses the entire tree starting from the fld folder.
• At each iteration, it returns a triple (fldpath, dirs, fls):
• fldpath: the path of the current folder.
• dirs: list of subfolders in fldpath.
• fls: list of files in fldpath.
• We first display the path of the current folder (fldpath), then for each file in this folder, we display it with an indentation (\t).

Example output

Folder: ./files
    01_file.csv
    01_file.txt
    01_file_test.csv
    ...
    text.txt
Folder: ./files/subfolder
    01_file_test.csv
    01_file_test.txt
    ...

os.walk() versus os.listdir()

With only four lines of code, os.walk() allows you to traverse an entire directory tree, regardless of its depth.

Run script

py "./M3_Working_with_Files_and_Folders/02_Demo_Traversing_a_Directory.py"

3.4 Demo: Copy files

The shutil (shell utilities) module provides high-level functions for copying files and folders. It is much more convenient than manipulating files manually by reading and rewriting their contents.

Code — 03_Demo_Copying_Files.py

import shutil

def copy_file(src, dst):
    shutil.copy(src, dst)

def copy_folder(src, dst):
    shutil.copytree(src, dst)

#copy_file('./files/02_file.txt', './files/subfolder')
copy_folder('./files', './files/new_flder')

Detailed explanation

Function copy_file

• shutil.copy(src, dst): copies the source file src to the destination dst.
• If dst is a folder, the file is copied inside with its original name.
• If dst is a file path, the file is copied under this new name.
• Copies the contents and permissions of the file, but not the metadata (dates).

Function copy_folder

• shutil.copytree(src, dst): copies recursively the folder tree from src to dst.
• dst must not exist before the call — copytree creates it automatically.
• Copies all files and subfolders.

Other copy functions in shutil

Run script

py "./M3_Working_with_Files_and_Folders/03_Demo_Copying_Files.py"

3.5 Demo: Moving files

shutil.move() allows you to move an entire file or folder. If the destination is on the same file system, the move is carried out by a simple (very quick) rename. Otherwise, the file is copied and then deleted at the source.

Code — 04_Demo_Moving_Files.py

import shutil

def move_files(src, dst):
    shutil.move(src, dst)

#mv_files('./files/text.txt', './files/subfolder/text.txt')
#mv_files('./files', './xyz')
#mv_files('./xyz', './files')

Detailed explanation

• shutil.move(src, dst): moves src to dst.
• If dst is an existing folder, src is moved inside.
• If dst does not exist, src is renamed to dst (move + rename simultaneously).
• Works for both files and entire folders.

Illustrated use cases

Move individual file:

move_files('./files/text.txt', './files/subfolder/text.txt')
# text.txt est déplacé dans le sous-dossier avec le même nom

Move an entire folder (including renaming):

move_files('./files', './xyz')
# Le dossier 'files' est déplacé et renommé 'xyz'

move_files('./xyz', './files')
# Remet le dossier à son emplacement d'origine

Important: shutil.move() may overwrite an existing file at the destination without warning. Check the destination before traveling if necessary.

Run script

py "./M3_Working_with_Files_and_Folders/04_Demo_Moving_Files.py"

3.6 Demo: Renaming files

Python offers two ways to rename a file: via the os module or via pathlib. The two approaches are equivalent.

Code — 05_Demo_Renaming_Files.py

import os
from pathlib import Path

def rename_file_1(src, dst):
    os.rename(src, dst)

def rename_file_2(src, dst):
    f = Path(src)
    f.rename(dst)

#rename_file_1('./files/text.txt', './files/test.txt')
rename_file_1('./files/test.txt', './files/text.txt')

Detailed explanation

Method 1: os.rename(src, dst)

• Rename src file or folder to dst.
• If dst already exists, behavior depends on the operating system:
• On Unix/Linux: dst is silently replaced.
• On Windows: a FileExistsError error is raised.

Method 2: Path.rename(dst)

• f = Path(src): creates a Path object representing the source file.
• f.rename(dst): renames the file.
• Same behavior as os.rename().
• Returns a new Path object pointing to the destination.

When to use one or the other?

• os.rename(): classic procedural approach, familiar to those coming from other languages.
• Path.rename(): object-oriented approach, consistent with the use of pathlib in the rest of the code.

Run script

py "./M3_Working_with_Files_and_Folders/05_Demo_Renaming_Files.py"

3.7 Demo: Delete files

Deletion is an irreversible operation. It is therefore important to verify that the file exists before attempting to delete it, and to handle potential errors.

Code — 06_Demo_Deleting_Files.py

from genericpath import isfile
import os

def remove_file(f):
    if os.path.isfile(f):
        try:
            os.remove(f)
        except OSError as e:
            print(f'Error: {f} : {e.strerror}')
    else:
        print(f'Error: {f} is not a valid file')

remove_file('./files/text.txt')

Detailed explanation

• os.path.isfile(f): returns True if f is an existing file (and not a broken folder or symbolic link).
• This preliminary check is essential to avoid errors during deletion.
• os.remove(f): removes file f.
• This function cannot delete folders. To delete an empty folder, use os.rmdir(). For a non-empty folder, use shutil.rmtree().
• except OSError as e: captures any system errors during deletion (locked file, insufficient permissions, etc.).
• e.strerror: human-readable error message.

Behavior in case of error

# Si le fichier n'existe pas :
Error: ./files/text.txt is not a valid file

# Si une erreur OS se produit :
Error: ./files/text.txt : Permission denied

Delete functions in os and shutil

Warning: shutil.rmtree() deletes everything without confirmation. Use it with caution.

Run script

py "./M3_Working_with_Files_and_Folders/06_Demo_Deleting_Files.py"

3.8 Module 3 Summary

In this module, we explored fundamental file and folder operations:

In the next module we will learn how to work with archives and ZIP files.

4. Archiving Files

4.1 Introduction and Overview

An archive is a collection of compressed files that together take up less space than if the files were not grouped together. The most common archiving and compression format in the world is the ZIP format. The term “ZIP file” has become synonymous with archiving, and these two terms will be used interchangeably.

This module covers:

1. Create a ZIP archive
2. Add files to an existing ZIP archive
3. Read the contents of a ZIP archive
4. Extract files from a ZIP archive

Python provides the zipfile module in its standard library for all these operations.

4.2 Demo: Create a ZIP archive

Code — 01_Demo_Creating_a_ZIP_Archive.py

import zipfile

to_zip = ['./files/subfolder/01_file_test.csv', 
    './files/subfolder/01_file_test.txt', 
    './files/subfolder/01_test_file.csv', 
    './files/subfolder/01_test_file.txt',
    './files/01_file_test.csv',
    './files/01_file_test.txt']

def create_zip(zipf, files, opt):
    with zipfile.ZipFile(zipf, opt, allowZip64=True) as archive:
        for f in files:
            archive.write(f)

create_zip('./files.zip', to_zip, 'w')

Detailed explanation

• to_zip: list of files to compress. It contains CSV and TXT files from both the files folder and its subfolder subfolder.

Function create_zip

• zipfile.ZipFile(zipf, opt, allowZip64=True): opens (or creates) a ZIP file.
• zipf: name of the resulting ZIP file.
• opt: opening mode. Here 'w' for write — creates a new ZIP or overwrites the existing one.
• allowZip64=True: allows the creation of ZIP files larger than 2 GB (ZIP64 format). Recommended for large archives.
• with ... as archive: use of context manager — the file is automatically closed at the end of the block, even in the event of an exception.
• archive.write(f): adds file f to the archive keeping the relative path as the name in the ZIP.

ZipFile opening modes

Run script

py "./M4_Archiving_Files/01_Demo_Creating_a_ZIP_Archive.py"

4.3 Demo: Add files to a ZIP archive

To add files to an existing ZIP archive without overwriting its contents, use the 'a' (append) mode. We also check if the file to be added does not already appear in the ZIP to avoid duplicates.

Code — 02_Demo_Adding_Files_to_a_ZIP_Archive.py

import zipfile

to_add = ['files/01_file.csv',
    'files/01_file.txt']

def add_to_zip(zipf, files, opt):
    with zipfile.ZipFile(zipf, opt) as archive:
        for f in files:
            lst = archive.namelist()
            if not f in lst:
                archive.write(f)
            else:
                print(f'File exists in zip: {f}')

add_to_zip('./files.zip', to_add, 'a')

Detailed explanation

• archive.namelist(): returns the list of names of all files contained in the ZIP archive. This list is used to check if a file is already present.
• if not f in lst: if the file f is not in the list, we add it with archive.write(f).
• else: if the file already exists in the ZIP, we display a warning message instead of overwriting.

Best practice: always check with namelist() before adding a file to avoid duplicates and potentially corrupt the archive.

Run script

py "./M4_Archiving_Files/02_Demo_Adding_Files_to_a_ZIP_Archive.py"

4.4 Demo: Read a ZIP archive

Python allows you to read the contents of a ZIP archive — listing the files it contains and recovering their metadata — without having to extract them.

Code — 03_Demo_Reading_a_ZIP_Archive.py

import zipfile

def read_zip(zipf):
    with zipfile.ZipFile(zipf, 'r') as archive:
        lst = archive.namelist()
        for l in lst:
            zfinf = archive.getinfo(l)
            print(f'{l} => {zfinf.file_size} bytes, {zfinf.compress_size} compressed')

read_zip('./files.zip')

Detailed explanation

• zipfile.ZipFile(zipf, 'r'): opens the archive in read-only mode.
• archive.namelist(): returns the list of all file names in the ZIP.
• archive.getinfo(l): returns a ZipInfo object containing the metadata of the l file in the archive:
• zfinf.file_size: original file size in bytes.
• zfinf.compress_size: compressed size of the file in bytes.

Information available via ZipInfo

Example output

files/subfolder/01_file_test.csv => 1024 bytes, 512 compressed
files/subfolder/01_file_test.txt => 2048 bytes, 890 compressed
files/01_file.csv => 768 bytes, 400 compressed
...

Run script

py "./M4_Archiving_Files/03_Demo_Reading_a_ZIP_Archive.py"

4.5 Demo: Extract a ZIP archive

Python allows you to extract either a single file or all files from a ZIP archive.

Code — 04_Demo_Extracting_a_ZIP_Archive.py

import zipfile

def extract_file(zipf, fn, path):
    with zipfile.ZipFile(zipf, 'r') as archive:
        archive.extract(fn, path=path)

def extract_all(zipf, path):
    with zipfile.ZipFile(zipf, 'r') as archive:
        archive.extractall(path=path)

#extract_file('./files.zip', 'files/01_file_test.txt', 'extracted')
extract_all('./files.zip', 'extracted')

Detailed explanation

Function extract_file

• archive.extract(fn, path=path): extract a single file fn from the archive to the path directory.
• fn must match exactly a name returned by namelist().
• If path does not exist, it is created automatically.
• The folder tree of the file in the ZIP is reproduced at the destination.

Function extract_all

• archive.extractall(path=path): extract all files from the archive to path.
• The entire ZIP internal folder tree is reproduced.
• If path does not exist, it is created automatically.

Run script

py "./M4_Archiving_Files/04_Demo_Extracting_a_ZIP_Archive.py"

4.6 Summary of Module 4

In this module, we learned how to work with ZIP archives via the zipfile module:

In the next module, we will learn how to read and write different common file types.

5. Reading and Writing Files

5.1 Introduction and Overview

This module focuses on reading and writing the most common file types in Python:

1. Text files (TXT): read completely, line by line, write, add
2. CSV files (Comma-Separated Values): reading and writing tabular data
3. XML files (eXtensible Markup Language): reading, adding and modifying elements
4. JSON files (JavaScript Object Notation): reading, displaying and updating
5. Object persistence via the pickle module: binary serialization/deserialization

At the end of this module, you will know how to work with the most common file formats in Python applications.

5.2 Demo: Working with text files

Text files are the easiest to handle. Python provides open(), read(), readlines(), readline(), write() and writelines() functions to interact with them.

Data file: files_to_read/example.txt

this is a test...
this is an extra line

Code — 01_Demo_Working_with_Text_Files.py

def read_txt(fn):
    with open(fn) as f:
        print(f.read())

def read_txt_by_line(fn):
    with open(fn) as f:
        lines = f.readlines()
        for line in lines:
            print(line, end='')
            line = f.readline()

def write_new_txt(fn, str):
    with open(fn, 'w', encoding='utf-8') as f:
        f.write(str)

def append_line_txt(fn, str):
    with open(fn, 'a', encoding='utf-8') as f:
        f.write('\n')
        f.write(str)

#read_txt('./files_to_read/backup.py')
#read_txt_by_line('./files_to_read/backup.py')
#write_new_txt('./files_to_read/example.txt', 'this is a test...')
append_line_txt('./files_to_read/example.txt', 'this is an extra line')

Detailed explanation

Function read_txt — Complete reading

• open(fn): opens the file in reading mode (default). Returns an f file object.
• f.read(): reads all contents of the file at once and returns a character string.
• with ... as f: context manager — automatically closes the file at the end of the block.
• Advantage: simple and fast for small files.
• Disadvantage: loads the entire file into memory — avoid for very large files.

Function read_txt_by_line — Read line by line

• f.readlines(): reads all lines of the file and returns a list of strings, each element being a line (including the \n character).
• print(line, end=''): print each line without adding an extra line break (since \n is already included in line).
• f.readline(): reads and returns a single line of the file on each call. Used here as a complement in the loop.

Function write_new_txt — Writing

• open(fn, 'w', encoding='utf-8'): opens the file in write mode ('w').
• If the file does not exist, it is created.
• If the file exists, its contents are completely replaced.
• encoding='utf-8': Specifies the encoding to use — recommended for special characters.
• f.write(str): writes the string str to the file.

Function append_line_txt — Add at end of file

• open(fn, 'a', encoding='utf-8'): opens the file in append mode ('a').
• If the file does not exist, it is created.
• If the file exists, the cursor is positioned at the end of the file — existing content is preserved.
• f.write('\n'): first add a newline.
• f.write(str): then writes the new line.

File opening modes

Run script

py "./M5_Reading_and_Writing_Files/01_Demo_Working_with_Text_Files.py"

5.3 Demo: Working with CSV files

The CSV (Comma-Separated Values) format is omnipresent in tabular data exchanges. Python provides the csv module to read and write it robustly.

Data file: files_to_read/names.csv

name,lastname,age,sex
John,Smith,41,male
Joe,Clark,25,male

Code — 02_Demo_Working_with_CSV_Files.py

import csv

def read_csv(fn, delimiter):
    with open(fn) as csv_f:
        cnt = -1
        rows = csv.reader(csv_f, delimiter=delimiter)
        for r in rows:
            if cnt == -1:
                print(f'{" | ".join(r)}')
            else:
                print(f'{r[0]} | {r[1]} | {r[2]} | {r[3]}')
            cnt += 1
        print(f'{cnt} lines')

def write_csv(fn, header, row):
    with open(fn, mode='w', newline='') as csv_f:
        writer = csv.writer(csv_f, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
        writer.writerow(header)
        writer.writerow(row)
        
#read_csv('./files_to_read/names.csv', ',')
write_csv('./files_to_read/names2.csv', 
['name', 'lastname', 'age', 'sex'], 
['Foo', 'Fighter', '82', 'male'])

Detailed explanation

Function read_csv

• open(fn): opens the CSV file.
• csv.reader(csv_f, delimiter=delimiter): creates a reader object that iterates over the lines of the CSV file, separating each line by the specified delimiter (here, the comma ,).
• cnt = -1: counter initialized to -1 to distinguish the header line (cnt == -1) from the data lines.
• Display header: " | ".join(r) joins all elements of the first line with |.
• Data display: access by index r[0], r[1], r[2], r[3] for each column.
• cnt += 1: we increment the counter on each line.

Function write_csv

• open(fn, mode='w', newline=''): opens the file for writing with newline='' — important to avoid double empty lines on Windows (management of line endings).
• csv.writer(csv_f, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL): creates a writer object.
• delimiter=',': column separator.
• quotechar='"': character used for quotes.
• quoting=csv.QUOTE_MINIMAL: only puts quotes if necessary (if the value contains the delimiter or quotechar).
• writer.writerow(header): writes the header line.
• writer.writerow(row): writes a row of data.

Resulting file: files_to_read/names2.csv

name,lastname,age,sex
Foo,Fighter,82,male

Quoting options

Run script

py "./M5_Reading_and_Writing_Files/02_Demo_Working_with_CSV_Files.py"

5.4 Demo: Working with XML files

XML (eXtensible Markup Language) is a structured data format widely used in enterprise systems, web services (SOAP, REST XML) and configuration files. Python provides the xml.etree.ElementTree module to manipulate it.

Data file: files_to_read/ef_author.xml

<author name="Eduardo Freitas">
    <domain name="Azure" />
    <domain name="AWS" />
    <domain name=".NET" />
    <domain name="Python" />
    <domain name="TypeScript" />
</author>

Code — 03_Demo_Working_with_XML_Files.py

import xml.etree.ElementTree as ET

def parse_xml_et(fn):
    tree = ET.parse(fn)
    root = tree.getroot()
    print('Domains for: ' + root.attrib['name'])
    for child in root:
        print('\t' + child.attrib['name'], child.tag)

def add_xml_element_et(fn, el, attr, val):
    tree = ET.parse(fn)
    root = tree.getroot()
    child = ET.Element(el)
    child.attrib[attr] = val
    root.append(child)
    tree.write(fn)

def change_xml_element_et(fn, el, attr, oldval, newval):
    tree = ET.parse(fn)
    root = tree.getroot()
    child = root.find("./" + el + "[@" + attr + "='" + oldval + "']")
    child.attrib[attr] = newval
    tree.write(fn)

#parse_xml_et('./files_to_read/ef_author.xml')
#add_xml_element_et('./files_to_read/ef_author.xml', 'domain', 'name', 'Java')

change_xml_element_et('./files_to_read/ef_author.xml', 'domain', 'name', 'Java', 'TypeScript')

Detailed explanation

Function parse_xml_et — Reading

• ET.parse(fn): parses the XML file and returns an ElementTree object representing the XML tree.
• tree.getroot(): returns the root element of the XML tree (here <author>).
• root.attrib['name']: accesses the value of the name attribute of the root element.
• for child in root: iterates over all direct children of the root element.
• child.attrib['name']: accesses the child’s name attribute.
• child.tag: returns the name of the child’s tag (here domain).

Function add_xml_element_et — Adding an element

• ET.Element(el): creates a new XML element with the tag name el.
• child.attrib[attr] = val: sets an attribute on the new element.
• root.append(child): adds the new element as a child of the root.
• tree.write(fn): rewrites the modified XML tree to the file.

Function change_xml_element_et — Modification of an element

• root.find("./" + el + "[@" + attr + "='" + oldval + "']"): Uses an XPath expression to find an element.
• ./domain[@name='Java']: searches for a direct child of type domain whose name attribute is 'Java'.
• child.attrib[attr] = newval: updates the attribute value.
• tree.write(fn): save changes.

XPath expressions supported by ElementTree

Run script

py "./M5_Reading_and_Writing_Files/03_Demo_Working_with_XML_Files.py"

5.5 Demo: Working with JSON files

JSON (JavaScript Object Notation) is the most used data exchange format on the web. Python provides json module to easily read and write it.

Data file: files_to_read/authors.json

{"authors": 
    [
        {"name": "John Doe", 
            "courses": 10}, 
        {"name": "Jane Smith", 
            "courses": 10}, 
        {"name": "Foo Fighter", 
            "courses": 5}
    ]
}

Code — 04_Demo_Working_with_JSON_Files.py

import json

def read_print_json(fn, pretty, sort):
    with open(fn) as json_file:
        data = json.load(json_file)
        print(json.dumps(data, sort_keys=sort, indent=4) 
        if pretty else data)

def update_author_json(fn, arr_name, pos, key, value):
    with open(fn, 'r') as read_file:
        data = json.load(read_file)
        data[arr_name][pos][key] = value
        with open(fn, 'w') as write_file:
            json.dump(data, write_file)

#read_print_json('./files_to_read/authors.json', True, True)
update_author_json(
    './files_to_read/authors.json', 
    'authors', 1, 'courses', 10)

Detailed explanation

Function read_print_json — Reading and display

• open(fn): opens the JSON file.
• json.load(json_file): deserialize the JSON content of the file into a Python object (dict, list, str, int, float, bool, None).
• json.dumps(data, sort_keys=sort, indent=4): serialize the Python object data into a formatted JSON string.
• sort_keys=True: sorts keys alphabetically.
• indent=4: indentation of 4 spaces for readable display (pretty-print).
• Ternary expression: json.dumps(...) if pretty else data — if pretty is True, we display the formatted JSON; otherwise, we display the raw Python representation.

Function update_author_json — Update

• Open for reading ('r'): load JSON into memory with json.load().
• data[arr_name][pos][key] = value: accesses the element at position pos in array arr_name and modifies the value of key key.
• Example: data['authors'][1]['courses'] = 10 updates the courses field of the second author.
• Open for write ('w'): rewrites the modified JSON with json.dump(data, write_file).

Note: json.dump() writes directly to a file, while json.dumps() returns a string.

Key functions of the json module

Matching Python ↔ JSON types

Run script

py "./M5_Reading_and_Writing_Files/04_Demo_Working_with_JSON_Files.py"

5.6 Demo: Persisting Python Objects (Pickle)

The pickle module allows you to serialize (convert to binary) and deserialize (reconstruct from binary) complex Python objects. This is useful for saving the internal state of an application, transmitting it over the network, or storing it in a database.

Why persist objects?

As a developer, it may be necessary to:

• Save the internal state of an application to disk.
• Transmit complex objects over the network.
• Store objects in a database.

The pickle module solves these needs by allowing any Python object to be serialized into a binary representation.

Code — 05_Demo_Persisting_Objects.py

import pickle

class Person:
    age = 45
    name = 'John Smith'
    kids = ['Pete', 'Lilly', 'Kate']
    employers = {'AWS': 2022, 'Microsoft': 2018, 'Yahoo': 2005}
    shoe_sizes = (11, 12)

def serialize(obj):
    pickled = pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL)
    print(f'Serialized object: \n{pickled}\n')
    return pickled

def deserialize(obj):
    unpickled = pickle.loads(obj)
    print(f'Deserialized: \n{unpickled}\n')

def deserialize_employers(obj):
    unpickled = pickle.loads(obj)
    print(f'Deserialized employers: \n{unpickled.employers}\n')

def obj_to_file(fn, obj):
    with open(fn, 'wb') as pf:
        pickle.dump(obj, pf, protocol=pickle.HIGHEST_PROTOCOL)

def file_to_obj(fn, obj):
    with open(fn, 'rb') as pf:
        obj = pickle.load(pf)
        print(obj)
        return obj

# pickled = serialize(Person())
# deserialize(pickled)
# deserialize_employers(pickled)

obj = obj_to_file('./files_to_read/person.xyz', Person())
person = file_to_obj('./files_to_read/person.xyz', obj)

Detailed explanation

Class Person

The Person class demonstrates that pickle can serialize objects containing various data types:

serialize function

• pickle.dumps(obj, protocol=pickle.HIGHEST_PROTOCOL): serialize obj into bytes (memory).
• protocol=pickle.HIGHEST_PROTOCOL: uses the most recent version of the pickle protocol for better efficiency.
• Returns bytes (bytes object).

Deserialize function

• pickle.loads(obj): deserialize bytes into Python objects.
• obj here is the bytes object returned by pickle.dumps().

Function deserialize_employers

• Same as deserialize, but specifically accesses the employers attribute of the deserialized object.
• Illustrates that after deserialization, the Python object is completely reconstructed with all its attributes accessible.

Function obj_to_file — Serialization to file

• open(fn, 'wb'): opens the file in binary write mode ('wb').
• pickle.dump(obj, pf, protocol=pickle.HIGHEST_PROTOCOL): serialize obj directly into the pf file.
• Notice the .xyz extension — pickle can use any extension.

Function file_to_obj — Deserialization from file

• open(fn, 'rb'): opens the file in binary reading mode ('rb').
• pickle.load(pf): deserialize the object from the file.
• Returns the fully reconstructed Python object.

Pickle protocols

Safety Warnings

Important: Never deserialize data from an untrusted source with pickle.loads(). Pickle deserialization can execute arbitrary code — this is a known vulnerability.

Run script

py "./M5_Reading_and_Writing_Files/05_Demo_Persisting_Objects.py"

5.7 Summary and Final Thoughts

In this module we covered the most common file types:

We have thus reached the end of this course. It is now up to you to put this knowledge into practice in your projects or in your daily tasks.

6. Structure of exercise files

python-3-working-files/
├── texte_vocal.txt
├── exercise-files.zip
├── 02/
│   ├── finding-files-slides.pdf
│   └── demos/
│       ├── run.txt
│       ├── files/
│       │   ├── subfolder/
│       │   │   ├── 01_file_test.csv
│       │   │   ├── 01_file_test.txt
│       │   │   ├── 01_test_file copy.txt
│       │   │   ├── 01_test_file.csv
│       │   │   └── 01_test_file.txt
│       │   ├── 01_file.csv       ├── 01_file.txt
│       │   ├── 01_file_test.csv  ├── 01_file_test.txt
│       │   ├── 01_test.csv       ├── 01_test.txt
│       │   ├── 01_test_file.csv  ├── 01_test_file.txt
│       │   ├── 02_file.csv       ├── 02_file.txt
│       │   ├── 02_file_test.csv  ├── 02_file_test.txt
│       │   ├── 02_test.csv       ├── 02_test.txt
│       │   ├── 02_test_file.csv  ├── 02_test_file.txt
│       │   └── text.txt
│       ├── files_to_read/
│       │   ├── authors.json
│       │   ├── backup.py
│       │   ├── ef_author.xml
│       │   ├── example.txt
│       │   ├── names.csv
│       │   └── names2.csv
│       └── M2_Finding_Files/
│           ├── 01_Demo_Listing_a_Directory.py
│           ├── 02_Demo_Using_String_Methods.py
│           ├── 03_Demo_Pattern_Matching_with_fnmatch.py
│           ├── 04_Demo_Advanced_Pattern_Matching.py
│           └── 05_Demo_Pattern_Matching_with_glob.py
├── 03/
│   ├── working-with-files-and-folders-slides.pdf
│   └── demos/
│       ├── run.txt
│       ├── files/         (même structure que 02/demos/files/)
│       ├── files_to_read/ (même structure que 02/demos/files_to_read/)
│       └── M3_Working_with_Files_and_Folders/
│           ├── 01_Demo_Getting_File_Attributes.py
│           ├── 02_Demo_Traversing_a_Directory.py
│           ├── 03_Demo_Copying_Files.py
│           ├── 04_Demo_Moving_Files.py
│           ├── 05_Demo_Renaming_Files.py
│           └── 06_Demo_Deleting_Files.py
├── 04/
│   ├── archiving-files-slides.pdf
│   └── demos/
│       ├── run.txt
│       ├── files/         (même structure)
│       ├── files_to_read/ (même structure)
│       └── M4_Archiving_Files/
│           ├── 01_Demo_Creating_a_ZIP_Archive.py
│           ├── 02_Demo_Adding_Files_to_a_ZIP_Archive.py
│           ├── 03_Demo_Reading_a_ZIP_Archive.py
│           └── 04_Demo_Extracting_a_ZIP_Archive.py
└── 05/
    ├── reading-and-writing-files-slides.pdf
    └── demos/
        ├── run.txt
        ├── files/         (même structure)
        ├── files_to_read/
        │   ├── authors.json
        │   ├── backup.py
        │   ├── ef_author.xml
        │   ├── example.txt
        │   ├── names.csv
        │   ├── names2.csv
        │   └── person.xyz
        └── M5_Reading_and_Writing_Files/
            ├── 01_Demo_Working_with_Text_Files.py
            ├── 02_Demo_Working_with_CSV_Files.py
            ├── 03_Demo_Working_with_XML_Files.py
            ├── 04_Demo_Working_with_JSON_Files.py
            └── 05_Demo_Persisting_Objects.py

run.txt file — Run commands

Each demos/ folder contains a run.txt file with all the commands to run the scripts:

M2 >>
    py "./M2_Finding_Files/01_Demo_Listing_a_Directory.py"
    py "./M2_Finding_Files/02_Demo_Using_String_Methods.py"
    py "./M2_Finding_Files/03_Demo_Pattern_Matching_with_fnmatch.py"
    py "./M2_Finding_Files/04_Demo_Advanced_Pattern_Matching.py"
    py "./M2_Finding_Files/05_Demo_Pattern_Matching_with_glob.py"
M3 >>
    py "./M3_Working_with_Files_and_Folders/01_Demo_Getting_File_Attributes.py"
    py "./M3_Working_with_Files_and_Folders/02_Demo_Traversing_a_Directory.py"
    py "./M3_Working_with_Files_and_Folders/03_Demo_Copying_Files.py"
    py "./M3_Working_with_Files_and_Folders/04_Demo_Moving_Files.py"
    py "./M3_Working_with_Files_and_Folders/05_Demo_Renaming_Files.py"
    py "./M3_Working_with_Files_and_Folders/06_Demo_Deleting_Files.py"
M4 >>
    py "./M4_Archiving_Files/01_Demo_Creating_a_ZIP_Archive.py"
    py "./M4_Archiving_Files/02_Demo_Adding_Files_to_a_ZIP_Archive.py"
    py "./M4_Archiving_Files/03_Demo_Reading_a_ZIP_Archive.py"
    py "./M4_Archiving_Files/04_Demo_Extracting_a_ZIP_Archive.py"
M5 >>
    py "./M5_Reading_and_Writing_Files/01_Demo_Working_with_Text_Files.py"
    py "./M5_Reading_and_Writing_Files/02_Demo_Working_with_CSV_Files.py"
    py "./M5_Reading_and_Writing_Files/03_Demo_Working_with_XML_Files.py"
    py "./M5_Reading_and_Writing_Files/04_Demo_Working_with_JSON_Files.py"
    py "./M5_Reading_and_Writing_Files/05_Demo_Persisting_Objects.py"

Reference file: files_to_read/backup.py

This file is used as a data source in some text file reading demos. It represents a concrete example of a Python backup automation module, demonstrating real-world use of the os, zipfile, shutil, ftplib modules as well as third-party libraries like ftpsync and termcolor.

"""Backup Module"""

import os
import zipfile
import shutil
import time
import threading

from ftplib import FTP
from ftpsync.targets import FsTarget
from ftpsync.ftp_target import FtpTarget
from ftpsync.synchronizers import UploadSynchronizer
from termcolor import colored

from base import Base

class Backup(Base):
    """Backup Class"""

    __omit = []

    @classmethod
    def __init__(cls, bf, database):
        super(Backup, cls).__init__()
        cls.__list = cls.readcfg(bf + '.ini')
        cls.__svrf = bf + '.ftp'
        cls.__database = database
        Backup.__omit = []

    @classmethod
    def __numfiles(cls, fld):
        return sum([len(files) for r, d, files in os.walk(fld)])

    @classmethod
    def __chkftpfld(cls, remotefolder, server, user, password):
        if not Base.hasended():
            ftp = FTP(server)
            ftp.login(user, password)
            try:
                ftp.cwd(remotefolder)
            except BaseException as err:
                Base.log(str(err), True)
                ftp.mkd(remotefolder)
            ftp.close()
    ...

This file illustrates a real-world application that uses many of the concepts taught in this course: os.walk() for counting files, zipfile for creating archives, shutil for file operations, and ftplib for FTP transfer.

7. Python modules used — quick reference

Module os

Provides a portable interface for operating system dependent functionality.

Module pathlib

Object-oriented interface for manipulating file paths (Python 3.4+).

Module fnmatch

Unix-style pattern matching for file names.

Wildcards: * (all), ? (one character), [seq] (character in seq)

Module shutil

High-level file and tree operations.

Module zipfile

Creating and manipulating ZIP archives.

Module csv

Reading and writing CSV files.

Module xml.etree.ElementTree

Parsing and manipulating XML files.

Module json

JSON serialization and deserialization.

pickle module

Binary serialization of Python objects.

Module datetime

Handling dates and times.

Search Terms

python · foundations · data · analysis · engineering · analytics · run · detailed · explanation · script · filestoread · fnmatch · pattern · archive · json · matching · zip · directory · functions · glob · methods · pickle · shutil · string