Up and Running with Pandas

Prerequisites: Python 3 installed, basic Python knowledge.

Table of Contents

1. Course Overview
2. Introduction — Essential Tools
   • pandas
   • Jupyter Notebook
   • Installation
   • Working Workflow
3. Understanding Data Fundamentals with pandas
   • What Is a DataFrame?
   • Creating an Empty DataFrame
   • Creating a DataFrame with Data
   • Renaming Columns
   • Combining DataFrames with concat
   • DataFrame vs Series Structure
4. Representing Data Programmatically
   • Reading a JSON File with read_json
   • Reading a CSV File with read_csv
   • Exporting with to_json and to_csv
   • Read → Transform → Export Flow
5. Exploring and Evaluating Data with pandas
   • High-level Exploration
   • Filtering with iloc and Boolean Indexers
   • Descriptive Statistics with describe and corr
6. Complete Reference Tables
7. Summary and Best Practices

1. Course Overview

Up and Running with Pandas covers fundamental concepts for working with data using the pandas library. It is one of the most widely used data analysis and manipulation libraries in the Python and data science ecosystem.

What You Will Learn

• Understand how data is structured and how to build DataFrames
• Read and write CSV and JSON files
• Explore and analyze data using pandas functions and methods
• Import, clean, and manipulate data
• Perform basic exploratory data analysis (EDA)

Course Structure

2. Introduction — Essential Tools

pandas

pandas is a third-party Python library for data manipulation. It enables you to:

• Transform, modify, and access data very efficiently
• Provide memory-efficient data structures, ideal for large amounts of industry data
• Offer utilities for cleaning (data cleaning), aggregating, and visualizing data
• Write concise code: most operations can be done in a single line

pandas focuses on data analysis rather than low-level code logic.

Jupyter Notebook

Jupyter is a third-party Python library for web-based interactive Python documents. The Jupyter Notebook is the interactive document where you write and execute Python code.

Key characteristics:

• Cells contain Python code or Markdown
• The last line of a cell is automatically displayed (rendered) by Jupyter
• Ideal for data science because you can see results immediately
• Allows mixing code, explanations, and visualizations in the same document

Installation

# Install pandas
pip install pandas

# Install Jupyter
pip install jupyter

# Launch the Jupyter server (from your project directory)
cd Desktop/project
jupyter notebook

Working Workflow

1. Navigate to the project directory in the terminal
2. Run jupyter notebook
3. Create a new notebook (.ipynb)
4. Import pandas in the first cell: import pandas as pd
5. Write and execute cells one by one
6. To reset: Kernel → Restart & Run All

3. Understanding Data Fundamentals with pandas

What Is a DataFrame?

A DataFrame is a two-dimensional object used to represent a tabular data structure. It is similar to a spreadsheet table where data is presented as rows and columns.

DataFrame Characteristics

When to Use a DataFrame?

• To represent raw data that is not yet in programmatic format
• For Exploratory Data Analysis (EDA) thanks to the many available methods
• To aggregate data (groupby, sum, mean, etc.)
• To convert CSV, JSON, Excel files into a manipulable Python object

DataFrame vs Series Structure

graph TD
    A[pandas] --> B[DataFrame]
    A --> C[Series]
    
    B --> D["2D structure\n- Indexed rows\n- Named columns\n- Heterogeneous data"]
    C --> E["1D structure\n- Single column\n- Associated index\n- Homogeneous data"]
    
    D --> F["Example:\n  Date    | Amount\n  Jan 1   | 500.00\n  Jan 2   | 600.00"]
    E --> G["Example:\n  0  500.00\n  1  600.00\n  Name: Amount"]
    
    B --> H["Each column\nof a DataFrame\nis a Series"]
    H --> C

Series: A single column of a DataFrame. When you access df['Amount'], you get a Series.
DataFrame: A collection of Series sharing the same index.

Creating an Empty DataFrame

import pandas as pd

# Create an empty DataFrame with defined columns
df_sales = pd.DataFrame(columns=["Date", "Amount"])

df_sales
# Result: empty DataFrame with Date and Amount columns

Naming convention: It is customary to prefix variable names with df_ to indicate it is a DataFrame. E.g.: df_sales, df_customers, df_phoneprices.

Creating a DataFrame with Data

The most common method is to pass a Python dictionary to the pd.DataFrame() constructor. The dictionary keys become the columns, and the values (arrays) become the rows.

import pandas as pd

# Create a data dictionary
new_data = {
    "Date": ["January 1", "January 2"],
    "Amount": [500.00, 600.00]
}

# Build the DataFrame from the dictionary
df_new_data = pd.DataFrame(new_data)

df_new_data

Result:

The number on the left (0, 1, 2…) is the index automatically assigned by pandas.

Complete Example — DataFrame with Multiple Data Types

import pandas as pd

column_names = [
    "Date of Transaction",
    "Branch Name",
    "Amount of Sales",
    "Is Active"
]

# Empty DataFrame with defined columns
df_primary = pd.DataFrame(columns=column_names)

# Data to insert
default_data = {
    "Date of Transaction": ["January 1", "January 2"],
    "Branch Name": ["Branch A", "Branch B"],
    "Amount of Sales": [500.00, 600.00],
    "Is Active": [True, False]
}

df_default_data = pd.DataFrame(default_data)

df_default_data

Result:

Renaming Columns

# Rename columns by passing a new list of names
new_column_names = ["Transaction Date", "Sales Amount"]
df_sales.columns = new_column_names

df_sales

Combining DataFrames with concat

The pd.concat() method combines multiple DataFrames, either by rows (axis=0, default) or by columns (axis=1).

import pandas as pd

# --- Row combination (adding new rows) ---
df_sales = pd.concat([df_sales, df_new_data])
df_sales  # Rows from df_new_data added to df_sales

# --- Column combination (adding new columns) ---
new_column_data = {
    "Location": ["Store A", "Store B"]
}
df_column_data = pd.DataFrame(new_column_data)

df_sales = pd.concat([df_sales, df_column_data], axis=1)
df_sales  # Location column added to df_sales

Difference between axis=0 and axis=1

graph LR
    subgraph "axis=0 (default) — Adding rows"
        A1["DataFrame A\nDate | Amount\nJan 1| 500"] 
        A2["DataFrame B\nDate | Amount\nJan 3| 700"]
        A3["Result\nDate | Amount\nJan 1| 500\nJan 3| 700"]
        A1 --> A3
        A2 --> A3
    end
    
    subgraph "axis=1 — Adding columns"
        B1["DataFrame A\nDate | Amount\nJan 1| 500"]
        B2["DataFrame B\nLocation\nStore A"]
        B3["Result\nDate | Amount | Location\nJan 1| 500  | Store A"]
        B1 --> B3
        B2 --> B3
    end

4. Representing Data Programmatically

Reading a JSON File with read_json

JSON (JavaScript Object Notation) is a key-value format very similar to a Python dictionary. To represent tabular data in JSON, use an array of objects where each object represents a row of the DataFrame.

[
  {"Column 1": 1, "Column 2": 4},
  {"Column 1": 2, "Column 2": 5},
  {"Column 1": 3, "Column 2": 6}
]

import pandas as pd

# Read a JSON file
df_from_json = pd.read_json("example-1.json")

df_from_json
# Result: DataFrame with columns Column 1 and Column 2

pandas also accepts the column-oriented format (format generated by to_json()):

{
  "Column 1": {"0": 1, "1": 2, "2": 3},
  "Column 2": {"0": 4, "1": 5, "2": 6}
}

# Both formats are accepted by read_json
df_from_json2 = pd.read_json("example-2.json")
df_from_json2  # Same result

Reading a CSV File with read_csv

CSV (Comma-Separated Values) is a text format where each value is separated by a comma. It is the most common tabular data format. The first line is usually the header.

Branch,Date,Amount
Branch A,January 1,500
Branch B,January 2,600

Case 1 — CSV with header (most common case)

import pandas as pd

# Header is automatically detected (first line)
df_csv = pd.read_csv("phoneprices.csv")

df_csv
# Result: DataFrame with first row columns as headers

Case 2 — CSV without header

import pandas as pd

# If CSV has no header line, pass header=None
# pandas assigns numeric indices as column names (0, 1, 2...)
df_csv = pd.read_csv("example-1.csv", header=None)

df_csv
# Columns named 0, 1, 2... instead of text names

Warning: If you don’t specify header=None for a headerless CSV, pandas will use the first data row as column names, which is incorrect.

Exporting with to_json and to_csv

import pandas as pd

# Read a CSV
df_csv = pd.read_csv("example-1.csv", header=None)

# Export to CSV (without pandas index)
df_csv.to_csv("data.csv", index=None)

# Export to JSON
json_string = df_from_json.to_json()
print(json_string)  # Displays the JSON representation

index=None in to_csv() prevents pandas from writing the index column (0, 1, 2…) to the output file. Without it, you’ll get an unwanted extra column.

Read → Transform → Export Flow

flowchart LR
    subgraph Sources ["Data Sources"]
        CSV["CSV File\n(.csv)"]
        JSON["JSON File\n(.json)"]
        RAW["Python Dictionary\n/ Raw Data"]
    end

    subgraph Reading ["Reading (Import)"]
        RC["pd.read_csv()"]
        RJ["pd.read_json()"]
        DF_C["pd.DataFrame(dict)"]
    end

    subgraph DataFrame ["pandas DataFrame"]
        DFO["DataFrame Object\n- Indexed rows\n- Named columns\n- Available methods"]
    end

    subgraph Transformation ["Transformation / Cleaning"]
        FILT["Filtering\niloc / Boolean"]
        RENAME["Column rename\ndf.columns = [...]"]
        CONCAT["Concatenation\npd.concat()"]
        DESCRIBE["Analysis\ndescribe() / corr()"]
    end

    subgraph Export ["Export (Output)"]
        TC["df.to_csv()"]
        TJ["df.to_json()"]
    end

    CSV --> RC
    JSON --> RJ
    RAW --> DF_C
    RC --> DFO
    RJ --> DFO
    DF_C --> DFO
    DFO --> FILT
    DFO --> RENAME
    DFO --> CONCAT
    DFO --> DESCRIBE
    FILT --> TC
    RENAME --> TC
    CONCAT --> TJ
    DESCRIBE --> TJ

5. Exploring and Evaluating Data with pandas

High-level Exploration

The process of exploring, examining, and deriving information from a DataFrame is called data wrangling. The main reasons:

1. Filter — Extract specific rows/columns for analysis
2. Preprocess — Clean data for subsequent analysis
3. Summarize — Aggregate or derive information for other applications

Knowing the DataFrame Size

import pandas as pd

# Read the dataset (no header)
df_yeast = pd.read_csv("yeast.csv", header=None)

# Number of rows
len(df_yeast)  # Returns 1484

# Display full DataFrame
df_yeast

head() and tail() — Data Preview

import pandas as pd

df_phoneprices = pd.read_csv("phoneprices.csv")
# DataFrame: 2000 rows × 21 columns

# Display first 5 rows (default)
df_phoneprices.head()

# Display first 10 rows
df_phoneprices.head(10)

# Display last 5 rows (default)
df_phoneprices.tail()

# Display last 10 rows
df_phoneprices.tail(10)

Selecting Specific Columns

# Select multiple columns by name (returns a DataFrame)
df_phoneprices[["battery_power", "blue", "clock_speed"]]

# Select columns by numeric index (for DataFrame without header)
df_yeast[[0, 1, 9]]

Filtering with iloc and Boolean Indexers

iloc — Filtering by Position

The iloc property allows filtering by numeric index (position). It accepts two arguments:

• row_indexer: i:n — rows from index i up to n (excluded)
• column_indexer (optional): i:n — columns from index i up to n (excluded)

Important: Value n must always be greater than i, otherwise no data is returned.

import pandas as pd

df_phoneprices = pd.read_csv("phoneprices.csv")

# Select rows 0 to 3 (4 rows), all columns
df_phoneprices.iloc[0:4]

# Select rows 0 to 3, columns 0 to 2 (3 columns)
df_phoneprices.iloc[0:4, 0:3]

# Select rows 8 to 11 of yeast dataset
df_yeast.iloc[8:12]

# Select rows 0 to 3, columns 0 to 2 in yeast
df_yeast.iloc[0:4, 0:3]

Boolean Indexer — Filtering by Condition

Pass a boolean expression in brackets after the DataFrame to filter rows satisfying the condition.

import pandas as pd

df_phoneprices = pd.read_csv("phoneprices.csv")

# Filter rows where price_range == 1
df_phoneprices[df_phoneprices["price_range"] == 1]

# Filter rows where ram >= 2000
df_phoneprices[df_phoneprices["ram"] >= 2000]

# Yeast dataset: filter rows where column 9 == 'MIT'
df_yeast[df_yeast[9] == 'MIT']

# Filter rows where column 1 >= 0.5
df_yeast[df_yeast[1] >= 0.5]

Descriptive Statistics with describe and corr

describe() — Statistical Summary

The describe() method returns a statistical summary of all numeric columns:

import pandas as pd

df_phoneprices = pd.read_csv("phoneprices.csv")

# Full statistical summary
df_phoneprices.describe()

Result (example):

describe() returns for each numeric column: count, mean, std, min, 25%, 50% (median), 75%, max.

corr() — Correlation Matrix

The corr() method computes the correlation between all numeric columns. A score close to 1 indicates strong positive correlation, close to -1 strong negative correlation, and close to 0 no correlation.

# Correlation matrix between all numeric columns
df_phoneprices.corr()

6. Complete Reference Tables

Read/Write Methods

Exploration Methods

Filtering and Indexing Methods

Transformation and Aggregation Methods

Combination Methods

Statistics and Analysis

7. Summary and Best Practices

What You Learned

✅ Module 1 — Install and configure pandas and Jupyter Notebook; define an efficient working workflow.

✅ Module 2 — Create DataFrames from scratch or from Python dictionaries; rename columns; combine DataFrames with pd.concat().

✅ Module 3 — Read JSON files with read_json() and CSV files with read_csv(); export with to_csv() and to_json().

✅ Module 4 — Explore real datasets with head(), tail(), len(); filter with iloc and boolean indexers; derive statistics with describe() and corr().

Best Practices

Typical Use Case — Complete Pipeline

import pandas as pd

# 1. LOAD — Import data
df = pd.read_csv("phoneprices.csv")

# 2. EXPLORE — Understand structure
print(df.shape)        # (2000, 21)
print(df.dtypes)       # Data types
df.head()              # First 5 rows
df.describe()          # Descriptive statistics

# 3. FILTER — Extract data of interest
df_affordable = df[df["price_range"] == 1]       # Affordable phones
df_high_ram = df[df["ram"] >= 2000]              # RAM >= 2000 MB
df_subset = df.iloc[0:100, 0:5]                  # First 100 rows, 5 columns

# 4. TRANSFORM — Modify / clean
df.dropna()                                       # Remove null rows
df["price_range"].replace({1: "Cheap", 4: "Premium"}, inplace=True)

# 5. AGGREGATE — Summarize data
df.groupby("price_range")["ram"].mean()           # Average RAM by price category

# 6. ANALYZE — Derive insights
df.corr()              # Correlations between numeric variables

# 7. EXPORT — Save results
df_affordable.to_csv("affordable_phones.csv", index=None)

Datasets Used in the Course

Search Terms

running · pandas · python · foundations · data · analysis · engineering · analytics · dataframe · methods · filtering · case · csv · boolean · columns · corr · describe · exploration · header · iloc · read · reading · statistics