Normalize Data for Analysis with Pandas

Versions: Python 3.9 · NumPy 1.4.4 · Pandas 1.22.4 · Scikit-learn 1.2.2

Table of Contents

1. Course Overview
2. Why Normalize?
   • Problem of Native Distribution
   • What Normalization Accomplishes
   • Version Compatibility
3. Simple Normalization Techniques
   • Z-score / Standard Scaling
   • Demo: Standard Scaling with Pandas
   • Min-Max Scaling
   • Demo: Min-Max Scaling with Pandas
   • Sensitivity to Outliers
4. Gaussian Normalization
   • Gaussian Distribution — Refresher
   • Norms: L2, L1, Max
   • Demo: Gaussian Normalization with Pandas
5. Reference Diagrams
   • Normalization Pipeline
   • Data Transformation Flow
6. Method Reference Tables
7. Complete Code Snippets
8. Key Takeaways

1. Course Overview

Data normalization is the act of transforming a dataset from its raw (but clean) format into a refined version with a better signal-to-noise ratio for downstream applications.

Fundamental problem: each feature has its own native distribution. If they are not normalized, features with higher nominal values will artificially take on more importance in optimization algorithms.

What you will learn:

2. Why Normalize?

2.1 Problem of Native Distribution

Imagine two features in the same dataset:

• Feature A — uniform distribution, values between −5 and −1
• Feature B — uniform distribution, values between 3 and 14
• Feature C — normal distribution, mean ≈ 5

Without normalization, the optimization algorithm assigns disproportionate weight to features with higher nominal values (here Feature B), regardless of their actual importance.

import pandas as pd
import matplotlib.pyplot as plt

# Visualize distribution before normalization
data.boxplot()
plt.title("Raw feature distribution")
plt.show()

# Statistically describe the dataset
data.describe()

2.2 What Normalization Accomplishes

After normalization:

• Features still differ in values (as they should)
• But they share the same patterns → the algorithm assigns equal weight to each feature
• Result: an even play between all features

Definition: Normalization is the act of transforming a dataset from its raw format into a refined version with a better signal-to-noise ratio for downstream applications — whether a machine learning model or a Power BI dashboard.

2.3 Version Compatibility

3. Simple Normalization Techniques

3.1 Z-score / Standard Scaling

Z-score scaling (or Standard Scaling) rests on two mathematical premises:

Premise 1 — Centering (mean = 0):

$$Y = X - \bar{X}$$

By subtracting the mean from each data point, the resulting distribution has a mean of 0. All features end up in the same location.

Premise 2 — Scaling (variance = 1):

$$Z = \frac{X - \bar{X}}{\sigma}$$

By dividing by the variance, the resulting variance is 1. This controls the spread of the distribution.

Z-score properties:

• Preserves the distribution type (a uniform distribution remains uniform)
• The resulting distribution always has mean = 0 and std = 1
• Inspired by the standard Gaussian distribution (which ties everything together)

3.2 Demo: Standard Scaling with Pandas

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler

# --- Step 1: Load the dataset ---
# data is a DataFrame with 5000 points and 3 features:
# uniform_negative, uniform_positive, normal
data = pd.read_csv("dataset.csv")

# --- Step 2: Analyze the raw dataset ---
print(data.describe())
data.boxplot()

# --- Step 3: Instantiate the scaler ---
scaler = StandardScaler()

# --- Step 4: Fit (learn statistics) ---
scaler.fit(data)

# --- Step 5: Transform (apply normalization) ---
scaled_array = scaler.transform(data)

# --- Step 6: Convert back to DataFrame ---
standard_df = pd.DataFrame(scaled_array, columns=data.columns)

# --- Verification ---
print(standard_df.describe())
# Expected result: mean ≈ 0, std ≈ 1 for each feature

# Visualize after normalization
standard_df.boxplot()

Result: Each feature now has mean ≈ 0 and std ≈ 1, regardless of its original distribution.

3.3 Min-Max Scaling

Min-Max Scaling transforms the range of a feature to a given interval, typically [0, 1].

Formula:

$$X_{scaled} = \frac{X - X_{min}}{X_{max} - X_{min}} \times (max - min) + min$$

By default, the target interval is [0, 1], but it can be freely configured (e.g., [2, 5]).

Min-Max properties:

• Does not change the distribution of the variable (it stays identical, just shifted)
• Only changes the range of values
• mean becomes ≈ 0.5 and std ≈ 0.28 for a uniform distribution normalized to [0,1]

⚠️ Outlier sensitivity: A single extreme data point can compress the entire distribution toward one end of the interval.

3.4 Demo: Min-Max Scaling with Pandas

import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler

# --- Step 1: Load the same dataset ---

# --- Step 2: Instantiate both scalers for comparison ---
standard_scaler = StandardScaler()
minmax_scaler = MinMaxScaler()  # Default: range [0, 1]

# For a custom range:
# minmax_scaler = MinMaxScaler(feature_range=(2, 5))

# --- Step 3: Fit on both scalers ---
standard_scaler.fit(data)
minmax_scaler.fit(data)

# --- Step 4: Transform ---
standard_array = standard_scaler.transform(data)
minmax_array = minmax_scaler.transform(data)

# --- Step 5: Convert back to DataFrames ---
standard_df = pd.DataFrame(standard_array, columns=data.columns)
minmax_df = pd.DataFrame(minmax_array, columns=data.columns)

# --- Comparative analysis ---
print("=== Standard Scaling ===")
print(standard_df.describe())
# mean ≈ 0, std ≈ 1

print("\n=== Min-Max Scaling ===")
print(minmax_df.describe())
# min ≈ 0, max ≈ 1, mean ≈ 0.5, std ≈ 0.28

# Visualize side by side
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
standard_df.boxplot(ax=axes[0])
axes[0].set_title("Standard Scaling")
minmax_df.boxplot(ax=axes[1])
axes[1].set_title("Min-Max Scaling")
plt.show()

3.5 Sensitivity to Outliers

import pandas as pd
from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Clean dataset
data_clean = pd.DataFrame({'feature': [1, 2, 3, 4, 5, 4, 3, 2, 1, 2]})

# Dataset with an extreme outlier
data_outlier = pd.DataFrame({'feature': [1, 2, 3, 4, 5, 4, 3, 2, 1, 1000]})

# Demonstrate impact on Min-Max
scaler = MinMaxScaler()

clean_scaled = scaler.fit_transform(data_clean)
outlier_scaled = scaler.fit_transform(data_outlier)

print("Without outlier:")
print(pd.DataFrame(clean_scaled).describe())

print("\nWith outlier (value 1000):")
print(pd.DataFrame(outlier_scaled).describe())
# The entire distribution is compressed toward 0!
# The outlier takes the entire [0, 1] range

4. Gaussian Normalization

4.1 Gaussian Distribution — Refresher

The Gaussian distribution (or normal distribution) is a bell-shaped distribution governed by two parameters:

• μ (mu) — the mean (center of the curve)
• σ (sigma) — the variance (width of the curve)

The standard Gaussian distribution is: μ = 0, σ = 1 — which is exactly what the StandardScaler produces!

Effects of parameters:

Why does this matter? Most modern machine learning algorithms assume that data follows a Gaussian distribution. The term “normalization” comes directly from “normal distribution” (= Gaussian).

4.2 Norms: L2, L1, Max

The Scikit-learn Normalizer takes one argument: the norm. A norm is a measure of the size of a data point.

Example with data point (7, 5):

L2 Norm (Euclidean)

$$|x|_2 = \sqrt{x_1^2 + x_2^2} = \sqrt{7^2 + 5^2} \approx 8.6$$

• Also known as the Euclidean norm (hypotenuse)
• Use when you want to preserve geometric properties (e.g., petal length and width in the Iris dataset)
• Result: no data point has an L2 norm greater than 1

L1 Norm (Manhattan)

$$|x|_1 = |x_1| + |x_2| = 7 + 5 = 12$$

• Also known as the Manhattan norm (sum of horizontal + vertical distances)
• More robust to outliers than L2
• Tends to push uniform_positive and normal distributions closer to 0

Max Norm (Chebyshev)

$$|x|_\infty = \max(|x_1|, |x_2|) = \max(7, 5) = 7$$

• Uses the maximum absolute value of the vector
• Clips all values greater than 1 and less than −1
• Creates spikes at the extremes → distorts the distribution more
• Useful to prevent outliers (extreme values are capped)

4.3 Demo: Gaussian Normalization with Pandas

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, Normalizer
import matplotlib.pyplot as plt

# --- Step 1: Instantiate 4 scalers ---
scalers = [
    ("Standard",  StandardScaler()),
    ("L2",        Normalizer(norm='l2')),
    ("L1",        Normalizer(norm='l1')),
    ("Max",       Normalizer(norm='max')),
]

# --- Step 2: Fit each scaler ---
for name, scaler in scalers:
    scaler.fit(data)

# --- Step 3: Transform and create DataFrames ---
scaled_dfs = {}
for name, scaler in scalers:
    scaled_array = scaler.transform(data)
    scaled_dfs[name] = pd.DataFrame(scaled_array, columns=data.columns)

# --- Analysis: Side-by-side boxplots ---
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
for ax, (name, df) in zip(axes.flatten(), scaled_dfs.items()):
    df.boxplot(ax=ax)
    ax.set_title(f"Normalizer: {name}")
plt.tight_layout()
plt.show()

# --- Analysis: Density plots ---
fig, axes = plt.subplots(2, 2, figsize=(14, 10))
for ax, (name, df) in zip(axes.flatten(), scaled_dfs.items()):
    for col in df.columns:
        df[col].plot.density(ax=ax, label=col)
    ax.set_title(f"Density: {name}")
    ax.legend()
plt.tight_layout()
plt.show()

# --- Compare statistics ---
for name, df in scaled_dfs.items():
    print(f"\n=== {name} ===")
    print(df.describe().round(3))

Observed results:

5. Reference Diagrams

5.1 Normalization Pipeline

flowchart TD
    A[Raw dataset\nfeatures with native distributions] --> B{Exploratory analysis\ndata.describe\ndata.boxplot}
    B --> C{Technique\nchoice}

    C -->|Center and scale\nmean=0, std=1| D[StandardScaler\nZ-score Scaling]
    C -->|Change range\nmin-max→range| E[MinMaxScaler\nMin-Max Scaling]
    C -->|Approximate Gaussian\nnorm-based| F[Normalizer\nGaussian Normalization]

    F --> F1[norm='l2'\nEuclidean]
    F --> F2[norm='l1'\nManhattan]
    F --> F3[norm='max'\nChebyshev]

    D --> G[scaler.fit data]
    E --> G
    F1 --> G
    F2 --> G
    F3 --> G

    G --> H[scaler.transform data]
    H --> I[pd.DataFrame scaled_array\ncolumns=data.columns]
    I --> J[Normalized dataset\nready for ML / BI]

    style A fill:#ff9999
    style J fill:#99ff99
    style D fill:#9999ff
    style E fill:#ffcc99
    style F fill:#cc99ff

5.2 Data Transformation Flow

flowchart LR
    subgraph Before["Before normalization"]
        direction TB
        A1["Feature A\nUnif[-5, -1]\nmean=-3, std=1.1"]
        A2["Feature B\nUnif[3, 14]\nmean=8.5, std=3.2"]
        A3["Feature C\nNormal\nmean=5, std=2"]
    end

    subgraph Choice["Scaler Choice"]
        direction TB
        S1["StandardScaler\nmean=0, std=1"]
        S2["MinMaxScaler\nrange=[0,1]"]
        S3["Normalizer L2\nEuclidean norm ≤1"]
    end

    subgraph After["After normalization"]
        direction TB
        B1["Feature A'\nmean≈0, std≈1"]
        B2["Feature B'\nmean≈0, std≈1"]
        B3["Feature C'\nmean≈0, std≈1"]
    end

    subgraph Problem["Problem without normalization"]
        P["Feature B dominates\nthe algorithm\ndue to its\nhigher values"]
    end

    Before --> Problem
    Before --> Choice
    Choice --> After

    subgraph Apps["Downstream Applications"]
        ML["ML Models\nLinear Regression\nSVM, KNN..."]
        BI["Power BI\nDashboards"]
    end

    After --> Apps

    style Problem fill:#ffcccc
    style After fill:#ccffcc
    style Apps fill:#cce5ff

6. Method Reference Tables

Comparison of Normalization Techniques

Key Scaler Parameters

When to Use Which Technique?

Normalization Workflow Methods

7. Complete Code Snippets

Complete Normalization Workflow

import pandas as pd
import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler, Normalizer

# === 1. Loading and exploration ===
data = pd.read_csv("dataset.csv")

print("Shape:", data.shape)
print("\nDescriptive statistics:")
print(data.describe())

# Initial visualization
data.boxplot(figsize=(10, 6))

# === 2. Standard Scaling (Z-score) ===
standard_scaler = StandardScaler()
standard_array = standard_scaler.fit_transform(data)
standard_df = pd.DataFrame(standard_array, columns=data.columns)

print("\nAfter Standard Scaling:")
print(standard_df.describe().round(3))
# mean ≈ 0, std ≈ 1 for each column

# === 3. Min-Max Scaling ===
minmax_scaler = MinMaxScaler(feature_range=(0, 1))
minmax_array = minmax_scaler.fit_transform(data)
minmax_df = pd.DataFrame(minmax_array, columns=data.columns)

print("\nAfter Min-Max Scaling:")
print(minmax_df.describe().round(3))
# min ≈ 0, max ≈ 1

# === 4. Gaussian Normalization (L2) ===
normalizer_l2 = Normalizer(norm='l2')
l2_df = pd.DataFrame(normalizer_l2.fit_transform(data), columns=data.columns)

# === 5. Gaussian Normalization (L1) ===
normalizer_l1 = Normalizer(norm='l1')
l1_df = pd.DataFrame(normalizer_l1.fit_transform(data), columns=data.columns)

# === 6. Gaussian Normalization (Max) ===
normalizer_max = Normalizer(norm='max')
max_df = pd.DataFrame(normalizer_max.fit_transform(data), columns=data.columns)

# === 7. Visual comparison ===
import matplotlib.pyplot as plt

all_scalers = {
    "Original": data,
    "Standard": standard_df,
    "Min-Max": minmax_df,
    "L2": l2_df,
    "L1": l1_df,
    "Max": max_df,
}

fig, axes = plt.subplots(2, 3, figsize=(18, 10))
for ax, (name, df) in zip(axes.flatten(), all_scalers.items()):
    df.boxplot(ax=ax)
    ax.set_title(name)
plt.tight_layout()
plt.show()

Manual Z-score Verification

import pandas as pd
import numpy as np

# Manual Z-score verification
feature = pd.Series([2, 4, 6, 8, 10, 3, 5, 7, 9, 1])

# Manual method
z_manual = (feature - feature.mean()) / feature.std()

# Scikit-learn method
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
z_sklearn = scaler.fit_transform(feature.values.reshape(-1, 1)).flatten()

print(f"Original mean:  {feature.mean():.2f}")
print(f"Original std:   {feature.std():.2f}")
print(f"Mean after Z:   {z_manual.mean():.10f}")  # ≈ 0
print(f"Std after Z:    {z_manual.std():.4f}")    # ≈ 1

Manual Norm Calculation

import numpy as np

# Data point: (7, 5)
x = np.array([7, 5])

# L2 norm (Euclidean)
l2 = np.sqrt(np.sum(x**2))
print(f"L2 norm: {l2:.4f}")   # ≈ 8.6023

# L1 norm (Manhattan)
l1 = np.sum(np.abs(x))
print(f"L1 norm: {l1}")       # = 12

# Max norm (Chebyshev)
max_norm = np.max(np.abs(x))
print(f"Max norm: {max_norm}") # = 7

# Normalized vectors
print(f"\nL2 normalized: {x / l2}")    # each value ÷ 8.6
print(f"L1 normalized: {x / l1}")    # each value ÷ 12
print(f"Max normalized: {x / max_norm}") # each value ÷ 7

Scikit-learn Pipeline with Normalization

import pandas as pd
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

# Create an ML pipeline with built-in normalization
pipeline = Pipeline([
    ('scaler', StandardScaler()),       # Automatic normalization
    ('model', LinearRegression())       # ML model
])

# Train/test split
X = data.drop('target', axis=1)
y = data['target']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# The pipeline normalizes automatically
pipeline.fit(X_train, y_train)
score = pipeline.score(X_test, y_test)
print(f"R² Score: {score:.4f}")

8. Key Takeaways

Module 1 — Why Normalize?

• Normalization is the process of applying a transformation to features to make them more tractable for a downstream process
• Without normalization, features with higher nominal values artificially dominate optimization algorithms
• After normalization, all features have an even play

Module 2 — Simple Normalization Techniques

• Standard Scaling (Z-score): transforms the feature to have mean = 0 and variance = 1
• Min-Max Scaling: changes the range of the feature without affecting other properties
• Both techniques are sensitive to outliers — a single extreme point can distort everything

Module 3 — Gaussian Normalization

• There are 3 main norms: L2 (Euclidean), L1 (Manhattan), Max (Chebyshev)
• Each norm has a different effect on features, but all attempt to approximate the distribution toward a Gaussian shape
• The Max norm is the exception: it clips values, creating spikes at the extremes — useful for preventing outliers
• The term “normalization” comes directly from “normal distribution” (= Gaussian)

Practical Tips

Practice with different dataset distributions and different keyword arguments for the scalers.

Test the difference by plugging each scaler before an algorithm like LinearRegression to observe performance differences.

Analyze why Min-Max Scaling is so sensitive to outliers — understand, not just apply.

Additional Resources

• Python for Data Analysis — Wes McKinney (recommended next step)
• GitHub Repo: axel-sirota/normalise-data-pandas

Search Terms

normalize · data · analysis · pandas · python · foundations · engineering · analytics · normalization · gaussian · norm · scaling · techniques · distribution · manual · max · min-max · pipeline · reference · standard · workflow · z-score