Building Machine Learning Models in Python with scikit-learn

Level: beginner to intermediate | Language: Python 3 | Main library: scikit-learn

Table of Contents

1. Course Overview
2. Data Processing with scikit-learn
   • 2.1 Types of Machine Learning Problems
   • 2.2 Supervised vs Unsupervised Learning
   • 2.3 Numerical Data — Mean and Variance
   • 2.4 Numerical Data Standardization
   • 2.5 Categorical Data Encoding
   • 2.6 Representing Text as Numbers
   • 2.7 CountVectorizer, TfidfVectorizer, HashingVectorizer
   • 2.8 Representing Images as Numbers
3. Specialized Regression Models
   • 3.1 Linear Regression (Ordinary Least Squares)
   • 3.2 Measuring Fit — R-squared
   • 3.3 L1 and L2 Norms
   • 3.4 Overfitting and the Bias-Variance Tradeoff
   • 3.5 Multicollinearity and Regularization
   • 3.6 Lasso and Ridge Regression
   • 3.7 Support Vector Regression (SVR)
4. SVM and Gradient Boosting Models
   • 4.1 Support Vector Machines — Classification
   • 4.2 Demo: Text Classification with SVM
   • 4.3 Demo: MNIST Image Classification with SVM and Grid Search
   • 4.4 Decision Trees
   • 4.5 Random Forests
   • 4.6 Gradient Boosting Regression
5. Clustering and Dimensionality Reduction
   • 5.1 Clustering — Core Concepts
   • 5.2 K-means Clustering
   • 5.3 Mean Shift Clustering
   • 5.4 K-means vs Mean Shift
   • 5.5 Principal Components Analysis (PCA)
6. Resources and Further Reading

1. Course Overview

Prerequisites

• Proficiency in Python 3
• Basic knowledge of Jupyter Notebooks
• Machine learning basics (not required)

Libraries Used

Course Structure

graph TD
    A[Building ML Models with scikit-learn] --> B[Module 2\nData Processing]
    A --> C[Module 3\nSpecialized Regression]
    A --> D[Module 4\nSVM & Gradient Boosting]
    A --> E[Module 5\nClustering & PCA]

    B --> B1[Numerical/Categorical Data]
    B --> B2[Text Representation]
    B --> B3[Image Feature Extraction]

    C --> C1[Linear Regression OLS]
    C --> C2[Lasso & Ridge]
    C --> C3[Support Vector Regression]

    D --> D1[SVM for Classification]
    D --> D2[Decision Trees]
    D --> D3[Gradient Boosting]

    E --> E1[K-means Clustering]
    E --> E2[Mean Shift Clustering]
    E --> E3[PCA]

2. Data Processing with scikit-learn

2.1 Types of Machine Learning Problems

graph LR
    ML[Machine Learning] --> CL[Classification\nEx: spam or ham?]
    ML --> RE[Regression\nEx: house price]
    ML --> CL2[Clustering\nEx: user groups]
    ML --> RX[Rule Extraction\nEx: recommendations]

    CL --> SL[Supervised]
    RE --> SL
    CL2 --> UL[Unsupervised]
    RX --> UL

Concrete examples:

2.2 Supervised vs Unsupervised Learning

flowchart TD
    subgraph Supervised
        direction TB
        T1[Training data\nwith labels] --> M1[ML Model]
        M1 --> P1[Prediction]
        P1 -->|Compare with\ntrue value| LF[Loss Function / Cost Function]
        LF -->|Adjust\nparameters| M1
    end

    subgraph Unsupervised["Unsupervised"]
        direction TB
        T2[Data without labels] --> M2[ML Model]
        M2 --> P2[Patterns / Structures]
    end

Typical supervised model pipeline:

Training phase:
  [Labeled data] → [ML Model] → [Prediction] ──→ [Loss Function]
                                      ↑                  │
                                      └── adjustment ─────┘

Prediction phase:
  [New data] → [Trained model] → [Predicted label]

2.3 Numerical Data — Mean and Variance

Two fundamental data types:

Mean: $$\bar{x} = \frac{\sum_{i=1}^{n} x_i}{n}$$

Variance: $$\sigma^2 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n}$$

Standard Deviation: $$\sigma = \sqrt{\sigma^2}$$

Z-score (standardization): $$z = \frac{x - \bar{x}}{\sigma}$$

2.4 Numerical Data Standardization

Standardization transforms data to have a mean of 0 and a standard deviation of 1. This is an important prerequisite for many ML algorithms.

Demo: m1-demo1-CategoricalAndNumericData

import pandas as pd
from sklearn import preprocessing

# Load the "exams" dataset
exam_data = pd.read_csv('../data/exams.csv', quotechar='"')

# Check means before standardization
math_average    = exam_data['math score'].mean()
reading_average = exam_data['reading score'].mean()
writing_average = exam_data['writing score'].mean()
print('Math Avg: ',    math_average)
print('Reading Avg: ', reading_average)
print('Writing Avg: ', writing_average)

# Apply z-score scaling to numerical columns
exam_data[['math score']]    = preprocessing.scale(exam_data[['math score']])
exam_data[['reading score']] = preprocessing.scale(exam_data[['reading score']])
exam_data[['writing score']] = preprocessing.scale(exam_data[['writing score']])

# After scaling: mean ≈ 0, standard deviation ≈ 1
print('Math Avg after scaling: ', exam_data['math score'].mean())

Why standardize? If exams are scored on different scales (math/100, reading/50, writing/20), they cannot be compared directly. The z-score expresses them in a common unit.

2.5 Categorical Data Encoding

ML algorithms only accept numerical data. Categorical variables must be encoded.

Label Encoding

Useful when:

• There are only two values (binary → 0/1)
• There is a meaningful ordinal relationship (e.g., low < medium < high)

from sklearn import preprocessing

le = preprocessing.LabelEncoder()

# Encode 'female' → 0, 'male' → 1
exam_data['gender'] = le.fit_transform(exam_data['gender'].astype(str))

print(le.classes_)  # ['female' 'male']

One-Hot Encoding

Useful when values have no ordinal relationship between them.

One-Hot Encoding principle:

Days of the week:
         Sun Mon Tue Wed Thu Fri Sat
Monday → [ 0   1   0   0   0   0   0 ]
Thursday→ [ 0   0   0   0   1   0   0 ]
Saturday→ [ 0   0   0   0   0   0   1 ]

# Using pandas for One-Hot Encoding
import pandas as pd

# Encode the 'race/ethnicity' column (values: Group A to Group E)
race_dummies = pd.get_dummies(exam_data['race/ethnicity'])

# One-Hot Encoding of multiple columns at once
exam_data = pd.get_dummies(
    exam_data,
    columns=['parental level of education', 'lunch', 'test preparation course']
)

Note: The ML model does not use ordinal representations (0=low, 1=medium, 2=high) as real ordinal relationships — that interpretation is only for human understanding.

2.6 Representing Text as Numbers

Text must be converted to numbers for ML algorithms (e.g., sentiment analysis).

Three approaches for word embeddings:

graph TD
    WE[Word Embeddings] --> OH[1. One-Hot Representation]
    WE --> FB[2. Frequency-Based Embeddings]
    WE --> PB[3. Prediction-Based Embeddings\nex: Word2Vec, BERT]

    FB --> CV[Count Vectors\nword frequency]
    FB --> TF[TF-IDF\nweighted frequency]

    PB --> DL[Deep Learning\nCaptures semantic meaning]

Count Vectors (Bag of Words)

Example with two movie reviews:

Review 1: "The movie was bad, the actors were bad, the sets were bad"
Review 2: "The actors were bad, sets were bad"

Vocabulary: [the, movie, was, bad, actors, were, sets]

Review 1: [3, 1, 1, 3, 1, 1, 1]
Review 2: [1, 0, 0, 2, 1, 1, 1]
           ^                ^
           3×"the"          2×"bad"

Drawbacks:

• Very large and sparse vectors for large vocabularies
• Does not account for word order
• Gives too much weight to common words (“the”, “a”, “is”…)

TF-IDF (Term Frequency — Inverse Document Frequency)

Formula:

$$\text{TF-IDF}(i, j) = \text{TF}(i, j) \times \text{IDF}(i)$$

$$\text{TF}(i,j) = \frac{\text{occurrences of word } i \text{ in doc } j}{\text{total words in doc } j}$$

$$\text{IDF}(i) = \log\left(\frac{\text{total documents}}{\text{documents containing word } i}\right)$$

Principle:

• High score if the word is frequent in this document but rare in the global corpus
• Low score for very common words (“the”, “a”, “is”…) — they appear everywhere

2.7 CountVectorizer, TfidfVectorizer, HashingVectorizer

Demo: m1-demo2-TextFeatureExtraction

from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Corpus of 4 documents
corpus = [
    'This is the first document.',
    'This is the second document.',
    'Third document. Document number three',
    'Number four. To repeat, number four'
]

# --- CountVectorizer ---
vectorizer = CountVectorizer()
bag_of_words = vectorizer.fit_transform(corpus)
# Result: sparse matrix 4×12 (4 docs, 12 unique words)
print(bag_of_words)

# View word → index mapping
print(vectorizer.vocabulary_)
# {'document': 1, 'first': 2, 'four': 3, 'is': 4, ...}

# Display as DataFrame
import pandas as pd
pd.DataFrame(bag_of_words.toarray(), columns=vectorizer.get_feature_names())

# --- TfidfVectorizer ---
tfidf_vectorizer = TfidfVectorizer()
bag_tfidf = tfidf_vectorizer.fit_transform(corpus)
pd.DataFrame(bag_tfidf.toarray(), columns=tfidf_vectorizer.get_feature_names())

Difference between TfidfVectorizer and TfidfTransformer:

TfidfVectorizer == CountVectorizer + TfidfTransformer

HashingVectorizer — for large vocabularies:

from sklearn.feature_extraction.text import HashingVectorizer

# Limits the number of features to n (avoids memory issues)
# Note: collisions can occur (two words → same hash)
hashing_vectorizer = HashingVectorizer(n_features=10)
X = hashing_vectorizer.transform(corpus)

2.8 Representing Images as Numbers

Image structure:

Color image (3 RGB channels):
  Dimensions: height × width × 3
  Each pixel: [R, G, B] where each value ∈ [0, 255]

  Examples:
    Pure red   → [255,   0,   0]
    Pure green → [  0, 255,   0]
    Pure blue  → [  0,   0, 255]

Grayscale image (1 channel):
  Dimensions: height × width × 1
  Each pixel: intensity ∈ [0, 1] (after normalization)

graph LR
    IMG[Image] --> COLOR[Color\n3 RGB channels\nh × w × 3]
    IMG --> GRAY[Grayscale\n1 channel\nh × w × 1]

    COLOR --> FLAT[Flatten\n1D vector\nh×w×3]
    GRAY --> FLAT2[Flatten\n1D vector\nh×w×1]

    FLAT --> ML[ML Model]
    FLAT2 --> ML

Demo: m1-demo3-ImageFeatureExtraction

import cv2
import matplotlib.pyplot as plt
%matplotlib inline

# Read a color image
imagePath = '../data/dog.jpg'
image = cv2.imread(imagePath)

# Display
plt.imshow(image)

# Shape: (130, 173, 3) → height=130, width=173, 3 channels
print(image.shape)

# RGB value of the first pixel
print(image[0][0])

# Resize to 32×32
size = (32, 32)
resized_image = cv2.resize(image, size)
print(resized_image.shape)  # (32, 32, 3)

# Flatten to 1D vector
flat_image = resized_image.flatten()
print(flat_image.shape)  # (3072,) = 32 × 32 × 3

# Convert to grayscale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
print(gray_image.shape)  # (130, 173)

# Normalize (values between 0 and 1)
gray_image_normalized = gray_image / 255.0

3. Specialized Regression Models

3.1 Linear Regression (Ordinary Least Squares)

Regression is used to predict a continuous value (Y) from explanatory variables (X).

Simple regression:   Y = A + B·X
Multiple regression: Y = A + B₁X₁ + B₂X₂ + ... + BₙXₙ

Goal: minimize the Least Squares Error

   Y
   │          ●
   │      ●  /
   │    ●   /  ← Line 1 (best fit)
   │   ●  /
   │ ● --/-------- Line 2
   │  ↗
   └──────────── X

   Error = Σ(Yi_actual - Yi_predicted)²
   → Find A and B that minimize this sum

Data preparation — Demo: m2-demo1-LassoRidge

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

# Load the automobile dataset (UCI)
auto_data = pd.read_csv('../data/imports-85.data', sep=r'\s*,\s*', engine='python')

# Replace missing values
auto_data = auto_data.replace('?', np.nan)

# Convert price to numeric
auto_data['price'] = pd.to_numeric(auto_data['price'], errors='coerce')

# Drop unnecessary columns
auto_data = auto_data.drop('normalized-losses', axis=1)

# Convert number of cylinders (text → number)
cylinders_dict = {'two': 2, 'three': 3, 'four': 4,
                  'five': 5, 'six': 6, 'eight': 8, 'twelve': 12}
auto_data['num-of-cylinders'].replace(cylinders_dict, inplace=True)

# One-Hot Encoding of categorical variables
auto_data = pd.get_dummies(auto_data, columns=[
    'make', 'fuel-type', 'aspiration', 'num-of-doors',
    'body-style', 'drive-wheels', 'engine-location',
    'engine-type', 'fuel-system'
])

# Drop rows with missing values
auto_data = auto_data.dropna()

# Separate features / label
X = auto_data.drop('price', axis=1)
Y = auto_data['price']

# Train/test split (80% / 20%)
X_train, X_test, Y_train, Y_test = train_test_split(
    X, Y, test_size=0.2, random_state=0
)

# Linear regression
linear_model = LinearRegression()
linear_model.fit(X_train, Y_train)

# R² score on training data
print(linear_model.score(X_train, Y_train))  # ~0.967

# Feature coefficients
print(linear_model.coef_)

# Predictions on test data
y_predict = linear_model.predict(X_test)

# MSE and RMSE
from sklearn.metrics import mean_squared_error
import math
mse  = mean_squared_error(y_predict, Y_test)
rmse = math.sqrt(mse)
print(f'RMSE: {rmse:.2f}')

3.2 Measuring Fit — R-squared

The coefficient of determination R² measures the quality of the model fit.

$$R^2 = 1 - \frac{\text{SS}{\text{res}}}{\text{SS}{\text{tot}}} = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$$

A high R² on training but low on test = overfitting!

3.3 L1 and L2 Norms

These norms are fundamental to understanding Lasso and Ridge.

L1 distance (Manhattan / City Block):

Point A = (1, 0)    Point B = (5, 4)

L1 distance = |5-1| + |4-0| = 4 + 4 = 8

   ↑ Y
 4 │    B
   │    │
   │    │ ← "block" path (Manhattan)
 0 │A───┘
   └────────→ X
   1    5

$$|x|1 = \sum{i} |x_i|$$

L2 distance (Euclidean):

L2 distance = √[(5-1)² + (4-0)²] = √[16 + 16] = √32 ≈ 5.66

   ↑ Y
 4 │    B
   │   ↗ ← straight line (Euclidean)
 0 │A
   └────────→ X

$$|x|2 = \sqrt{\sum{i} x_i^2}$$

Summary:

3.4 Overfitting and the Bias-Variance Tradeoff

graph LR
    subgraph Underfitting["Underfitting (High Bias)"]
        A[Model too simple\nEx: straight line on complex data]
        A --> A1[High error on\nboth training AND test]
    end

    subgraph Overfitting["Overfitting (High Variance)"]
        B[Model too complex\nMemorizes training data]
        B --> B1[Low error on training\nHigh error on test]
    end

    subgraph Ideal["Ideal"]
        C[Good balance\nbias-variance]
        C --> C1[Good generalization]
    end

The Bias-Variance Tradeoff:

Total error = Bias² + Variance + Irreducible noise

High bias     → Model too simple (underfitting)
High variance → Model too complex (overfitting)

→ Finding the right balance is key!

Techniques to combat overfitting:

3.5 Multicollinearity and Regularization

Multicollinearity occurs when multiple features (X) are highly correlated. This makes the model unstable and prone to overfitting.

Regularization adds a penalty to the objective function to constrain the coefficients:

$$\text{Objective function} = \underbrace{\text{MSE}}{\text{error}} + \underbrace{\alpha \cdot \text{Penalty}(\beta)}{\text{regularization}}$$

• Alpha (α): hyperparameter controlling penalty strength
  • α = 0 → classic OLS regression
  • High α → smaller coefficients, simpler model

3.6 Lasso and Ridge Regression

graph TD
    OLS["OLS Regression\nMinimizes: RSS"] --> LASSO["Lasso (L1)\nMinimizes: RSS + α·Σ|βᵢ|"]
    OLS --> RIDGE["Ridge (L2)\nMinimizes: RSS + α·Σβᵢ²"]
    LASSO --> ELASTIC["Elastic Net\nCombines Lasso + Ridge"]
    RIDGE --> ELASTIC

Lasso Demo — m2-demo1-LassoRidge

from sklearn.linear_model import Lasso

# Lasso with α=0.5 and data normalization
lasso_model = Lasso(alpha=0.5, normalize=True)
lasso_model.fit(X_train, Y_train)

# R² on training data (slightly lower than OLS)
print(lasso_model.score(X_train, Y_train))

# Inspect coefficients → many are ZERO (feature selection!)
import pandas as pd
coef_series = pd.Series(lasso_model.coef_, index=X_train.columns)
print(coef_series[coef_series != 0])

# Prediction and comparison
y_predict_lasso = lasso_model.predict(X_test)
r2_lasso = lasso_model.score(X_test, Y_test)
print(f'R² test (Lasso): {r2_lasso:.4f}')

# Visualization
import matplotlib.pyplot as plt
%pylab inline
pylab.rcParams['figure.figsize'] = (15, 6)
plt.plot(y_predict_lasso, label='Predicted')
plt.plot(Y_test.values, label='Actual')
plt.ylabel('Price')
plt.legend()
plt.show()

Ridge Demo — m2-demo1-LassoRidge

from sklearn.linear_model import Ridge

# Ridge with α=0.05
ridge_model = Ridge(alpha=0.05, normalize=True)
ridge_model.fit(X_train, Y_train)

print(ridge_model.score(X_train, Y_train))  # ~95.38%

# Coefficients: all non-zero, but smaller than with OLS
print(pd.Series(ridge_model.coef_, index=X_train.columns).sort_values())

# R² on test data
y_predict_ridge = ridge_model.predict(X_test)
print(f'R² test (Ridge): {ridge_model.score(X_test, Y_test):.4f}')  # ~88.75%

# Increasing α → simpler model, training R² decreases
ridge_model_v2 = Ridge(alpha=0.5, normalize=True)
ridge_model_v2.fit(X_train, Y_train)
print(f'R² training (Ridge α=0.5): {ridge_model_v2.score(X_train, Y_train):.4f}')  # ~92%

3.7 Support Vector Regression (SVR)

SVR uses the same principles as SVMs for classification, but with a different objective function.

Concept of the ε (epsilon) tube:

       ┌─────────────────────────────────────────┐
       │         Points inside the tube           │  ← Not penalized
       │    ε ─────────────────────────           │
     Y │   ─────────── Hyperplane ────────        │
       │    ε ─────────────────────────           │
       │  ● Point outside the tube                │  ← Penalized by C
       └─────────────────────────────────────────┘
                              X

• ε (epsilon): tube width — errors inside are ignored
• C: penalty parameter for points outside the tube
  • High C → focus on distant points (risk of overfitting)
  • Low C → better global fit

Demo: m2-demo2-SVR

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.svm import SVR

# Auto MPG dataset (UCI)
auto_data = pd.read_csv('../data/auto-mpg.data',
    delim_whitespace=True, header=None,
    names=['mpg', 'cylinders', 'displacement', 'horsepower',
           'weight', 'acceleration', 'model', 'origin', 'car_name'])

# Drop the car name column (305 unique values, not useful)
auto_data = auto_data.drop('car_name', axis=1)

# Convert origin to meaningful values + One-Hot
auto_data['origin'] = auto_data['origin'].replace(
    {1: 'america', 2: 'europe', 3: 'asia'}
)
auto_data = pd.get_dummies(auto_data, columns=['origin'])

# Clean missing data
auto_data = auto_data.replace('?', np.nan)
auto_data = auto_data.dropna()

# Separate features / label
X = auto_data.drop('mpg', axis=1)
Y = auto_data['mpg']
X_train, X_test, Y_train, Y_test = train_test_split(
    X, Y, test_size=0.2, random_state=0
)

# SVR model with linear kernel, C=1.0
regression_model = SVR(kernel='linear', C=1.0)
regression_model.fit(X_train, Y_train)

print(f'R² training: {regression_model.score(X_train, Y_train):.4f}')

# Reduce C → better global model
regression_model_v2 = SVR(kernel='linear', C=0.5)
regression_model_v2.fit(X_train, Y_train)
print(f'R² training (C=0.5): {regression_model_v2.score(X_train, Y_train):.4f}')  # ~81%
print(f'R² test (C=0.5): {regression_model_v2.score(X_test, Y_test):.4f}')        # ~83%

4. SVM and Gradient Boosting Models

4.1 Support Vector Machines — Classification

SVM evolution by dimensionality:

graph LR
    D1[1D Data\nWord count] --> S1[Separation by\na point]
    D2[2D Data\nWord count\n+ publication time] --> S2[Separation by\na line]
    D3[3D Data] --> S3[Separation by\na plane]
    DN[N-dimensional Data] --> SN[Separation by\na hyperplane]

Support Vectors and margin:

     ●  ●  (class +)
      ●
  ─ ─ ─ ─ ─ ─   ← +ε boundary (positive support vectors)
  ───────────── ← decision hyperplane
  ─ ─ ─ ─ ─ ─   ← -ε boundary (negative support vectors)
      ○
   ○    ○  (class -)

  ← margin →

  SVM objective: maximize this margin!

Hard margin vs Soft margin:

Kernels for non-linearly separable data:

graph TD
    NL[Non-linearly separable data] --> K[Kernel Trick]
    K --> KL[Linear Kernel\nClassic dot product]
    K --> KP[Polynomial Kernel\n(x·z + c)^d]
    K --> KR[RBF Kernel\nexp(-γ||x-z||²)]
    K --> KS[Sigmoid Kernel]

4.2 Demo: Text Classification with SVM

Dataset: 20 Newsgroups (20 document categories)

from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.svm import LinearSVC
from sklearn.pipeline import Pipeline

# Download the dataset
twenty_train = fetch_20newsgroups(subset='train', shuffle=True)

# Available keys: data, target, target_names, filenames, ...
print(twenty_train.target_names)
# ['alt.atheism', 'comp.graphics', 'comp.os.ms-windows.misc', ...]

# --- Manual approach ---
# Step 1: CountVectorizer → bag of words
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)
print(X_train_counts.shape)  # (11314, 130107)

# Step 2: TF-IDF Transformer
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)

# Step 3: Linear SVM classifier
clf_svc = LinearSVC(penalty="l2", dual=False, tol=1e-3)
clf_svc.fit(X_train_tfidf, twenty_train.target)

# --- Pipeline approach (recommended) ---
clf_pipeline = Pipeline([
    ('vect',  CountVectorizer()),
    ('tfidf', TfidfTransformer()),
    ('clf',   LinearSVC(penalty="l2", dual=False, tol=0.001))
])
clf_pipeline.fit(twenty_train.data, twenty_train.target)

# Evaluation
twenty_test = fetch_20newsgroups(subset='test', shuffle=True)
predicted = clf_pipeline.predict(twenty_test.data)

from sklearn.metrics import accuracy_score
print(f'Accuracy: {accuracy_score(twenty_test.target, predicted):.4f}')

Pipeline: a sequence of transformations followed by an estimator. The output of each step is passed as input to the next.

4.3 Demo: MNIST Image Classification with SVM and Grid Search

MNIST dataset: 28×28 pixel images of handwritten digits (0-9)

Digit "4" in MNIST:
  0.0  0.0  0.0  0.0  0.0  0.0  0.0
  0.0  0.0  0.8  0.0  0.0  0.0  0.0
  0.0  0.7  0.0  0.9  0.0  0.0  0.0
  0.0  0.9  0.0  0.8  0.0  0.0  0.0
  0.9  0.9  0.9  1.0  0.9  0.9  0.0
  0.0  0.0  0.0  0.7  0.0  0.0  0.0
  0.0  0.0  0.0  0.8  0.0  0.0  0.0
  (intensity = 0 for white background, ~1 for strokes)

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.metrics import accuracy_score

# Load MNIST data (Kaggle)
mnist_data = pd.read_csv("../data/mnist/train.csv")

# Features (pixels) and labels (digit 0-9)
features = mnist_data.columns[1:]
X = mnist_data[features]
Y = mnist_data['label']

# Split 90% training / 10% test
# Normalize: divide by 255 to get values between 0 and 1
X_train, X_test, Y_train, Y_test = train_test_split(
    X / 255., Y, test_size=0.1, random_state=0
)

# Basic linear SVM model
clf_svm = LinearSVC(penalty="l2", dual=False, tol=1e-5)
clf_svm.fit(X_train, Y_train)

y_pred = clf_svm.predict(X_test)
print(f'SVM Accuracy: {accuracy_score(Y_test, y_pred):.4f}')  # ~91%

# --- Grid Search to optimize hyperparameters ---
from sklearn.model_selection import GridSearchCV

param_grid = {
    'penalty':   ['l1', 'l2'],
    'tol':       [1e-3, 1e-4, 1e-5]
}

grid_search = GridSearchCV(LinearSVC(dual=False), param_grid, cv=3)
grid_search.fit(X_train, Y_train)

print(f'Best parameters: {grid_search.best_params_}')
# Ex: {'penalty': 'l1', 'tol': 0.001}

# Retrain with best parameters
clf_best = LinearSVC(penalty="l1", dual=False, tol=1e-3)
clf_best.fit(X_train, Y_train)
y_pred_best = clf_best.predict(X_test)
print(f'Accuracy after Grid Search: {accuracy_score(Y_test, y_pred_best):.4f}')

Grid Search — Principle:

param_grid = {
    'penalty': ['l1', 'l2'],     # 2 values
    'tol':     [1e-3, 1e-4, 1e-5] # 3 values
}
→ 2 × 3 = 6 combinations tested
→ cv=3 means 3-fold cross-validation per combination
→ Total: 6 × 3 = 18 training runs

4.4 Decision Trees

Decision trees are the building blocks of Gradient Boosting.

Example: Classify athletes (jockeys vs basketball players)

graph TD
    Q1{Weight > 150 lbs?}
    Q1 -->|No| Q2{Height > 6 feet?}
    Q1 -->|Yes| B[Basketball player]
    Q2 -->|No| J[Jockey]
    Q2 -->|Yes| B2[Basketball player]

Characteristics:

4.5 Random Forests

Random Forests combine multiple decision trees to reduce overfitting.

graph TD
    DATA[Training data] --> S1[Subset 1]
    DATA --> S2[Subset 2]
    DATA --> SN[Subset N]

    S1 --> T1[Tree 1]
    S2 --> T2[Tree 2]
    SN --> TN[Tree N]

    T1 --> V1[Prediction 1]
    T2 --> V2[Prediction 2]
    TN --> VN[Prediction N]

    V1 --> AGG{Aggregation}
    V2 --> AGG
    VN --> AGG

    AGG -->|Regression| MEAN[Average]
    AGG -->|Classification| MODE[Majority vote]

“If everyone in the room is thinking the same thing, then somebody isn’t thinking”
Individual models must be as different as possible for the ensemble to be effective.

4.6 Gradient Boosting Regression

Gradient Boosting sequentially builds weak models, each one correcting the errors of the previous one.

graph LR
    D[Data] --> M1[Weak model 1\nShallow Decision Tree]
    M1 -->|Residual E1| M2[Weak model 2\nLearns E1]
    M2 -->|Residual E2| M3[Weak model 3\nLearns E2]
    M3 -->|Residual E3| MN[...]

    M1 --> COMB[Combined model]
    M2 --> COMB
    M3 --> COMB
    MN --> COMB

    COMB --> PRED[Final prediction\nmore robust]

Mathematically (with linear learners):

Model 1: Y = A₁ + B₁X    →  Residual E₁
Model 2: E₁ ≈ A₂ + B₂X  →  Residual E₂
Model 3: E₂ ≈ A₃ + B₃X  →  Residual E₃

Combined model:
Y_final = (A₁+A₂+A₃) + (B₁+B₂+B₃)X + E₃

Shrinkage Factor:

• Each learner is multiplied by a factor λ (learning rate)
• Slows convergence → reduces overfitting

Demo: m3-demo3-GradientBoosting

from sklearn.ensemble import GradientBoostingRegressor
import matplotlib.pyplot as plt

# Model parameters
params = {
    'n_estimators':    500,   # Number of weak learners (trees)
    'max_depth':         6,   # Max depth of each tree (shallow!)
    'min_samples_split': 2,   # Minimum samples to split
    'learning_rate':  0.01,   # Shrinkage factor
    'loss':            'ls'   # Least squares (regression)
}

gbr_model = GradientBoostingRegressor(**params)
gbr_model.fit(X_train, Y_train)

# R² on training data
print(f'R² training: {gbr_model.score(X_train, Y_train):.4f}')

# Predictions on test data
y_predict = gbr_model.predict(X_test)

# Visualize predictions vs actual values
%pylab inline
pylab.rcParams['figure.figsize'] = (15, 6)
plt.plot(y_predict, label='Predicted')
plt.plot(Y_test.values, label='Actual')
plt.ylabel('Price')
plt.legend()
plt.show()

# --- Grid Search for Gradient Boosting ---
from sklearn.model_selection import GridSearchCV

param_grid = {
    'n_estimators':  [500, 1000, 2000],
    'max_depth':     [3, 6],
    'learning_rate': [0.01, 0.05]
}

grid_search = GridSearchCV(
    GradientBoostingRegressor(loss='ls', min_samples_split=2),
    param_grid,
    cv=3,
    scoring='r2'
)
grid_search.fit(X_train, Y_train)
print(f'Best parameters: {grid_search.best_params_}')

Ensemble learning comparison:

5. Clustering and Dimensionality Reduction

5.1 Clustering — Core Concepts

Clustering is an unsupervised learning technique that groups similar data points.

graph TD
    PRINC[Fundamental principle\nAnything can be represented\nby a set of numbers] --> PERSON[Person\nage + height + weight\n= point in 3D space]
    PRINC --> DOC[Document\nTF-IDF vector\n= point in N-D space]
    PRINC --> IMG[Image\npixels\n= point in N-D space]

    PERSON --> CLUST[Clustering]
    DOC --> CLUST
    IMG --> CLUST

    CLUST --> G1[Group 1\nSimilar characteristics]
    CLUST --> G2[Group 2\nSimilar characteristics]
    CLUST --> GN[Group N]

Applications:

• Segmenting Facebook users by interests
• Grouping documents by topic
• Anomaly detection (fraud detection)
• Image compression (color quantization)

5.2 K-means Clustering

flowchart TD
    A[Initialize K centroids randomly] --> B[Assign each point\nto the nearest centroid]
    B --> C[Recalculate centroids\nas the mean of cluster points]
    C --> D{Have the centroids\nchanged?}
    D -->|Yes| B
    D -->|No| E[Convergence!\nFinal clusters]

The centroid is the average point (center of gravity) of all points in a cluster:

$$\text{centroid}k = \frac{1}{|C_k|} \sum{x_i \in C_k} x_i$$

Key limitation: K must be specified in advance.

5.3 Mean Shift Clustering

Mean Shift does not require specifying the number of clusters in advance.

Principle: gradient ascent on density

1. For each point, define a neighborhood (circle of radius = bandwidth)
2. Compute the kernel function over this neighborhood
3. Move the point toward the center of mass of the neighborhood
4. Repeat until convergence
5. Points that converge to the same peak → same cluster

Available kernels:

Key parameter — Bandwidth (h):

$$K_h(x) = \frac{1}{h} K\left(\frac{x}{h}\right)$$

• Small bandwidth → more clusters, finer granularity
• Large bandwidth → fewer clusters, coarser granularity

Demo: m4-demo1-ClusteringWithMeanShift

import pandas as pd
import numpy as np
from sklearn import preprocessing
from sklearn.cluster import MeanShift, estimate_bandwidth

# Titanic dataset
titanic_data = pd.read_csv('../data/titanic.csv', quotechar='"')

# Drop irrelevant columns
titanic_data.drop(['PassengerId', 'Name', 'Ticket', 'Cabin'],
                  'columns', inplace=True)

# Encode gender (female=0, male=1)
le = preprocessing.LabelEncoder()
titanic_data['Sex'] = le.fit_transform(titanic_data['Sex'].astype(str))

# One-Hot for embarkation port
titanic_data = pd.get_dummies(titanic_data, columns=['Embarked'])

# Drop rows with missing values
titanic_data = titanic_data.dropna()

# Automatically estimate bandwidth
bandwidth = estimate_bandwidth(titanic_data)
print(f'Estimated bandwidth: {bandwidth:.2f}')

# Apply Mean Shift
analyzer = MeanShift(bandwidth=30)
analyzer.fit(titanic_data)

# Retrieve cluster labels
labels = analyzer.labels_
print(f'Clusters found: {np.unique(labels)}')

# Add labels to DataFrame
titanic_data['cluster_group'] = labels

# Cluster analysis
cluster_summary = titanic_data.groupby('cluster_group').mean()
cluster_summary['counts'] = titanic_data.groupby('cluster_group').size()
print(cluster_summary)

Typical results on the Titanic dataset:

Cluster 0 (~680 people):
  - Survived: ~38%
  - Pclass: 2-3 (lower class)
  - Sex: 0.65 (mostly male)
  - Average fare: ~25

Cluster 1 (~207 people):
  - Survived: ~74% ← high rate!
  - Pclass: 1 (first class)
  - Sex: 0.25 (mostly female)
  - Average fare: ~192

Cluster 2 (~3 people):
  - Too small to be significant

5.4 K-means vs Mean Shift

graph LR
    subgraph KM[K-means]
        direction TB
        KM1[Simple and fast]
        KM2[K must be known]
        KM3[Spherical clusters]
    end

    subgraph MS[Mean Shift]
        direction TB
        MS1[Auto number of clusters]
        MS2[Handles complex shapes]
        MS3[Computationally intensive O N²]
    end

5.5 Principal Components Analysis (PCA)

PCA is an unsupervised dimensionality reduction technique.

Goal: represent complex data with a minimum number of dimensions, preserving maximum variance.

graph LR
    HD[High-dimensional data\nN features] --> PCA[PCA]
    PCA --> LD[Low-dimensional data\nk features << N]
    LD --> ML[ML Model\nfaster]

    style HD fill:#ff9999
    style LD fill:#99ff99

Geometric intuition:

2D data:
      ●
    ●   ●          PC1 →  direction of maximum variance
  ●   ●   ●        PC2 ↗  perpendicular to PC1
    ●   ●
      ●

Projection onto PC1 = 1D representation that preserves most information

The Principal Components:

• PC1: direction that maximizes variance of projections
• PC2: direction perpendicular to PC1, maximizing remaining variance
• PCk: perpendicular to all previous PCs

PCs must be orthogonal (perpendicular) to express maximum variation with minimum directions.

Demo: m4-demo3-PCAandDimensionalityReduction

import pandas as pd
import numpy as np
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import seaborn as sns

# Dataset: white wine quality (UCI)
wine_data = pd.read_csv('../data/winequality-white.csv',
    names=['Fixed Acidity', 'Volatile Acidity', 'Citric Acid',
           'Residual Sugar', 'Chlorides', 'Free Sulfur Dioxide',
           'Total Sulfur Dioxide', 'Density', 'pH', 'Sulphates',
           'Alcohol', 'Quality'],
    skiprows=1, sep=r'\s*;\s*', engine='python')

# 7 possible quality scores → random prediction = 14%
print(wine_data['Quality'].unique())

# Standardize features
X = wine_data.drop('Quality', axis=1)
Y = wine_data['Quality']
X = preprocessing.scale(X)

X_train, X_test, Y_train, Y_test = train_test_split(
    X, Y, test_size=0.2, random_state=0
)

# --- Baseline model WITHOUT PCA ---
clf_baseline = LinearSVC(penalty='l1', dual=False, tol=1e-3)
clf_baseline.fit(X_train, Y_train)
print(f'Accuracy without PCA: {clf_baseline.score(X_test, Y_test):.4f}')  # ~49%

# --- Visualize feature correlations ---
corrmat = wine_data.corr()
f, ax = plt.subplots(figsize=(10, 10))
sns.set(font_scale=1.1)
sns.heatmap(corrmat, vmax=.8, square=True, annot=True,
            fmt='.2f', cmap="winter")
plt.show()

# --- Apply PCA ---
pca = PCA(n_components=1, whiten=True)
# whiten=True: normalizes components for covariance close to identity
X_reduced = pca.fit_transform(X)

# Eigenvalues of each component
print(pca.explained_variance_)

# Variance explained ratio per component
print(pca.explained_variance_ratio_)

# --- Scree Plot to visualize PC importance ---
pca_full = PCA(n_components=11)  # All components
pca_full.fit(X)

plt.plot(pca_full.explained_variance_ratio_)
plt.xlabel('Dimension (Principal Component)')
plt.ylabel('Explained variance ratio')
plt.title('Scree Plot')
plt.show()

# --- Model WITH PCA ---
X_train_pca, X_test_pca, Y_train_pca, Y_test_pca = train_test_split(
    X_reduced, Y, test_size=0.2, random_state=0
)

clf_pca = LinearSVC(penalty='l1', dual=False, tol=1e-3)
clf_pca.fit(X_train_pca, Y_train_pca)
print(f'Accuracy with PCA (1 component): {clf_pca.score(X_test_pca, Y_test_pca):.4f}')

PCA parameters:

Scree Plot interpretation:

Explained variance
   |
1.0│●
   │  ●
0.5│     ●
   │        ●  ●  ●  ●  ●  ●  ●
0.0│________________________________
   1  2  3  4  5  6  7  8  9  10 11
                  PC (dimension)

→ The "elbow" indicates the right number of dimensions to retain

6. Resources and Further Reading

Recommended Book

Hands-On Machine Learning with Scikit-Learn and TensorFlow
by Aurélien Géron
— Easy to read, practical, and very comprehensive

Related Courses

Datasets Used

Final Algorithm Overview

graph TD
    ML[Machine Learning] --> SL[Supervised]
    ML --> UL[Unsupervised]

    SL --> REG[Regression\n→ continuous value]
    SL --> CLF[Classification\n→ discrete class]

    REG --> OLS[Linear Regression OLS]
    REG --> LAS[Lasso Regression L1]
    REG --> RID[Ridge Regression L2]
    REG --> SVR[Support Vector Regression]
    REG --> GBR[Gradient Boosting Regression]

    CLF --> SVM[Support Vector Machine]
    CLF --> DT[Decision Tree]
    CLF --> RF[Random Forest]
    CLF --> GBC[Gradient Boosting Classifier]

    UL --> CLUST[Clustering]
    UL --> DIMRED[Dimensionality Reduction]

    CLUST --> KM[K-means]
    CLUST --> MS[Mean Shift]

    DIMRED --> PCA[PCA]

scikit-learn Estimator Reference

Search Terms

machine · models · python · scikit-learn · ml · fundamentals · engineering · data · science · regression · clustering · classification · encoding · mean · svm · boosting · gradient · k-means · numbers · numerical · representing · shift · support · text