Machine Learning Boosting Techniques

Table of Contents

1. Module 1 — Understanding Boosting Algorithms

   • 1.1 System and Software Requirements
   • 1.2 Ensemble Learning — Concepts and Techniques
   • 1.3 Averaging vs. Boosting
   • 1.4 Boosting Techniques
   • 1.5 Adaptive Boosting (AdaBoost)
   • 1.6 Gradient Boosting
   • 1.7 Gradient Boosting Variants

2. Module 2 — Implementing Boosting Algorithms

   • 2.1 XGBoost — Classification on UCI Bank Dataset
   • 2.2 XGBoost — Regularization, Early Stopping and Cross-validation
   • 2.3 XGBoost on GPU (Google Colab)
   • 2.4 LightGBM — Classification on HR Analytics
   • 2.5 LightGBM — Max Bins, Imbalanced Data and Early Stopping
   • 2.6 LightGBM — Native API, GOSS and Feature Importances
   • 2.7 CatBoost — Regression on the Insurance Dataset
   • 2.8 CatBoost — Early Stopping, Feature Importances and Snapshots

3. Module 3 — Tuning and Interpretability

   • 3.1 Hyperparameter Tuning
   • 3.2 GridSearchCV with XGBoost
   • 3.3 SHAP Values
   • 3.4 Demo — Computing SHAP Values with LightGBM

4. Appendix — Algorithm Comparison

Module 1 — Understanding Boosting Algorithms

1.1 System and Software Requirements

pip install -U scikit-learn xgboost lightgbm catboost

1.2 Ensemble Learning — Concepts and Techniques

Ensemble learning: a machine learning technique where multiple models are combined to achieve better performance than any individual model.

Advantages of Ensemble Learning:

The three key questions in Ensemble Learning:

┌──────────────────────────────────────────────────────────────────┐
│               Fundamental Questions                              │
├──────────────────┬───────────────────────┬───────────────────────┤
│  Which learners  │  How to train them    │  How to combine       │
│  to use?         │  (on which data)      │  their predictions?   │
├──────────────────┼───────────────────────┼───────────────────────┤
│ Weak learners    │ Data subsets or       │ Averaging             │
│ (shallow         │ different feature     │ Majority voting       │
│  trees)          │ subsets               │ Stacking              │
└──────────────────┴───────────────────────┴───────────────────────┘

Choosing individual learners:

• Each learner should be as different as possible from the others to learn distinct patterns
• An individual learner is a weak learner — a simple model with limited power alone
• In practice: use shallow decision trees

Methods for combining predictions:

• Averaging: average the individual predictions
• Voting: majority vote
• Stacking: train another ML model on the predictions of the individual learners

Learner diversity — fundamental principle:

Same model + same data = same errors  ❌
Different models + different data = diversity in predictions  ✅

1.3 Averaging vs. Boosting

flowchart TD
    A[Ensemble Learning] --> B[Averaging]
    A --> C[Boosting]
    B --> D[Bagging\nSampling with\nreplacement]
    B --> E[Pasting\nSampling without\nreplacement]
    C --> F[Adaptive Boosting\nAdaBoost]
    C --> G[Gradient Boosting\nXGBoost / LightGBM / CatBoost]

Averaging vs. Boosting Comparison:

Averaging (detail):

• Trains multiple learners in parallel
• The final prediction is the average (or vote) of individual predictions
• Examples: Random Forest, Bagging

Boosting (detail):

• Trains multiple learners in sequence
• Each model learns from the errors of the previous model
• The contribution of each model can be controlled via a learning rate
• Each added learner boosts the model’s accuracy

1.4 Boosting Techniques

Core principle: “Turn your weaknesses into strengths.”

There are two main families of boosting:

graph LR
    A[Boosting] --> B[Adaptive Boosting\nAdaBoost]
    A --> C[Gradient Boosting]
    B --> B1[Re-weights\nmisclassified\nexamples]
    C --> C1[Predicts residual\nerrors of the\nprevious model]
    C --> D[XGBoost]
    C --> E[LightGBM]
    C --> F[CatBoost]

Advantages of Boosting models:

• High accuracy on real-world predictions
• Focuses on improving previous weaknesses
• Good bias-variance balance
• Works well with weak learners (shallow trees)

Disadvantages of Boosting models:

• Tendency to overfit
• Slower to train (sequential)
• Sensitive to outliers
• Harder to tune

1.5 Adaptive Boosting (AdaBoost)

How AdaBoost works:

1. Build and train models sequentially
2. Instances misclassified by each model receive a higher weight
3. The next model sees these instances more often → it focuses on them
4. The accuracy of each model determines its influence on the final prediction

sequenceDiagram
    participant D as Data
    participant M1 as Model 1
    participant W as Re-weighting
    participant M2 as Model 2
    participant Mn as Model N
    participant F as Final Prediction

    D->>M1: Uniform weights (1/N)
    M1->>W: Identifies errors
    W->>M2: Increases weight\nof misclassified examples
    M2->>Mn: Process repeated...
    M1->>F: Weighted vote
    M2->>F: Weighted vote
    Mn->>F: Weighted vote
    F-->>F: Combined prediction

Concrete AdaBoost example (4 records):

Learner 1 — Uniform initial weights:

Learner 2 — Adjusted weights:

Model weighting rules:

• More accurate models → higher weights → greater influence
• Persistently incorrect models → negative weights
• Random models → weight ≈ 0 → do not influence the ensemble

Advantages of AdaBoost:

• Effectively boosts weak learners
• Automatically focuses on hard cases
• Surprisingly resistant to overfitting
• Simple and interpretable models

1.6 Gradient Boosting

Key concept: each model in the sequence predicts the pseudo-residuals (errors) of the previous model.

Mathematical formula — Gradient Boosting with 3 learners:

$$Y = F_1(x) + \eta F_2(x) + \eta F_3(x) + r_3$$

Where:

• $F_1(x)$ = prediction of Model 1
• $r_1$ = pseudo-residuals of Model 1 (what Model 1 did not capture)
• $\eta$ = learning rate (controls the contribution of each model)
• $r_i$ = gradient of the loss function with respect to the model’s prediction

Learning sequence:

Model 1:  y  = F₁(x) + r₁        ← learns target y
Model 2:  r₁ = F₂(x) + r₂        ← learns Model 1 errors
Model 3:  r₂ = F₃(x) + r₃        ← learns Models 1+2 errors
...
Model N:  r_{N-1} = F_N(x) + r_N  ← learns remaining errors

What is a pseudo-residual?

A pseudo-residual is not a raw error. It is the negative gradient of the loss function with respect to the model’s prediction:

$$r_i = -\frac{\partial \mathcal{L}(y, F(x))}{\partial F(x)}$$

• For regression: loss = MSE (Mean Squared Error)
• For classification: loss = Log Loss

Gradient descent visualization:

Loss ▲
     │    ╭─────╮
     │   /       \
     │  /         \       ← Gradient = slope of the ascent
     │ /           \  ╭──
     │/             ╰─╯   ← Loss minimum = objective
     └─────────────────────► Parameters
          ↑
          Descend in the direction of the negative gradient

In practice:

• 100 to 200 weak learners, each learning from the errors of the previous
• Each weak learner is a shallow decision tree
• The ensemble constitutes a powerful strong learner

AdaBoost vs. Gradient Boosting Differences:

1.7 Gradient Boosting Variants

graph TD
    GB[Gradient Boosting] --> XGB[XGBoost\neXtreme Gradient Boosting]
    GB --> LGB[LightGBM\nLight Gradient Boosting Machine]
    GB --> CB[CatBoost\nCategorical Boosting]

    XGB --> X1[1st and 2nd derivative\nof the loss function]
    XGB --> X2[L1 + L2 Regularization]
    XGB --> X3[Histogram-based\nsplits]
    XGB --> X4[Feature-level\nparallelization]
    XGB --> X5[Native handling\nof missing values]

    LGB --> L1[Leaf-wise\ntree growth]
    LGB --> L2[Discretized\nhistograms]
    LGB --> L3[GOSS - Gradient-based\nOne-Side Sampling]
    LGB --> L4[Exclusive Feature\nBundling EFB]
    LGB --> L5[Native categorical\nsupport]

    CB --> C1[Native categorical\nhandling]
    CB --> C2[Ordered Boosting]
    CB --> C3[Strong out-of-the-box\nperformance]
    CB --> C4[Symmetric oblivious\ntrees]

XGBoost — Key Innovations:

LightGBM — Key Innovations:

CatBoost — Key Innovations:

Module 2 — Implementing Boosting Algorithms

2.1 XGBoost — Classification on UCI Bank Dataset

Objective: Predict whether a bank customer will subscribe to a term deposit.
Dataset: UCI Bank Marketing Dataset (~45,000 records)

Feature Descriptions:

⚠️ Important note: The duration feature (duration of the last call) is highly predictive but should not be used in a realistic model, as it is only known after the call.

Step 1 — Loading and preprocessing data:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

import xgboost as xgb

# Load data
bank_data = pd.read_csv("uci_bank_data.csv")

# Drop the duration column (biased feature)
bank_data.drop(columns=['duration'], inplace=True)

# Encode target to numeric values (XGBoost only accepts numbers)
bank_data['target'] = bank_data['y'].map({'no': 0, 'yes': 1})
bank_data.drop(columns=['y'], inplace=True)

# Encode categorical variables as pandas 'category' type
# (required for native XGBoost support)
categorical_variables = bank_data.select_dtypes(exclude=np.number).columns.tolist()
for col in categorical_variables:
    bank_data[col] = bank_data[col].astype('category')

# Split features / target
X = bank_data.drop('target', axis=1)
y = bank_data['target']

# Train / test split (70% / 30%, stratified)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y)

Automatic categorical encoding in XGBoost:

• Required: pandas DataFrame with category type columns
• Required: use tree_method='hist' or 'gpu_hist'
• Required: enable_categorical=True
• Available since XGBoost 1.3+

Step 2 — Create and train the XGBoost model:

xgb_clf = xgb.XGBClassifier(
    booster='gbtree',           # Learner type (decision trees)
    n_estimators=300,           # Number of boosting rounds (trees)
    enable_categorical=True,    # Native categorical support
    objective='binary:logistic',# Objective: binary classification
    tree_method='hist',         # Fast histogram-based algorithm
    eval_metric='auc',          # Evaluation metric: AUC
    random_state=42
)

xgb_clf.fit(X_train, y_train)

Key XGBClassifier Parameters:

Step 3 — Evaluation and class imbalance handling:

# Basic predictions
y_pred = xgb_clf.predict(X_test)

print("Accuracy:", round(accuracy_score(y_test, y_pred), 4))
print("Precision:", round(precision_score(y_test, y_pred), 4))
print("Recall:", round(recall_score(y_test, y_pred), 4))
print("F1 Score:", round(f1_score(y_test, y_pred), 4))

# Calculate ratio to compensate for class imbalance
from collections import Counter
counts = Counter(y_train)
scale = counts[0] / counts[1]  # negative / positive class ratio

# Retrain with scale_pos_weight to improve recall
xgb_clf = xgb.XGBClassifier(
    booster='gbtree',
    n_estimators=300,
    enable_categorical=True,
    objective='binary:logistic',
    tree_method='hist',
    eval_metric='auc',
    scale_pos_weight=scale,   # ← compensates class imbalance
    random_state=42
)
xgb_clf.fit(X_train, y_train)

Step 4 — Feature Importance Visualization:

importances = xgb_clf.feature_importances_
features = X.columns

feature_importance_df = pd.DataFrame({
    'Feature': features,
    'Importance': importances
}).sort_values(by='Importance', ascending=False)

plt.figure(figsize=(8, 6))
plt.barh(feature_importance_df['Feature'], feature_importance_df['Importance'])
plt.xlabel("Importance Score")
plt.title("Feature Importance — XGBoost")
plt.gca().invert_yaxis()
plt.show()

2.2 XGBoost — Regularization, Early Stopping and Cross-validation

Regularization Parameters

gamma — Split threshold:

• Defines the minimum loss improvement required for a split to be performed
• High value → fewer splits, simpler trees → reduces overfitting
• Low value → deeper and more complex trees → overfitting risk

reg_lambda (L2):

• Ridge regularization on leaf weights
• High value → smaller weights, less overfitting
• Low value → leaf scores can grow freely

reg_alpha (L1):

• Lasso regularization on leaf weights
• High value → automatic feature selection (weights → 0)
• Low value → all features can contribute freely

def train_evaluate_model(params):
    model = xgb.XGBClassifier(
        booster='gbtree',
        n_estimators=300,
        enable_categorical=True,
        objective='binary:logistic',
        tree_method='hist',
        eval_metric='auc',
        scale_pos_weight=scale,
        random_state=42,
        **params
    )
    model.fit(X_train, y_train)
    preds = model.predict(X_test)
    return (
        accuracy_score(y_test, preds),
        precision_score(y_test, preds),
        recall_score(y_test, preds),
        f1_score(y_test, preds)
    )

params_list = [
    # Baseline (very low regularization)
    {"gamma": 0, "reg_lambda": 1, "reg_alpha": 0},
    # Regularization via L1 and L2
    {"gamma": 0.1, "reg_lambda": 2, "reg_alpha": 1},
    # Balanced L1/L2 with moderate gamma
    {"gamma": 0.3, "reg_lambda": 2, "reg_alpha": 0.5}
]

Early Stopping with the Native XGBoost API

# Split training / validation / test
X_train_es, X_val, y_train_es, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42, stratify=y_train)

# Create DMatrix objects (native XGBoost format)
dtrain = xgb.DMatrix(X_train_es, label=y_train_es, enable_categorical=True)
dval   = xgb.DMatrix(X_val,      label=y_val,      enable_categorical=True)
dtest  = xgb.DMatrix(X_test,     label=y_test,     enable_categorical=True)

params = {
    'booster': 'gbtree',
    'objective': 'binary:logistic',
    'eval_metric': 'auc',
    'tree_method': 'hist',
    'scale_pos_weight': scale,
    'random_state': 42
}

# Train with early stopping
model = xgb.train(
    params,
    dtrain,
    num_boost_round=1000,            # Maximum number of rounds
    evals=[(dval, 'validation')],    # Validation set
    early_stopping_rounds=10,        # Stop if no improvement for 10 rounds
    verbose_eval=True
)

# Predict and evaluate
y_pred_es = model.predict(dtest)
y_pred_es = [round(value) for value in y_pred_es]

Note: early_stopping_rounds monitors the metric on the validation set. If it does not improve for N consecutive rounds, training is stopped.

Cross-validation with XGBoost

dtrain = xgb.DMatrix(X_train, label=y_train, enable_categorical=True)

params = {
    'booster': 'gbtree',
    'objective': 'binary:logistic',
    'eval_metric': 'auc',
    'tree_method': 'hist',
    'scale_pos_weight': scale,
    'random_state': 42
}

# 5-fold cross-validation
results = xgb.cv(
    params,
    dtrain,
    nfold=5,           # Number of folds
    num_boost_round=30
)

results.head()

2.3 XGBoost on GPU (Google Colab)

Prerequisites: Google Colab with Runtime → Change runtime type → T4 GPU (free)

GPU Libraries:

Dataset: Covertype (>500,000 records, 54 features) — predicts forest cover type

import time
import cupy as cp
from cuml.model_selection import train_test_split
from sklearn.datasets import fetch_covtype
import xgboost as xgb

# Load data
X, y = fetch_covtype(return_X_y=True)

# Convert to GPU arrays
X = cp.array(X)
y = cp.array(y)

# Adjust labels (0-based indexing)
y -= y.min()  # From 1-7 → 0-6

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42)

# Training on CPU
clf_cpu = xgb.XGBClassifier(device="cpu", n_estimators=1000,
                              objective='multi:softprob')
start = time.time()
clf_cpu.fit(X_train, y_train, eval_set=[(X_test, y_test)])
print(f"CPU time: {time.time() - start:.2f}s")

# Training on GPU (cuda)
clf_gpu = xgb.XGBClassifier(device="cuda", n_estimators=1000,
                              objective='multi:softprob')
start = time.time()
clf_gpu.fit(X_train, y_train, eval_set=[(X_test, y_test)])
print(f"GPU time: {time.time() - start:.2f}s")

The device="cuda" parameter automatically enables GPU acceleration in XGBoost.

2.4 LightGBM — Classification on HR Analytics

Objective: Predict whether an employee will be promoted.
Dataset: HR Analytics (~55,000 records)

Dataset characteristics:

• Contains missing values (education, previous_year_rating)
• Highly imbalanced (few promoted employees)
• Categorical variables: department, region, education, gender, recruitment_channel

LightGBM, like XGBoost, handles missing values natively — during tree construction, missing values are treated as a separate category.

import lightgbm as lgb

# Load and preprocess
hr_data = pd.read_csv("hr_analytics.csv")
hr_data.drop("employee_id", axis=1, inplace=True)  # No predictive value

# Mark categoricals as pandas 'category' type
categorical_cols = ['department', 'region', 'education', 'gender', 'recruitment_channel']
for col in categorical_cols:
    hr_data[col] = hr_data[col].astype('category')

# Split features / target / train / test
X = hr_data.drop('is_promoted', axis=1)
y = hr_data['is_promoted']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y)

# Train a LightGBM classifier
lgbm_clf = lgb.LGBMClassifier(
    objective='binary',
    learning_rate=0.05,
    num_leaves=31,      # Controls tree complexity
    n_estimators=100,
    random_state=42,
    verbose=-1
)
lgbm_clf.fit(X_train, y_train)

Evaluation function:

from sklearn.metrics import roc_auc_score

def compute_metrics(clf):
    y_pred = clf.predict(X_test)
    y_pred_proba = clf.predict_proba(X_test)[:, 1]

    print("Accuracy:",  round(accuracy_score(y_test, y_pred), 4))
    print("Precision:", round(precision_score(y_test, y_pred), 4))
    print("Recall:",    round(recall_score(y_test, y_pred), 4))
    print("F1 Score:",  round(f1_score(y_test, y_pred), 4))
    print("ROC-AUC:",   round(roc_auc_score(y_test, y_pred_proba), 4))

2.5 LightGBM — Max Bins, Imbalanced Data and Early Stopping

max_bin Parameter

LightGBM uses a histogram-based algorithm to accelerate training:

• Instead of testing all possible split values, continuous features are grouped into discrete bins
• Only bin boundaries are tested as split points

lgbm_clf = lgb.LGBMClassifier(
    objective='binary',
    learning_rate=0.05,
    num_leaves=31,
    max_bin=400,        # Higher = more precise splits, but slower
    n_estimators=100,
    random_state=42,
    verbose=-1
)

Handling Imbalanced Classes

lgbm_clf = lgb.LGBMClassifier(
    objective='binary',
    learning_rate=0.05,
    num_leaves=31,
    n_estimators=500,
    class_weight='balanced',  # Adjusts weights inversely to frequencies
    random_state=42,
    verbose=1
)
lgbm_clf.fit(X_train, y_train)

class_weight='balanced' automatically adjusts class weights inversely proportional to their frequencies.

Early Stopping with LightGBM

# Create a validation set
X_train_es, X_val, y_train_es, y_val = train_test_split(
    X_train, y_train, test_size=0.30, random_state=42, stratify=y_train)

lgbm_clf = lgb.LGBMClassifier(
    objective='binary',
    learning_rate=0.05,
    num_leaves=31,
    n_estimators=3000,
    class_weight='balanced',
    random_state=42,
    verbose=1
)

lgbm_clf.fit(
    X_train_es, y_train_es,
    eval_metric='f1',
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(5)]  # Stop if no improvement for 5 rounds
)

LightGBM Parallelization

LightGBM can parallelize training in two ways:

2.6 LightGBM — Native API, GOSS and Feature Importances

Native LightGBM API

The native API is more flexible and powerful than the scikit-learn API. Ideal for:

• Large datasets
• Fine-grained training control
• Custom metrics and objective functions

Important: The predict() method of the native API returns probabilities, not labels. A threshold must be applied manually.

def compute_metrics_native(gbm):
    y_pred_probs = gbm.predict(X_test)
    y_pred = (y_pred_probs >= 0.5).astype(int)  # Threshold of 0.5

    print("Accuracy:",  round(accuracy_score(y_test, y_pred), 4))
    print("Precision:", round(precision_score(y_test, y_pred), 4))
    print("Recall:",    round(recall_score(y_test, y_pred), 4))
    print("F1 Score:",  round(f1_score(y_test, y_pred), 4))

# Create LightGBM Dataset objects
# reference=lgb_train → reuses histogram bins from the training set
lgb_train = lgb.Dataset(X_train_es, y_train_es)
lgb_val   = lgb.Dataset(X_val,      y_val,   reference=lgb_train)
lgb_test  = lgb.Dataset(X_test,     y_test,  reference=lgb_train)

# Model parameters (dict format)
params = {
    "boosting_type": "gbdt",        # Gradient Boosted Decision Trees
    "objective": "binary",
    "metric": "auc",
    "num_leaves": 31,
    "feature_fraction": 0.9,        # 90% of features per tree (anti-overfitting)
    "bagging_fraction": 0.8,        # 80% of rows per round (row subsampling)
    "is_unbalance": True,           # Adjusts weights for imbalanced classes
    "random_state": 42,
    "verbose": 1,
}

# Train with early stopping
gbm = lgb.train(
    params, lgb_train,
    num_boost_round=2000,
    valid_sets=[lgb_val],
    callbacks=[lgb.early_stopping(stopping_rounds=10)]
)

GOSS — Gradient-based One-Side Sampling

GOSS accelerates training by focusing on the most informative instances:

• Keeps all instances with large gradients (informative)
• Randomly samples instances with small gradients (less informative)

params = {
    "boosting_type": "gbdt",
    "data_sample_strategy": "goss",  # ← Enable GOSS
    "objective": "binary",
    "metric": "auc",
    "num_leaves": 31,
    "feature_fraction": 0.9,
    "bagging_fraction": 0.8,
    "is_unbalance": True,
    "random_state": 42,
    "verbose": 1,
}

gbm = lgb.train(params, lgb_train, num_boost_round=2000,
                valid_sets=[lgb_val],
                callbacks=[lgb.early_stopping(stopping_rounds=10)])

Feature Importances with the Native API

# Importance by number of splits (Split)
importance_split = gbm.feature_importance(importance_type='split')

# Importance by total gain (Gain)
importance_gain = gbm.feature_importance(importance_type='gain')

features = lgbm_clf.booster_.feature_name()

# Visualization
df_split = pd.DataFrame({'Feature': features, 'Split': importance_split}
                        ).sort_values('Split', ascending=False)

df_gain = pd.DataFrame({'Feature': features, 'Gain': importance_gain}
                       ).sort_values('Gain', ascending=False)

Gain is generally more representative of true importance as it measures the actual impact on performance.

2.7 CatBoost — Regression on the Insurance Dataset

Objective: Predict insurance charges.

Features:

⚠️ CatBoost rule: Never manually encode categorical columns (one-hot, label encoding) before passing them to CatBoost — this negates the advantages of its native handling.

from catboost import CatBoostRegressor, Pool
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

insurance_data = pd.read_csv('insurance.csv')

X = insurance_data.drop('charges', axis=1)
y = insurance_data['charges']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

# Identify categorical features (type 'object')
cat_features = X_train.select_dtypes(include='object').columns.tolist()

# CatBoost expects strings for categoricals (no NaN)
for col in cat_features:
    X_train[col] = X_train[col].astype(str).fillna('nan')
    X_test[col]  = X_test[col].astype(str).fillna('nan')

# Create Pool objects (native CatBoost format)
train_pool = Pool(X_train, y_train, cat_features=cat_features)
test_pool  = Pool(X_test,  y_test,  cat_features=cat_features)

# Train the regression model
model = CatBoostRegressor(
    iterations=200,
    learning_rate=0.1,
    depth=6,
    loss_function='RMSE',
    verbose=20
)
model.fit(train_pool)

Regression metrics evaluation:

def compute_metrics(model):
    y_pred = model.predict(test_pool)

    mae  = mean_absolute_error(y_test, y_pred)
    mse  = mean_squared_error(y_test, y_pred)
    rmse = np.sqrt(mse)
    r2   = r2_score(y_test, y_pred)

    print(f"MAE:  {mae:.2f}")
    print(f"MSE:  {mse:.2f}")
    print(f"RMSE: {rmse:.2f}")
    print(f"R²:   {r2:.4f}")

    # Actual vs Predicted chart
    plt.scatter(y_test, y_pred, alpha=0.5)
    plt.xlabel('Actual Charges')
    plt.ylabel('Predicted Charges')
    plt.title('Actual vs. Predicted')
    plt.plot([y_test.min(), y_test.max()], [y_test.min(), y_test.max()], 'r--')
    plt.show()

2.8 CatBoost — Early Stopping, Feature Importances and Snapshots

Early Stopping

# Split train / validation / test
X_train_es, X_val, y_train_es, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42)

train_pool = Pool(X_train_es, y_train_es, cat_features=cat_features)
val_pool   = Pool(X_val,      y_val,      cat_features=cat_features)
test_pool  = Pool(X_test,     y_test,     cat_features=cat_features)

# Model with early stopping
model_es = CatBoostRegressor(
    iterations=300,
    learning_rate=0.05,
    depth=6,
    loss_function='RMSE',
    bootstrap_type='Bernoulli', # Stochastic sampling type
    subsample=0.8,              # 80% of rows per tree
    od_type='Iter',             # Early stopping criterion by iterations
    od_wait=40,                 # Wait 40 rounds without improvement
    verbose=50
)

model_es.fit(train_pool, eval_set=val_pool)

Early stopping parameters:

Note: Even without explicit early stopping, CatBoost automatically shows the best model and can reduce the ensemble to the best number of iterations.

Feature Importances in CatBoost

# PredictionValuesChange — impact on prediction values
importances_pvc = model.get_feature_importance(
    train_pool, type='PredictionValuesChange')

# LossFunctionChange — impact on the loss function
importances_lfc = model.get_feature_importance(
    train_pool, type='LossFunctionChange')

features = X_train.columns

# Visualization
plt.figure(figsize=(10, 5))
plt.bar(features, importances_pvc)
plt.title('Feature Importance (PredictionValuesChange)')
plt.xticks(rotation=45)
plt.show()

Snapshots (resuming training)

CatBoost supports saving snapshots to resume training in case of interruption:

params = {
    'iterations': 5,
    'learning_rate': 0.5,
    'depth': 6,
    'loss_function': 'RMSE',
    'bootstrap_type': 'Bernoulli',
    'subsample': 0.8,
    'logging_level': 'Verbose'
}

# First training run — saves the snapshot
model = CatBoostRegressor(**params).fit(
    train_pool, eval_set=val_pool, save_snapshot=True)

# Resume training with more iterations
params.update({'iterations': 20, 'learning_rate': 0.1})
model = CatBoostRegressor(**params).fit(
    train_pool, eval_set=val_pool, save_snapshot=True)

The catboost_info/ folder is created in the working directory to track training progress.

Module 3 — Tuning and Interpretability

3.1 Hyperparameter Tuning

┌─────────────────────────────────────────────────────────────────┐
│                          MODEL                                   │
├──────────────────┬──────────────────────┬───────────────────────┤
│  Inputs          │  Internal Parameters │  Hyperparameters       │
│                  │                      │                        │
│ Training         │ Coefficients,        │ Tree depth,            │
│ data             │ intercepts           │ learning rate, alpha   │
│                  │                      │ regularization,        │
│ → what the       │ → learned during     │ number of estimators   │
│   model receives │   training           │                        │
│                  │                      │ → defined BEFORE       │
│                  │                      │   training             │
└──────────────────┴──────────────────────┴───────────────────────┘

Types of hyperparameter search:

graph LR
    HT[Hyperparameter Tuning] --> GS[Grid Search\nTests all\ncombinations]
    HT --> RS[Random Search\nRandom\nsampling]
    HT --> BO[Bayesian Optimization\nLearns from\nprevious results]

    GS --> GS1[Exhaustive but expensive]
    RS --> RS1[Faster, good for\nlarge spaces]
    BO --> BO1[Most efficient\nfor large spaces]

Important hyperparameters for XGBoost:

3.2 GridSearchCV with XGBoost

from sklearn.model_selection import GridSearchCV
from xgboost import XGBClassifier

# Preprocess the HR Analytics dataset
hr_data = pd.read_csv('hr_analytics.csv')
hr_data.drop("employee_id", axis=1, inplace=True)

cat_features = hr_data.select_dtypes(include='object').columns.tolist()
for col in cat_features:
    hr_data[col] = hr_data[col].astype('category')

X = hr_data.drop('is_promoted', axis=1)
y = hr_data['is_promoted']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y)

# Base model with default parameters
xgb_clf = XGBClassifier(
    enable_categorical=True,
    objective='binary:logistic',
    tree_method='hist',
    eval_metric='auc',
    random_state=42
)
xgb_clf.fit(X_train, y_train)

# Hyperparameter grid to test
param_grid = {
    'learning_rate':    [0.3, 0.01],
    'max_depth':        [2, 5, 10],
    'gamma':            [0, 0.1, 1],
    'reg_alpha':        [0, 0.1],
    'reg_lambda':       [1, 2],
    'scale_pos_weight': [2, 8, 10],  # To compensate class imbalance
    'n_estimators':     [200, 500, 800]
}

# GridSearchCV — optimizing recall
grid = GridSearchCV(
    estimator=XGBClassifier(
        enable_categorical=True,
        objective='binary:logistic',
        tree_method='hist',
        eval_metric='auc',
        random_state=42
    ),
    param_grid=param_grid,
    cv=5,              # 5-fold cross-validation
    scoring='recall',  # Optimize recall
    n_jobs=-1,         # Use all available CPUs
    return_train_score=True
)

grid.fit(X_train, y_train)

print("Best parameters:", grid.best_params_)

Objective: Maximize recall — we want to identify as many positive cases (promotions) as possible, even if that means more false positives.

3.3 SHAP Values

SHAP = SHapley Additive exPlanations

Quantifies the contribution of each feature to a specific prediction, drawing on cooperative game theory (Shapley values).

Game Theory Analogy

🎮 The prediction = a team game
👥 Team members = model features
💰 The gain = the difference between the prediction and the baseline

SHAP answers: "What is the fair share of contribution
               of each feature in this gain?"

Concrete example — House price prediction

Baseline prediction (average): $300,000
Final prediction for this house: $450,000
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Feature      │ SHAP Contribution
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Size         │ +$100,000 ↑
Age          │ -$200,000 ↓
Quality/Grade│ +$250,000 ↑
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Sum          │ +$150,000 = $450,000 ✓

Properties of SHAP Values

• Takes into account the impact of a feature across all possible combinations of other features
• Fairly handles feature interactions (non-linear effects)
• Provides per-instance (individual) and global (across all data) explanations

Types of SHAP Metrics

3.4 Demo — Computing SHAP Values with LightGBM

import shap

# Update libraries (avoid version incompatibilities)
# pip install --upgrade numpy shap lightgbm

Step 1 — Prepare and train the model:

# Load and preprocess the UCI Bank dataset
bank_data = pd.read_csv("uci_bank_data.csv")
bank_data.drop(columns=['duration'], inplace=True)

bank_data['target'] = bank_data['y'].map({'no': 0, 'yes': 1})
bank_data.drop(columns=['y'], inplace=True)

categorical_variables = bank_data.select_dtypes(exclude=np.number).columns.tolist()
for col in categorical_variables:
    bank_data[col] = bank_data[col].astype('category')

X = bank_data.drop('target', axis=1)
y = bank_data['target']
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.30, random_state=42, stratify=y)

# Train the LightGBM model
lgbm_clf = lgb.LGBMClassifier(random_state=42)
lgbm_clf.fit(X_train, y_train, categorical_feature=categorical_variables)

Step 2 — Compute SHAP values:

# TreeExplainer: fast and precise explainer for tree-based models
explainer = shap.TreeExplainer(lgbm_clf.booster_)

# Compute SHAP values on training data
shap_values = explainer.shap_values(X_train)

# Result shape: (31647, 15)
# - 31647 = number of samples in X_train
# - 15 = number of features
print(shap_values.shape)  # (31647, 15)

# SHAP values for a single record
print(shap_values[:1])

Step 3 — SHAP vs. LightGBM Feature Importance:

mean_abs_shap = np.abs(shap_values).mean(axis=0)

shap_importance = pd.DataFrame({
    'Feature': X_train.columns,
    'Mean_Absolute_SHAP': mean_abs_shap,
    'LightGBM_Importance': lgbm_clf.booster_.feature_importance()
}).sort_values(by='Mean_Absolute_SHAP', ascending=False)

print(shap_importance)

Key insight: The contact feature has the highest SHAP value (strong impact on predictions) even though it is not the most frequently used feature by LightGBM. Conversely, balance is often used (high LightGBM importance) but has a lower SHAP impact per prediction.

Step 4 — SHAP Visualizations:

Summary Plot

shap.summary_plot(shap_values, X_train)

Summary Plot interpretation:
- Each row = a feature
- X axis = SHAP value (centered on 0)
- Blue = low feature values
- Red = high feature values
- Width of dispersion = strength of impact on predictions

Dependence Plot

# Explore the relationship between balance and predictions
shap.dependence_plot('balance', shap_values, X_train)

Dependence Plot interpretation for 'balance':
- X axis = value of the 'balance' feature
- Y axis = SHAP value for 'balance'
- Color = contact type (cellular, telephone, unknown)
- Observation: as balance increases (0→20,000),
  SHAP values also increase
  Beyond 20,000: flatter or noisier effect

Waterfall Plot (individual prediction)

shap.waterfall_plot(shap.Explanation(
    values=shap_values[0],
    base_values=explainer.expected_value,
    data=X_train.iloc[0]
))

Waterfall Plot interpretation:
- Right vertical line (~-2.5) = model average output (baseline)
- Left vertical line (f(x) = -3.081) = actual model prediction
  (in log-odds score)
- Each bar = feature contribution
  - Blue = moves away from the average
  - Red = moves closer to the average
- The 'balance' feature has the greatest impact here

4. Appendix — Algorithm Comparison

XGBoost / LightGBM / CatBoost Comparison Table

When to use which algorithm?

flowchart TD
    Q1{Categorical\nfeatures?} -->|Yes, many| CB[CatBoost\nBest native handling]
    Q1 -->|A few| Q2
    Q1 -->|No| Q2

    Q2{Speed/memory\nconstraints?} -->|Yes| LGB[LightGBM\nFaster, less memory]
    Q2 -->|No| Q3

    Q3{Need fine\nregularization?} -->|Yes| XGB[XGBoost\nMore L1/L2/gamma control]
    Q3 -->|No| Q4

    Q4{Very large\ndataset?} -->|Yes| LGB2[LightGBM\nGOSS + EFB]
    Q4 -->|No| CB2[CatBoost\nOut-of-the-box]

Quick Reference — Versions and Installation

# Versions used in this course
pip install scikit-learn==1.6.1
pip install xgboost==3.0.0
pip install lightgbm==4.6.0
pip install catboost==1.23.5
pip install shap
pip install graphviz

Quick Reference — XGBoost Regularization Parameters

gamma      → split threshold (0 = no threshold, ∞ = never split)
reg_lambda → L2 (ridge): reduces weights without zeroing them
reg_alpha  → L1 (lasso): can set weights exactly to 0
max_depth  → maximum tree depth
subsample  → fraction of rows per tree (< 1 = bagging)
colsample_bytree → fraction of columns per tree

Resources and References

• XGBoost: https://xgboost.readthedocs.io
• LightGBM: https://lightgbm.readthedocs.io
• CatBoost: https://catboost.ai
• SHAP: https://shap.readthedocs.io
• UCI Bank Dataset: https://archive.ics.uci.edu/dataset/222/bank+marketing
• XGBoost GPU Demo (Covertype): https://github.com/dmlc/xgboost/blob/master/demo/gpu_acceleration/cover_type.py
• Google Colab: https://colab.research.google.com

Course based on “Boosting Techniques” by Janani Ravi — Loonycorn

Search Terms

machine · boosting · techniques · ml · fundamentals · engineering · data · science · lightgbm · xgboost · early · stopping · api · catboost · feature · importances · native · shap · plot · regularization · values · algorithm · algorithms · classification