Machine Learning Model Development

Table of Contents

1. Course Overview
2. Module 1 — Model Development
   • 2.1 Prerequisites
   • 2.2 ML Model Types
   • 2.3 Python Environment Setup
   • 2.4 UCI Adult Dataset
   • 2.5 Training and Comparing Models — demo-1-1.py
   • 2.6 Stress Test and Profiling — demo-1-2.py & demo-2-2.py
   • 2.7 Cold Deserialization — demo-2-3.py
   • 2.8 Memory Profile — demo-2-1.py
   • 2.9 Module 1 Summary
3. Module 2 — Deploying the ML Model
   • 3.1 Deployment Constraints
   • 3.2 Deployment Strategy
   • 3.3 Serialization to ONNX — demo-3-1.py
   • 3.4 FastAPI Microservice — main.py
   • 3.5 Docker Containerization — Dockerfile
   • 3.6 Testing the Deployment with curl
   • 3.7 Production Degradation Modes
   • 3.8 Load Testing with Locust — locustfile.py
   • 3.9 Module 2 Summary
4. Module 3 — Constrained Validation and Retraining
   • 4.1 Retraining Motivations
   • 4.2 Retraining Risks
   • 4.3 Validation Strategies
   • 4.4 Evaluation Metric — ROC AUC
   • 4.5 Validation Benchmark — demo-4-1.py
   • 4.6 Results and Comparative Analysis
   • 4.7 Module 3 Summary
5. Overall Project Architecture

1. Course Overview

This course aims to give you hands-on experience with Machine Learning (ML) models. The key concepts covered are:

• Developing a simple ML model
• Evaluating performance metrics
• Deploying a model in a portable container
• Validating and retraining the model

Prerequisites: basic knowledge of Python and Linux. The provided scripts are designed for Linux but can be adapted to Windows.

Total course duration:
  Module 1 — Model Development               : 17m 28s
  Module 2 — Deploying a ML Model            : 18m 19s
  Module 3 — Constrained Validation          :  6m 36s

2. Module 1 — Model Development

2.1 Prerequisites

• Basic understanding of Python
• Basic understanding of Linux

2.2 ML Model Types

Before developing an ML model, you need to understand what types of models are available and which best fits the use case. The general principle is: the more complex a model, the more capable it is — but also the more computationally expensive.

graph LR
    A[Increasing Complexity] --> B[Linear / Logistic Models]
    B --> C[Decision Trees\nRandom Forests\nGradient Boosted Trees]
    C --> D[SVM - Support Vector Machines]
    D --> E[MLP - Multi-Layer Perceptrons]
    E --> F[CNN - Convolutional Neural Networks]
    F --> G[Transformers]

    style A fill:#e8f4f8,stroke:#2980b9
    style B fill:#d5f5e3,stroke:#27ae60
    style C fill:#d5f5e3,stroke:#27ae60
    style D fill:#fef9e7,stroke:#f39c12
    style E fill:#fdebd0,stroke:#e67e22
    style F fill:#fadbd8,stroke:#e74c3c
    style G fill:#f9ebea,stroke:#c0392b

Note: For this course, we will use Logistic Regression, Random Forest, and MLP — a good balance between performance and simplicity for tabular data.

Selection Tip

General rule for tabular data with low-to-moderate variance:
  Minimum data points = 20 × number of parameters (features)

2.3 Python Environment Setup

# Install all required libraries in a single command
pip install psutil joblib pandas memory-profiler numpy scikit-learn

2.4 UCI Adult Dataset

The UCI Adult Dataset (also known as “Census Income”) is used throughout the course. It is publicly available online and contains socio-economic tabular data.

Dataset columns:

Class imbalance: only ~24% of individuals earn more than $50,000/year.

2.5 Training and Comparing Models — demo-1-1.py

This script loads the UCI Adult dataset, trains three models, measures their performance, and saves them to disk.

import pandas as pd
import joblib
import os
import time
import numpy as np
import warnings

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.exceptions import ConvergenceWarning

# ── Load UCI Adult dataset ────────────────────────────────────────────────────
df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None,
    names=[
        "age", "workclass", "fnlwgt", "education", "education-num",
        "marital-status", "occupation", "relationship", "race", "sex",
        "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"
    ]
)

# Cleanup: replace missing values "?" with NaN
df.replace(" ?", np.nan, inplace=True)
df.dropna(inplace=True)

X = df.drop("income", axis=1)
y = df["income"].apply(lambda x: x.strip() == ">50K")

# ── Identify column types ─────────────────────────────────────────────────────
cat_cols = X.select_dtypes(include="object").columns.tolist()
num_cols = X.select_dtypes(exclude="object").columns.tolist()

# ── Train/test split ──────────────────────────────────────────────────────────
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify=y, test_size=0.2, random_state=42
)

# ── Preprocessing ─────────────────────────────────────────────────────────────
# One-hot encoding for categoricals, StandardScaler for numerics
preprocessor = ColumnTransformer([
    ("onehot", OneHotEncoder(handle_unknown="ignore"), cat_cols),
    ("scale", StandardScaler(), num_cols)
])

# ── Model definitions ─────────────────────────────────────────────────────────
models = {
    "LogisticRegression": LogisticRegression(max_iter=2000, solver="lbfgs", n_jobs=-1),
    "RandomForest":       RandomForestClassifier(n_estimators=200, n_jobs=-1),
    "MLPClassifier":      MLPClassifier(hidden_layer_sizes=(128, 64), max_iter=400)
}

results = {}

for name, model in models.items():
    pipeline = make_pipeline(preprocessor, model)

    # Training with timing
    start_time = time.perf_counter()
    pipeline.fit(X_train, y_train)
    fit_time = time.perf_counter() - start_time

    # Save to disk
    filename = f"{name}.joblib"
    joblib.dump(pipeline, filename)
    model_size_kb = os.path.getsize(filename) / 1024

    # Inference latency (1000 calls on a single row)
    start_time = time.perf_counter()
    for _ in range(1000):
        pipeline.predict(X_test.iloc[[0]])  # 2D input required
    latency_ms = (time.perf_counter() - start_time) / 1000 * 1000

    # Accuracy
    y_pred = pipeline.predict(X_test)
    acc = accuracy_score(y_test, y_pred)

    results[name] = {
        "Train Time (s)":   round(fit_time, 3),
        "Latency (ms)":     round(latency_ms, 3),
        "Model Size (KB)":  round(model_size_kb, 1),
        "Accuracy":         round(acc * 100, 2)
    }

# ── Display results ───────────────────────────────────────────────────────────
print("\nModel Benchmark Results:\n")
print("{:<18} {:>15} {:>15} {:>18} {:>12}".format(
    "Model", "Train Time (s)", "Latency (ms)", "Model Size (KB)", "Accuracy (%)"
))
print("-" * 80)
for model_name, stats in results.items():
    print("{:<18} {:>15} {:>15} {:>18} {:>12}".format(
        model_name,
        stats["Train Time (s)"],
        stats["Latency (ms)"],
        stats["Model Size (KB)"],
        stats["Accuracy"]
    ))

Preprocessing pipeline:

flowchart LR
    A[Raw Data\nUCI Adult CSV] --> B[Cleanup\nreplace '?'→NaN\ndropna]
    B --> C[Split X / y\nincome >50K = True]
    C --> D{Column type}
    D -->|Categorical| E[OneHotEncoder\nhandle_unknown='ignore']
    D -->|Numeric| F[StandardScaler]
    E --> G[ColumnTransformer]
    F --> G
    G --> H[make_pipeline\n+ Model]
    H --> I[pipeline.fit\nX_train, y_train]
    I --> J[pipeline.predict\nX_test]
    J --> K[accuracy_score\njoblib.dump]

Example expected results:

Model Benchmark Results:

Model              Train Time (s)    Latency (ms)  Model Size (KB) Accuracy (%)
--------------------------------------------------------------------------------
LogisticRegression          2.1xx          0.8xx            xxx.x        85.xx
RandomForest               12.xxx          1.2xx          xxxxx.x        86.xx
MLPClassifier              35.xxx          1.8xx            xxx.x        85.xx

2.6 Stress Test and Profiling — demo-1-2.py & demo-2-2.py

Single-core stress test — demo-1-2.py

In a real deployment, each inference instance is typically pinned to a single thread or logical core. This script simulates this scenario with 1,000 calls on a single CPU core.

import os
import time
import joblib
import psutil
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

# ── Reload data ───────────────────────────────────────────────────────────────
df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None,
    names=[
        "age", "workclass", "fnlwgt", "education", "education-num",
        "marital-status", "occupation", "relationship", "race", "sex",
        "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"
    ]
)
df.replace(" ?", np.nan, inplace=True)
df.dropna(inplace=True)

X = df.drop("income", axis=1)
y = df["income"].apply(lambda x: x.strip() == ">50K")
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify=y, test_size=0.2, random_state=42
)

# ── Load saved model ──────────────────────────────────────────────────────────
model_file = "MLPClassifier.joblib"
if not os.path.exists(model_file):
    print(f"Error: {model_file} not found. Run training script first.")
    exit(1)

pipeline = joblib.load(model_file)
sample = X_test.iloc[[0]]  # keep DataFrame shape

# ── CPU info ──────────────────────────────────────────────────────────────────
cpu_list = psutil.cpu_count(logical=True)
print(f"\nAvailable logical CPUs: {cpu_list}")
print("Pinning to a single core for stress test...\n")

# ── Pin to a single CPU core ──────────────────────────────────────────────────
p = psutil.Process()
original_affinity = p.cpu_affinity()
p.cpu_affinity([0])  # Pin to CPU core 0

# ── 1000 inferences + timing ──────────────────────────────────────────────────
print("Running 1000 inferences on single CPU core...\n")
start_time = time.perf_counter()
for _ in range(1000):
    _ = pipeline.predict(sample)
elapsed = time.perf_counter() - start_time
latency_ms = (elapsed / 1000) * 1000

# ── Restore CPU affinity ──────────────────────────────────────────────────────
p.cpu_affinity(original_affinity)

# ── Results ───────────────────────────────────────────────────────────────────
print("===== Latency Results =====")
print(f"Model: {model_file}")
print(f"Total time for 1000 inferences (1 core): {elapsed:.3f} seconds")
print(f"Avg latency per inference: {latency_ms:.3f} ms")
print("===========================\n")

Typical result: ~1.8 ms per inference. Pinning to a single core avoids context switches and reduces overhead, despite the apparent resource limitation.

CPU profiling with cProfile — demo-2-2.py

To identify bottlenecks in the pipeline, use cProfile and pstats:

import cProfile
import pstats
import io
import joblib
import pandas as pd

# Reload data (same structure as before)
df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None,
    names=["age", "workclass", "fnlwgt", "education", "education-num",
           "marital-status", "occupation", "relationship", "race", "sex",
           "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"]
)
df.replace(" ?", pd.NA, inplace=True)
df.dropna(inplace=True)

X = df.drop("income", axis=1)
X = X.reset_index(drop=True)
X = X.iloc[:2000]  # Subset for fast profiling

# Load MLP model
mlp_model = joblib.load("MLPClassifier.joblib")

# Configure profiler
profiler = cProfile.Profile()
profiler.enable()

# Prediction
_ = mlp_model.predict(X)

# Stop profiler
profiler.disable()

# Display statistics (top 15 functions by cumulative time)
s = io.StringIO()
stats = pstats.Stats(profiler, stream=s).sort_stats("cumtime")
stats.print_stats(15)
print("\n=== CPU Hotspot Report ===\n")
print(s.getvalue())

Result: pipeline.predict leads, followed by preprocessing steps (OneHotEncoder, ColumnTransformer). This confirms that preprocessing can be a bottleneck — an argument for pre-computing or caching it.

2.7 Cold Deserialization — demo-2-3.py

This script measures the loading time (“cold-start”) of each model from disk:

import time
import joblib

models = {
    "Logistic Regression": "LogisticRegression.joblib",
    "Random Forest":       "RandomForest.joblib",
    "MLP Classifier":      "MLPClassifier.joblib"
}

print("\n=== Cold-Start Deserialization Times ===\n")
for name, path in models.items():
    start = time.perf_counter()
    _ = joblib.load(path)
    duration = time.perf_counter() - start
    print(f"{name:<20}: {duration:.4f} seconds")

Context: Cold-start time matters during the first model load in a container. A Random Forest with 200 trees will have a significantly longer cold-start than logistic regression.

2.8 Memory Profile — demo-2-1.py

from memory_profiler import memory_usage
import joblib
import pandas as pd

# Reload data
df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None,
    names=["age", "workclass", "fnlwgt", "education", "education-num",
           "marital-status", "occupation", "relationship", "race", "sex",
           "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"]
)
df.replace(" ?", pd.NA, inplace=True)
df.dropna(inplace=True)

X = df.drop("income", axis=1)
X = X.reset_index(drop=True)
X = X.iloc[:2000]

# Load all models
models = {
    "LogReg":     joblib.load("LogisticRegression.joblib"),
    "RandForest": joblib.load("RandomForest.joblib"),
    "MLP":        joblib.load("MLPClassifier.joblib")
}

# Wrapper for prediction
def profile_predict(model, X):
    return model.predict(X)

# Measure peak memory for each model
mem_results = {}
for name, model in models.items():
    mem_usage = memory_usage((profile_predict, (model, X)), max_usage=True)
    mem_results[name] = round(mem_usage, 2)

# Display
print("\nPeak Memory Usage During Inference (MB):\n")
for name, mem in mem_results.items():
    print(f"{name:<10} {mem} MB")

2.9 Module 1 Summary

mindmap
  root((Module 1))
    Model Selection
      Guided by data type
      Logistic Regression
      Random Forest
      MLP Classifier
    Training Metrics
      Train Time in seconds
      Model Size in KB
      Accuracy in percent
    Inference Metrics
      Average latency in ms
      Stress test on 1 core
      cProfile profiling
      Peak memory

3. Module 2 — Deploying the ML Model

3.1 Deployment Constraints

Three constraints guided the technology choices:

┌─────────────────────────────────────────────────────────────┐
│  DEPLOYMENT CONSTRAINTS                                     │
├─────────────────────────┬───────────────────────────────────┤
│  No proprietary         │  Avoid the costs of proprietary   │
│  services               │  cloud services                   │
├─────────────────────────┼───────────────────────────────────┤
│  Managed via pip / apt  │  Simple environment control       │
├─────────────────────────┼───────────────────────────────────┤
│  Portability via        │  Containerization for portability │
│  containerization       │  across host systems              │
└─────────────────────────┴───────────────────────────────────┘

Containerization limitation: A large-scale LLM would be too large for a container. Containers also integrate poorly with GPUs and multi-node scale-out.

3.2 Deployment Strategy

flowchart LR
    A[Trained model\n.joblib] -->|demo-3-1.py\nskl2onnx| B[logreg.onnx\nONNX Format]
    B -->|main.py\nFastAPI + onnxruntime| C[HTTP Microservice\nPOST /predict]
    C -->|Dockerfile| D[Docker Image]
    D -->|docker run| E[Deployed container\nPort 8000]
    E -->|curl / locust| F[Users]

    style A fill:#d5f5e3,stroke:#27ae60
    style B fill:#d6eaf8,stroke:#2980b9
    style C fill:#fdebd0,stroke:#e67e22
    style D fill:#f9ebea,stroke:#c0392b
    style E fill:#f4ecf7,stroke:#8e44ad

Serialization Options

3.3 Serialization to ONNX — demo-3-1.py

Serialization is the process of converting an in-memory object (e.g., an active scikit-learn model) into a portable representation that can be stored on disk or transmitted over a network. ONNX (Open Neural Network Exchange) is an open, hardware-agnostic format.

import joblib
import pandas as pd
from skl2onnx import convert_sklearn
from skl2onnx.common.data_types import StringTensorType, FloatTensorType

# ── Load data to retrieve columns ─────────────────────────────────────────────
df = pd.read_csv(
    "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",
    header=None,
    names=[
        "age", "workclass", "fnlwgt", "education", "education-num",
        "marital-status", "occupation", "relationship", "race", "sex",
        "capital-gain", "capital-loss", "hours-per-week", "native-country", "income"
    ]
)
df.dropna(inplace=True)
X = df.drop("income", axis=1)

# ── Build initial_types for ONNX ──────────────────────────────────────────────
# ONNX needs to know the type and shape of each input feature
initial_types = []
for col in X.columns:
    if X[col].dtype == object:
        initial_types.append((col, StringTensorType([None, 1])))
    else:
        initial_types.append((col, FloatTensorType([None, 1])))

# ── Load trained pipeline ─────────────────────────────────────────────────────
pipe = joblib.load("LogisticRegression.joblib")

# ── Convert to ONNX ───────────────────────────────────────────────────────────
onnx_model = convert_sklearn(pipe, initial_types=initial_types)

# ── Save ──────────────────────────────────────────────────────────────────────
with open("logreg.onnx", "wb") as f:
    f.write(onnx_model.SerializeToString())

print("ONNX model exported successfully to logreg.onnx")

3.4 FastAPI Microservice — main.py

This file creates an HTTP endpoint POST /predict that accepts individual data as JSON and returns the probability that their income exceeds $50,000/year.

from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
import onnxruntime as ort
import joblib, os, re

app = FastAPI(title="Income Prediction API")

# ── Load model (ONNX by default, joblib as fallback) ─────────────────────────
USE_ONNX = os.getenv("USE_ONNX", "1") == "1"
if USE_ONNX:
    sess = ort.InferenceSession("logreg.onnx", providers=["CPUExecutionProvider"])
    ONNX_INPUTS = [i.name for i in sess.get_inputs()]
else:
    pipe = joblib.load("LogisticRegression.joblib")

# ── Sanitize column names (skl2onnx rules) ───────────────────────────────────
def sanitise(col: str) -> str:
    """
    Apply exact skl2onnx rules:
    - Replace any character outside [A-Za-z0-9_] with '_'
    - Collapse multiple '__' to '_'
    - Strip leading/trailing '_'
    """
    col = re.sub(r"[^0-9A-Za-z_]", "_", col)
    col = re.sub(r"__+", "_", col)
    return col.strip("_")

SANITISED_LOOKUP = {c: sanitise(c) for c in [
    "age", "workclass", "fnlwgt", "education", "education-num",
    "marital-status", "occupation", "relationship", "race", "sex",
    "capital-gain", "capital-loss", "hours-per-week", "native-country"
]}

class Record(BaseModel):
    data: dict  # original column names

# ── Prediction endpoint ───────────────────────────────────────────────────────
@app.post("/predict")
def predict(rec: Record):
    if USE_ONNX:
        # Build the dictionary expected by ONNX
        input_dict = {}
        for raw_key, val in rec.data.items():
            onnx_key = SANITISED_LOOKUP.get(raw_key)
            if onnx_key is None:
                raise ValueError(f"Unexpected key: {raw_key}")
            arr = np.array([[val]], dtype=np.float32 if isinstance(val, (int, float)) else np.str_)
            input_dict[onnx_key] = arr

        # Check for missing keys
        missing = set(ONNX_INPUTS) - input_dict.keys()
        if missing:
            raise ValueError(f"Missing keys for ONNX: {missing}")

        # ONNX inference — probability of class 1
        proba = float(sess.run(None, input_dict)[1][0][1])
    else:
        import pandas as pd
        row = pd.DataFrame([rec.data])
        proba = float(pipe.predict_proba(row)[0][1])

    return {"p_gt_50k": proba}

Example payload (payload.json):

{
  "data": {
    "age": 39,
    "workclass": "State-gov",
    "fnlwgt": 77516,
    "education": "Bachelors",
    "education-num": 13,
    "marital-status": "Never-married",
    "occupation": "Adm-clerical",
    "relationship": "Not-in-family",
    "race": "White",
    "sex": "Male",
    "capital-gain": 2174,
    "capital-loss": 0,
    "hours-per-week": 40,
    "native-country": "United-States"
  }
}

3.5 Docker Containerization — Dockerfile

# Lightweight Python image
FROM python:3.11-slim

# Working directory inside the container
WORKDIR /app

# Install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application files
COPY main.py .
COPY logreg.onnx .

# Environment variable to use ONNX
ENV USE_ONNX=1

# Start FastAPI server via Uvicorn
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

requirements.txt:

fastapi
uvicorn[standard]
onnxruntime
numpy
pydantic
joblib
psutil>=5.9.0

Docker commands:

# Build the image
docker build -t income-predictor .

# Launch the container
docker run -p 8000:8000 income-predictor

# Verify the service is running
docker ps

3.6 Testing the Deployment with curl

curl -s -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "data": {
      "age": 39,
      "workclass": "State-gov",
      "fnlwgt": 77516,
      "education": "Bachelors",
      "education-num": 13,
      "marital-status": "Never-married",
      "occupation": "Adm-clerical",
      "relationship": "Not-in-family",
      "race": "White",
      "sex": "Male",
      "capital-gain": 2174,
      "capital-loss": 0,
      "hours-per-week": 40,
      "native-country": "United-States"
    }
  }' \
  http://localhost:8000/predict; echo

Expected response:

{"p_gt_50k": 0.3316}

This means the model estimates a 33.16% probability that this individual earns more than $50,000/year, given their age, education, occupation, etc.

curl command breakdown:

-s                          → Silent mode (suppresses progress)
-X POST                     → HTTP POST method
-H "Content-Type: app/json" → Header indicating JSON format
-d '{...}'                  → Request body (individual data)
; echo                      → Bash trick: adds a newline after the JSON

3.7 Production Degradation Modes

In a production environment, several issues can affect the performance of a deployed model:

graph TD
    A[Production Degradation] --> B[Latency Spikes]
    A --> C[Memory Blow-Up]
    A --> D[CPU Oversubscription]
    A --> E[Cold Start Drag]

    B --> B1[Causes: I/O bottlenecks,\nInference queuing,\nNoisy neighbors]
    C --> C1[Causes: Memory leaks,\nBatch too large,\nRepeated loading]
    D --> D1[Causes: Too many workers,\nConcurrent processes]
    E --> E1[Causes: Model and preprocessor\nloading from disk]

    B1 --> F[Mitigation Strategies]
    C1 --> F
    D1 --> F
    E1 --> F

    F --> G[Model Shunting\nRerouting the model]
    F --> H[Request Queuing]
    F --> I[Timeouts]

3.8 Load Testing with Locust — locustfile.py

Locust allows simulating thousands of simultaneous users to test load capacity of the deployed service.

from locust import HttpUser, task, between
import json

# Load payload from JSON file
with open('payload.json', 'r') as f:
    sample_payload = json.load(f)

class IncomePredictUser(HttpUser):
    wait_time = between(0.1, 0.3)  # Wait between 100ms and 300ms between requests

    @task
    def predict_income(self):
        self.client.post("/predict", json=sample_payload)

Launch the load test:

locust -f locustfile.py --host=http://localhost:8000
# Access the web interface: http://localhost:8089

3.9 Module 2 Summary

Module 2 Recap:
  ✅  Developed a FastAPI microservice based on our logistic regression model
  ✅  Deployed the microservice via Docker
  ✅  Tested the service functionality (curl + locust)

4. Module 3 — Constrained Validation and Retraining

4.1 Retraining Motivations

ML models in production are rarely static. In the background, new input data is continuously collected and models are periodically retrained. The main motivations are:

graph LR
    subgraph Benefits
        A[Increased Capability]
        B[Improved Accuracy]
        C[Reduced Model Overhead]
    end

    subgraph Mechanism
        D[Expanded dataset]
        E[Better coverage\nof edge cases]
        F[Hyperparameter\noptimization]
    end

    D --> A
    E --> B
    F --> C

4.2 Retraining Risks

graph TD
    R[Retraining Risks] --> D[Data Drift]
    R --> C[Computational Expense]
    R --> U[User Experience]

    D --> D1[Input data may be\nbiased by time or selection]
    C --> C1[Unsuccessful retraining\nwastes time and money]
    U --> U1[A degraded model deployed\ncauses service disruption]
    U1 --> U2[Requires new retraining\nand another disruption]

Solution: Validation allows determining model performance on unseen data before any deployment.

4.3 Validation Strategies

Module 3 explores four validation strategies, each representing a trade-off between statistical purity and engineering pragmatism:

graph LR
    subgraph Standard Strategies
        SK[Stratified k-Fold\nk=5 folds with\nclass balance]
        KF[Plain k-Fold\nk=5 folds without\nstratification]
        TS[Time Series Split\nSequential validation\nwithout data leakage]
    end

    subgraph Custom Strategy
        BP[Blocked Progressive\nSliding window\nuntil stabilization]
    end

    SK -->|"Ideal for small\nimbalanced datasets"| R[Results]
    KF -->|"Slightly cheaper,\nclasses already balanced"| R
    TS -->|"Essential if\nobservations have order"| R
    BP -->|"Optimal in production\nfor this use case"| R

Blocked Progressive Algorithm

1. Initialize: start = 0, block = 2000
2. WHILE (start + 2 × block ≤ total):
   a. Train on [start : start+block]
   b. Evaluate on [start+block : start+2×block]
   c. Compute ROC AUC
   d. IF |AUC_current - AUC_previous| < 0.001: STOP (stabilization)
   e. start += block
3. Return mean of AUC scores

4.4 Evaluation Metric — ROC AUC

ROC AUC (Area Under the Receiver Operating Characteristic Curve) is the chosen metric to compare all strategies:

Question asked by ROC AUC: “Did you rank all samples in the correct probability order?” — rather than “Did you classify this sample correctly?”

$$\text{AUC} = P(\hat{y}{pos} > \hat{y}{neg})$$

A model with an AUC of 0.9 means: “When predicting who earns more than 50K, I will rank a true positive above a true negative in 90% of cases.”

Advantages of ROC AUC for this problem:
  ✅  Threshold-agnostic
  ✅  Very useful when classes are imbalanced (~24% high-income)
  ✅  Measures the model's separation ability, not just accuracy

4.5 Validation Benchmark — demo-4-1.py

import os
import time
import joblib
import pandas as pd
from sklearn.model_selection import (
    StratifiedKFold, KFold, TimeSeriesSplit, cross_val_score
)
from sklearn.metrics import roc_auc_score

# ── 1. Download and prepare data ──────────────────────────────────────────────
URL  = "https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data"
COLS = [
    "age", "workclass", "fnlwgt", "education", "education-num",
    "marital-status", "occupation", "relationship", "race", "sex",
    "capital-gain", "capital-loss", "hours-per-week", "native-country",
    "income"
]

print("Fetching Adult data...")
df = pd.read_csv(
    URL, names=COLS, header=None,
    na_values=" ?", skipinitialspace=True
).dropna()

y = df["income"].str.contains(">50K").astype(int)
X = df.drop(columns="income")
print(f"Dataset ready — {len(X):,} rows, class balance: {y.mean():.2%} high-income\n")

# ── 2. Load pre-trained models ────────────────────────────────────────────────
FILES = {
    "LogReg":     "LogisticRegression.joblib",
    "RandForest": "RandomForest.joblib",
    "MLP":        "MLPClassifier.joblib"
}

models = {}
for name, path in FILES.items():
    if not os.path.exists(path):
        raise FileNotFoundError(f"Cannot find '{path}' in working directory.")
    models[name] = joblib.load(path)
    print(f"Loaded {name} pipeline from {path}")
print("")

# ── 3. Define validation strategies ──────────────────────────────────────────
splits = {
    "StratifiedK5": StratifiedKFold(n_splits=5, shuffle=True, random_state=0),
    "KFold5":       KFold(n_splits=5, shuffle=True, random_state=0),
    "TimeSeries3":  TimeSeriesSplit(n_splits=3, test_size=2000)
}

def blocked_progressive(X_df, y_ser, pipe, block=2000):
    """Sliding window validation until AUC stabilizes (delta < 0.001)."""
    scores, start = [], 0
    total = len(X_df)
    while start + 2 * block <= total:
        end_train = start + block
        end_test  = end_train + block
        X_tr, y_tr = X_df.iloc[start:end_train], y_ser.iloc[start:end_train]
        X_te, y_te = X_df.iloc[end_train:end_test], y_ser.iloc[end_train:end_test]
        pipe.fit(X_tr, y_tr)
        scores.append(roc_auc_score(y_te, pipe.predict_proba(X_te)[:, 1]))
        # Stop if stabilized
        if len(scores) > 2 and abs(scores[-1] - scores[-2]) < 0.001:
            break
        start += block
    return sum(scores) / len(scores)

# ── 4. Run benchmarks ─────────────────────────────────────────────────────────
rows = []

for m_name, pipe in models.items():
    # Standard strategies (cross-validation)
    for s_name, splitter in splits.items():
        t0  = time.perf_counter()
        auc = cross_val_score(
            pipe, X, y, cv=splitter, scoring="roc_auc", n_jobs=1
        ).mean()
        duration = time.perf_counter() - t0
        rows.append((m_name, s_name, round(auc, 4), round(duration, 2)))

    # Custom Blocked Progressive
    t0  = time.perf_counter()
    auc = blocked_progressive(X, y, pipe)
    duration = time.perf_counter() - t0
    rows.append((m_name, "BlockedProg", round(auc, 4), round(duration, 2)))

# ── 5. Display pivot table AUC + seconds ──────────────────────────────────────
df_results = pd.DataFrame(rows, columns=["Model", "Strategy", "AUC", "Seconds"])
df_pivot   = df_results.pivot(index="Model", columns="Strategy")

# Clean multi-index columns
df_pivot.columns = ['_'.join(col).strip() for col in df_pivot.columns.values]
df_pivot = df_pivot.reset_index()

with pd.option_context("display.width", None):
    print("\nAUC and Runtime (sec) by Model and Strategy\n")
    print(df_pivot)

4.6 Results and Comparative Analysis

Observed results:

xychart-beta
    title "AUC vs Time Trade-off (Blocked Progressive = optimal)"
    x-axis ["LogReg\nBlockedProg", "LogReg\nKFold5", "RandForest\nBlockedProg", "RandForest\nKFold5", "MLP\nBlockedProg", "MLP\nTimeSeries"]
    y-axis "Time (seconds)" 0 --> 400
    bar [1, 8, 1, 10, 5, 360]

Analysis by model:

• Logistic Regression: AUC ~0.90-0.91 for all strategies. Blocked Progressive finishes in ~1 second with near-zero performance loss compared to full k-Fold (2× faster).

• MLP: AUC slightly increases with deeper splits, but cost explodes. Stratified k-Fold takes nearly 6 minutes. While Time Series gives the best AUC, its extra cost over Blocked Progressive is not justified for marginally better performance.

• Random Forest: Robust AUC ~0.90+ for all strategies. Blocked Progressive achieves this in ~1 second vs 8-11 seconds for others — high efficiency without compromise.

Conclusion: For this workload, blocked_progressive is clearly the optimal choice. Other workloads — especially with different input/output data types — may require different strategies. Always explore multiple strategies before establishing a SOP (Standard Operating Procedure).

4.7 Module 3 Summary

Module 3 Recap:
  ✅  Defined validation and its motivations
  ✅  Discussed time and computational cost constraints
  ✅  Identified 4 validation strategies
  ✅  Applied all 4 strategies to the 3 pre-trained models
  ✅  Determined that Blocked Progressive is superior for this workload
  ✅  Recommended testing multiple strategies early in the model lifecycle

5. Overall Project Architecture

flowchart TD
    subgraph "Module 1 — Development"
        D1[(UCI Adult\nDataset)] --> P1[Preprocessing\nOneHotEncoder\nStandardScaler]
        P1 --> M1[LogisticRegression\n.joblib]
        P1 --> M2[RandomForest\n.joblib]
        P1 --> M3[MLPClassifier\n.joblib]
        M1 & M2 & M3 --> BENCH[Benchmark\nAccuracy / Latency\nMemory / Size]
    end

    subgraph "Module 2 — Deployment"
        M1 -->|demo-3-1.py\nskl2onnx| ONNX[logreg.onnx]
        ONNX -->|main.py\nFastAPI| API[POST /predict\n:8000]
        API -->|Dockerfile| DOCK[Docker Container]
        DOCK -->|curl / locust| USR[Users]
    end

    subgraph "Module 3 — Validation"
        D1 --> VAL[4-Strategy Benchmark\nStratifiedKFold\nKFold\nTimeSeries\nBlockedProgressive]
        M1 & M2 & M3 --> VAL
        VAL --> RPRT[AUC Report\n+ Execution Time]
    end

Project file structure:

pluralsight-machine-Learning-model-development/
├── demo-1-1.py          # Training + benchmark of 3 models
├── demo-1-2.py          # Stress test on 1 CPU core (psutil)
├── demo-2-1.py          # Memory profile (memory_profiler)
├── demo-2-2.py          # CPU profiling (cProfile + pstats)
├── demo-2-3.py          # Cold deserialization timing
├── demo-3-1.py          # Serialization to ONNX (skl2onnx)
├── demo-4-1.py          # 4-strategy validation benchmark
├── main.py              # FastAPI microservice (ONNX + joblib fallback)
├── locustfile.py        # Load testing (Locust)
├── payload.json         # Example payload for testing
├── Dockerfile           # Microservice containerization
└── requirements.txt     # Python dependencies

Search Terms

machine · model · development · ml · fundamentals · engineering · data · science · deployment · retraining · validation · demo-1-2.py · demo-2-2.py · profiling · serialization · stress · test · testing