Foundations of Machine Learning Engineering

Full Course — Maaike van Putten, Software Developer & Trainer

Table of Contents

• Module 1 — The Machine Learning Workflow
  • 1.1 Machine Learning Overview
  • 1.2 Data Collection and Preparation
  • 1.3 Model Development and Training
  • 1.4 Deployment and Monitoring
  • 1.5 Reproducibility: Model Versioning and Experiment Tracking
• Module 2 — Challenges in Machine Learning Engineering
  • 2.1 Performance Expectations
  • 2.2 Feature Engineering to Improve Model Performance
  • 2.3 Explainability and Bias
  • 2.4 Handling Drift and Degradation
• Running Case Study: TalentFlow
• Key Tools Summary

Module 1 — The Machine Learning Workflow

1.1 Machine Learning Overview

What is Machine Learning?

Machine learning consists of feeding a computer data and letting it produce predictions or decisions autonomously. At its core, almost everything boils down to a few fundamental tasks:

mindmap
  root((Machine Learning))
    Classification
      Spam detection
      Image recognition
    Regression
      House prices
      Task duration
    Clustering
      Customer segments
      Anomaly detection
    Recommendation
      Products
      Content

Machine Learning Engineering vs. Simple Modeling

Training a model with a good accuracy score is enough for a demo. In a real professional context, much more is required. Deploying machine learning means building and delivering systems that are:

graph TD
    A[ML System in Production] --> B[Accurate]
    A --> C[Fast]
    A --> D[Scalable]
    A --> E[Explainable]
    A --> F[Fair]
    A --> G[Maintainable]

Performance Metrics

Accuracy (Overall Accuracy)

$$\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$$

• Measures the percentage of correct predictions among all predictions
• Limitation: misleading when classes are imbalanced

Precision

$$\text{Precision} = \frac{TP}{TP + FP}$$

• Among all positive predictions, how many were actually positive?
• Question: “When the model predicts YES, is it right?”

Recall

$$\text{Recall} = \frac{TP}{TP + FN}$$

• Among all actually positive cases, how many did the model detect?
• Question: “Is the model missing important cases?”

F1 Score

$$\text{F1} = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

• Harmonic mean between precision and recall
• Useful for finding the balance between the two

quadrantChart
    title Precision vs Recall Trade-off
    x-axis Low Recall --> High Recall
    y-axis Low Precision --> High Precision
    quadrant-1 Ideal
    quadrant-2 Missing too many true positives
    quadrant-3 Unreliable
    quadrant-4 Too many false alarms
    Aircraft maintenance: [0.90, 0.85]
    Spam detection: [0.75, 0.92]
    Loan approval: [0.60, 0.88]

Example — Machine Maintenance:

Golden rule: For aircraft maintenance, very high recall is preferred (never miss a failure) even if it results in lower precision.

Other Expectations for an ML System in Production

• Latency: how quickly the model makes a prediction (critical for autonomous vehicles, intrusion detection, healthcare)
• Scalability: ability to handle growing volumes of requests, users, or data without degradation
• Fairness: are predictions equitable across groups? (hiring, loans, healthcare)
• Explainability: can we explain why the model made a decision?
• Reliability: can the system be trusted over time? Can results be reproduced?

The ML Lifecycle

flowchart LR
    A[Problem\nFormulation] --> B[Data\nCollection]
    B --> C[Data\nPreparation]
    C --> D[Model Training\n& Evaluation]
    D --> E[Model\nDeployment]
    E --> F[Monitoring &\nMaintenance]
    F --> G[Governance &\nReproducibility]
    G -.->|iterative| A
    D -.->|back if needed| C
    F -.->|back if drift| D

    style A fill:#4A90D9,color:#fff
    style B fill:#5BA85A,color:#fff
    style C fill:#5BA85A,color:#fff
    style D fill:#E8A838,color:#fff
    style E fill:#D95B5B,color:#fff
    style F fill:#9B59B6,color:#fff
    style G fill:#16A085,color:#fff

The steps do not proceed in a straight line. It is an iterative cycle: you often go back to adjust previous steps.

Lifecycle steps:

1. Problem Formulation — What exactly are we predicting? Why does it matter for the business?
2. Data Collection — What data is available and how to obtain it?
3. Data Preparation — Cleaning, transforming, labeling, and splitting the data
4. Model Training & Evaluation — Choose the algorithm, train it, validate its performance
5. Model Deployment — Put the model into production
6. Monitoring & Maintenance — Track performance, detect drift, update the model
7. Governance & Reproducibility — Track versions, experiments, and decisions for audit

1.2 Data Collection and Preparation

“Garbage in, garbage out” — bad input data produces bad output.

Data Collection

The goal is to identify relevant data sources for the problem and integrate them into the system.

Possible sources:

• APIs
• Internal databases
• Web scraping
• Third-party partners

For supervised learning, labeled data is required: each record contains both the input AND the expected output.

Data Preparation Phases

flowchart TD
    RAW[Raw Data] --> CLEAN[1. Cleaning\nRemove duplicates\nFix errors\nHandle missing values]
    CLEAN --> NORM[2. Normalization & Encoding\nUniform format\nScale normalization\nCategorical variable encoding]
    NORM --> FEAT[3. Feature Extraction\n& Transformation\nExtract useful signals\nCreate new features]
    FEAT --> PRIV[4. Sensitive Data Handling\nRemove or anonymize\npersonal identifiers]
    PRIV --> SPLIT[5. Splitting the Dataset\nTraining set / Validation set / Test set]

    style RAW fill:#888,color:#fff
    style CLEAN fill:#4A90D9,color:#fff
    style NORM fill:#5BA85A,color:#fff
    style FEAT fill:#E8A838,color:#fff
    style PRIV fill:#D95B5B,color:#fff
    style SPLIT fill:#9B59B6,color:#fff

Dataset Split

┌─────────────────────────────────────────────┐
│              FULL DATASET                   │
├──────────────────────┬──────────┬───────────┤
│    Training Set      │Validation│  Test Set │
│       (~70%)         │  (~15%)  │   (~15%)  │
│  Train the model     │  Tune    │ Evaluate  │
│                      │  hyp.    │  perf.    │
└──────────────────────┴──────────┴───────────┘

Code Example — Cleaning and Preparation with Pandas

import pandas as pd
from sklearn.model_selection import train_test_split

# Load data
df = pd.read_csv("employees.csv")

# 1. Cleaning — remove duplicates and handle missing values
df = df.drop_duplicates()
df["experience_years"] = df["experience_years"].fillna(df["experience_years"].median())
df["education"] = df["education"].fillna("Unknown")

# Anonymize personal data
df = df.drop(columns=["name", "birth_date", "photo_url"], errors="ignore")

# 2. Encoding — binary target variable
df["retained"] = (df["tenure_months"] >= 12).astype(int)

# 3. Feature extraction — lowercase text
df["job_title_clean"] = df["job_title"].str.lower().str.strip()

# 4. Splitting the dataset
X = df.drop(columns=["retained"])
y = df["retained"]

X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.30, random_state=42)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.50, random_state=42)

print(f"Training set: {len(X_train)} samples")
print(f"Validation set: {len(X_val)} samples")
print(f"Test set: {len(X_test)} samples")

Data Pipelines

To make data preparation reproducible and automated:

# Example of a simple pipeline with scikit-learn Pipeline
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer

numeric_features = ["experience_years", "interview_score"]
categorical_features = ["education", "department"]

numeric_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler())
])

categorical_transformer = Pipeline(steps=[
    ("imputer", SimpleImputer(strategy="constant", fill_value="missing")),
    ("onehot", OneHotEncoder(handle_unknown="ignore"))
])

preprocessor = ColumnTransformer(transformers=[
    ("num", numeric_transformer, numeric_features),
    ("cat", categorical_transformer, categorical_features)
])

⚠️ Each step of data preparation can introduce subtle problems. If data is not well prepared, the model may appear to work in development but silently fail in production.

1.3 Model Development and Training

Model Selection

Choosing the right model depends on:

• The type of problem (classification, regression, etc.)
• Data characteristics (size, type, quality)
• Trade-offs to accept (interpretability, performance, resources)

graph TD
    PROB[Problem Type] --> CLS{Classification?}
    CLS -->|Yes| LR[Logistic Regression\nsimple, interpretable]
    CLS -->|Yes| DT[Decision Tree\ninterpretable, fast]
    CLS -->|Yes| RF[Random Forest\nrobust, accurate]
    CLS -->|Yes| XGB[XGBoost / LightGBM\nvery high performance]
    CLS -->|Yes| SVM[Support Vector Machine\ngood for small datasets]
    CLS -->|Yes| NN[Neural Network\ncomplex, powerful]
    CLS -->|No, regression| LIN[Linear Regression]
    CLS -->|No, regression| TREE[Regression Tree / Forest]

    style LR fill:#5BA85A,color:#fff
    style DT fill:#5BA85A,color:#fff
    style RF fill:#4A90D9,color:#fff
    style XGB fill:#E8A838,color:#fff
    style SVM fill:#9B59B6,color:#fff
    style NN fill:#D95B5B,color:#fff

Hyperparameters

Hyperparameters are settings that control the behavior of the algorithm, but are not learned from data. They must be defined before training.

Example — Decision Tree:

from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import GridSearchCV

# Without tuning — default parameters
model = DecisionTreeClassifier(max_depth=5, min_samples_split=10)

# With GridSearchCV — searching for the best hyperparameter
param_grid = {
    "max_depth": [3, 5, 7, 10, None],
    "min_samples_split": [2, 5, 10],
    "min_samples_leaf": [1, 2, 4]
}

grid_search = GridSearchCV(
    DecisionTreeClassifier(random_state=42),
    param_grid,
    cv=5,              # 5-fold cross-validation
    scoring="f1",
    n_jobs=-1
)
grid_search.fit(X_train, y_train)
print(f"Best hyperparameters: {grid_search.best_params_}")
print(f"Best F1 score: {grid_search.best_score_:.4f}")

Overfitting vs. Underfitting

        Model Complexity
        
Low ←─────────────────────────→ High

┌────────────┐           ┌──────────────┐
│ UNDERFITTING│           │  OVERFITTING  │
│ (high bias) │           │(high variance)│
│ Model too   │           │ Model too     │
│ simple      │    ✅     │ complex       │
│ Misses the  │  BALANCE  │ Memorizes the │
│ patterns    │           │ training data │
└────────────┘           └──────────────┘

Cross-Validation

Technique for testing model robustness by training and testing on multiple different subsets of the data.

Full dataset
┌──────┬──────┬──────┬──────┬──────┐
│  F1  │  F2  │  F3  │  F4  │  F5  │
└──────┴──────┴──────┴──────┴──────┘

Fold 1: [TEST ][TRAIN][TRAIN][TRAIN][TRAIN]
Fold 2: [TRAIN][TEST ][TRAIN][TRAIN][TRAIN]
Fold 3: [TRAIN][TRAIN][TEST ][TRAIN][TRAIN]
Fold 4: [TRAIN][TRAIN][TRAIN][TEST ][TRAIN]
Fold 5: [TRAIN][TRAIN][TRAIN][TRAIN][TEST ]

Final score = average of the 5 scores

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np

model = RandomForestClassifier(n_estimators=100, max_depth=7, random_state=42)

# 5-fold cross-validation
scores = cross_val_score(model, X_train, y_train, cv=5, scoring="f1")

print(f"F1 scores per fold: {scores.round(4)}")
print(f"Mean F1: {scores.mean():.4f} (+/- {scores.std() * 2:.4f})")

Complete Example — Training and Evaluation

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import classification_report, confusion_matrix
import matplotlib.pyplot as plt
import seaborn as sns

# Training
model = GradientBoostingClassifier(
    n_estimators=200,
    max_depth=4,
    learning_rate=0.05,
    random_state=42
)
model.fit(X_train, y_train)

# Evaluation on the validation set
y_pred = model.predict(X_val)

print("=== Classification Report ===")
print(classification_report(y_val, y_pred, target_names=["Not retained", "Retained"]))

# Confusion matrix
cm = confusion_matrix(y_val, y_pred)
sns.heatmap(cm, annot=True, fmt="d", cmap="Blues",
            xticklabels=["Not retained", "Retained"],
            yticklabels=["Not retained", "Retained"])
plt.title("Confusion Matrix — Validation set")
plt.ylabel("Actual")
plt.xlabel("Predicted")
plt.show()

The best model is not only the one with the highest score, but the one that balances accuracy, latency, scalability, fairness, explainability, and reliability.

1.4 Deployment and Monitoring

Deployment

Deployment in machine learning engineering is the process of making a trained model accessible to whoever or whatever needs it.

graph LR
    subgraph Infrastructure
        D[Docker\nKubernetes]
        C[Cloud Services\nAWS SageMaker\nAzure ML\nGoogle Vertex AI]
    end

    CLIENT[Client / App] -->|HTTP request| API[REST API\n/predict]
    API --> MODEL[ML Model]
    MODEL -->|prediction| API
    API -->|JSON response| CLIENT

    MODEL --- D
    MODEL --- C

Inference Modes

Real-Time Inference

sequenceDiagram
    participant Recruiter
    participant API as REST API
    participant Model as ML Model

    Recruiter->>API: POST /score { resume: "..." }
    API->>Model: Preprocessing + prediction
    Model-->>API: Score = 0.87
    API-->>Recruiter: { "score": 0.87, "latency": "45ms" }
    Note over Recruiter,Model: Result in < 1 second

Characteristics:

• Low latency
• Instant response
• Often integrated in a user interface

Batch Inference

sequenceDiagram
    participant Scheduler as Scheduler (cron)
    participant Pipeline as Batch Pipeline
    participant Model as ML Model
    participant DB as Database

    Scheduler->>Pipeline: Weekly trigger (Friday 10pm)
    Pipeline->>DB: Retrieve N candidates
    DB-->>Pipeline: Candidate data
    Pipeline->>Model: Score all candidates
    Model-->>Pipeline: Computed scores
    Pipeline->>DB: Save results
    Note over Scheduler,DB: Processing outside peak hours

Characteristics:

• Processes large data volumes
• Scheduled execution (night, weekend)
• More economical (cheaper compute off-peak)

# Example of a simple REST API with Flask
from flask import Flask, request, jsonify
import joblib
import pandas as pd

app = Flask(__name__)

# Load model at startup
model = joblib.load("model_v2.pkl")
preprocessor = joblib.load("preprocessor_v2.pkl")

@app.route("/score", methods=["POST"])
def score_candidate():
    data = request.get_json()

    # Input validation
    required_fields = ["experience_years", "interview_score", "education"]
    for field in required_fields:
        if field not in data:
            return jsonify({"error": f"Missing field: {field}"}), 400

    # Preprocessing
    df = pd.DataFrame([data])
    X = preprocessor.transform(df)

    # Prediction
    score = model.predict_proba(X)[0][1]
    return jsonify({
        "score": round(float(score), 4),
        "decision": "recommended" if score >= 0.7 else "review",
        "model_version": "v2.1.0"
    })

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8080)

Monitoring

Monitoring consists of continuously observing data to verify whether the model is still performing well after deployment.

flowchart TD
    DEPLOY[Deployed Model] --> LOG[Logging\nInputs / Outputs / Latency / Errors]
    LOG --> MONITOR{Continuous Monitoring}

    MONITOR --> PD[Prediction Distribution\nAre scores drifting?]
    MONITOR --> DD[Data Drift\nDo inputs differ from training data?]
    MONITOR --> CD[Concept Drift\nHas the input→output relationship changed?]
    MONITOR --> LAT[Latency & Throughput\nAre performance targets being met?]

    PD & DD & CD & LAT --> ALERT{Alert triggered?}
    ALERT -->|Yes| RETRAIN[Retraining Pipeline]
    ALERT -->|No| MONITOR
    RETRAIN --> DEPLOY

Monitoring metrics:

# Example of data drift detection with Evidently AI
from evidently.report import Report
from evidently.metric_preset import DataDriftPreset

report = Report(metrics=[DataDriftPreset()])
report.run(
    reference_data=training_data,
    current_data=production_data_last_week
)
report.save_html("drift_report.html")

1.5 Reproducibility: Model Versioning and Experiment Tracking

Why Reproducibility is Crucial

In classical software engineering, version control is taken for granted. In machine learning, the same level of discipline is required, but the elements to track are more numerous:

mindmap
  root((ML Reproducibility))
    Code
      Git commit ID
      Library versions
      Dockerfile
    Data
      Dataset hash
      Data version
      Schema
    Model
      Model artifact
      Version tag
      Performance metrics
    Training
      Hyperparameters
      Random seed
      Training duration
    Environment
      Requirements.txt
      Conda env
      Docker image

Why reproducibility matters:

1. Know which model is running in production at any time
2. Reproduce past results for debugging or audits
3. Compare different experiments in a structured way
4. Demonstrate fairness or compliance in regulated environments
5. Roll back to a previous version if the new one causes problems

Model Versioning

Model v1.0.0  ──→  Model v1.1.0  ──→  Model v2.0.0
(Q1 Data)          (Q2 Data)           (Q3 Data)
(XGBoost)          (Tuned XGBoost)     (LightGBM)
F1 = 0.72          F1 = 0.78           F1 = 0.83
Deployed: Jan      Deployed: Apr       Deployed: Jul
         ↑                  ↑                  ↑
     Rollback if needed  Rollback if needed  Current

Experiment Tracking with MLflow

import mlflow
import mlflow.sklearn
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.metrics import f1_score, precision_score, recall_score

mlflow.set_experiment("talentflow-hiring-model")

with mlflow.start_run(run_name="GBT_v2_Q3_data"):
    # Log parameters
    params = {
        "n_estimators": 200,
        "max_depth": 4,
        "learning_rate": 0.05,
        "random_state": 42
    }
    mlflow.log_params(params)

    # Training
    model = GradientBoostingClassifier(**params)
    model.fit(X_train, y_train)

    # Log metrics
    y_pred = model.predict(X_val)
    mlflow.log_metrics({
        "f1_score": f1_score(y_val, y_pred),
        "precision": precision_score(y_val, y_pred),
        "recall": recall_score(y_val, y_pred)
    })

    # Log dataset used
    mlflow.log_param("dataset_hash", "sha256:abc123...")
    mlflow.log_param("dataset_version", "Q3_2024")

    # Save model with version
    mlflow.sklearn.log_model(
        model,
        artifact_path="model",
        registered_model_name="TalentFlowHiringModel"
    )

    print(f"Run ID: {mlflow.active_run().info.run_id}")

What TalentFlow tracks for each run:

Without versioning and experiment tracking, teams work blind. In high-stakes applications such as hiring, where decisions must be justified and validated, this is unacceptable.

Module 2 — Challenges in Machine Learning Engineering

2.1 Performance Expectations

What Companies Actually Expect

ML models are used in critical systems (hiring, fraud detection, healthcare). They must go beyond the prototype stage and meet concrete requirements:

graph TD
    PROD[ML System in Production] --> ROW1
    PROD --> ROW2

    subgraph ROW1[Technical Expectations]
        A[Accurate]
        B[Fast]
        C[Scalable]
    end

    subgraph ROW2[Ethical and Operational Expectations]
        D[Explainable]
        E[Fair]
        F[Maintainable]
    end

Inevitable Trade-offs

graph LR
    A[Maximum accuracy] <-->|trade-off| B[Maximum speed]
    B <-->|trade-off| C[Maximum fairness]
    C <-->|trade-off| D[Maximum explainability]
    D <-->|trade-off| A

Fairness & Compliance

Models must comply with legal and ethical standards:

• No discrimination against protected groups (race, gender, age)
• Compliance with GDPR
• Compliance with the European AI Act
• Passing internal compliance reviews for enterprise clients

Explainability

• Decision makers want to know why the model made a given prediction
• This builds trust and helps weigh results in final decisions
• In some jurisdictions, it is a legal requirement

Maintainability

• Systems evolve over time
• Models must be retrained when data changes
• The retraining process must be efficient and reliable
  • Data pipelines
  • Retraining workflows
  • Deployment procedures

2.2 Feature Engineering to Improve Model Performance

Definition

Feature engineering is the process of transforming raw data into meaningful inputs that increase the predictive power of ML models.

flowchart LR
    RAW[Raw Data\nResumes, scores, notes] -->|Feature Engineering| FEAT[Engineered\nFeatures]
    FEAT --> MODEL[More Performant\nML Model]

    subgraph Techniques
        ENC[Categorical\nEncoding]
        MISS[Handling\nMissing Values]
        NORM[Normalization\n& Scaling]
        CREATE[Creating\nNew Features]
        DIM[Dimensionality\nReduction]
    end
    RAW --> Techniques --> FEAT

Benefits of good feature engineering:

• Better accuracy: highlighting relevant patterns and relationships
• Faster convergence: simplifying the learning process
• Better interpretability: understanding the influence of features

1. Encoding Categorical Variables

Most ML algorithms cannot work directly with text.

import pandas as pd
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
from sklearn.compose import ColumnTransformer

df = pd.DataFrame({
    "department": ["HR", "Marketing", "Engineering", "HR", "Engineering"],
    "education": ["Bachelor", "Master", "PhD", "Master", "Bachelor"],
    "hired": [1, 0, 1, 1, 0]
})

# --- Label Encoding (ORDINAL variables only) ---
# Useful for education (Bachelor < Master < PhD)
label_map = {"Bachelor": 0, "Master": 1, "PhD": 2}
df["education_encoded"] = df["education"].map(label_map)

# --- One-Hot Encoding (NOMINAL variables) ---
# Useful for department (no order between HR, Marketing, Engineering)
df_encoded = pd.get_dummies(df, columns=["department"], prefix="dept")

print(df_encoded.head())

Comparison:

2. Handling Missing Values

from sklearn.impute import SimpleImputer, KNNImputer
import numpy as np

df = pd.DataFrame({
    "experience_years": [3, np.nan, 7, np.nan, 5],
    "interview_score": [8.5, 7.0, np.nan, 9.0, 6.5]
})

# Option 1: Drop rows with missing values
df_dropped = df.dropna()

# Option 2: Median imputation (robust to outliers)
imputer_median = SimpleImputer(strategy="median")
df["experience_years"] = imputer_median.fit_transform(df[["experience_years"]])

# Option 3: KNN imputation (uses nearest neighbors)
imputer_knn = KNNImputer(n_neighbors=3)
df_imputed = pd.DataFrame(
    imputer_knn.fit_transform(df),
    columns=df.columns
)
print(df_imputed)

3. Normalization and Scaling

from sklearn.preprocessing import MinMaxScaler, StandardScaler

data = pd.DataFrame({
    "experience_years": [1, 3, 5, 10, 15],     # Range: 1-15
    "interview_score": [5.5, 7.2, 8.8, 6.1, 9.5]  # Range: 5-10
})

# Min-Max Scaling — brings values into [0, 1]
min_max = MinMaxScaler()
data_minmax = pd.DataFrame(
    min_max.fit_transform(data),
    columns=data.columns
)

# Standardization — mean=0, std=1
standard = StandardScaler()
data_standard = pd.DataFrame(
    standard.fit_transform(data),
    columns=data.columns
)

print("Original :", data.values[0])
print("Min-Max  :", data_minmax.values[0])
print("Standard :", data_standard.values[0].round(3))

⚠️ Essential for distance-based algorithms: k-NN, SVM, neural networks.

4. Creating New Features

# TalentFlow example — creating derived features
import pandas as pd

df = pd.DataFrame({
    "application_date": pd.to_datetime(["2024-01-10", "2024-01-15", "2024-01-08"]),
    "job_posted_date": pd.to_datetime(["2024-01-01", "2024-01-01", "2024-01-01"]),
    "total_salary": [60000, 80000, 95000],
    "years_experience": [3, 5, 8],
    "resume_text": ["python machine learning data", "excel reporting management", "python java cloud"]
})

# Feature: days between job posting and application (interest signal)
df["days_to_apply"] = (df["application_date"] - df["job_posted_date"]).dt.days

# Feature: salary per year of experience
df["salary_per_year_exp"] = df["total_salary"] / df["years_experience"]

# Feature: presence of technical skills in the resume
tech_keywords = ["python", "java", "machine learning", "cloud", "sql"]
df["tech_skill_count"] = df["resume_text"].apply(
    lambda text: sum(1 for kw in tech_keywords if kw in text.lower())
)

print(df[["days_to_apply", "salary_per_year_exp", "tech_skill_count"]])

5. Dimensionality Reduction (PCA)

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

# When the dataset has many correlated features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_train)

pca = PCA(n_components=0.95)  # Keep 95% of variance
X_pca = pca.fit_transform(X_scaled)

print(f"Original dimensions: {X_train.shape[1]}")
print(f"Dimensions after PCA: {X_pca.shape[1]}")
print(f"Cumulative explained variance: {pca.explained_variance_ratio_.cumsum()[-1]:.2%}")

Tools for Feature Engineering

🔄 Feature engineering is iterative: try → train → evaluate → adjust → repeat.

2.3 Explainability and Bias

Explainability — Why It Is Essential

In regulated environments or high-impact decisions, explainability is not optional. People affected by a decision have the right to know why it was made.

graph TD
    EXP[Explainability] --> GLOBAL[Global Explainability\nHow does the model work\nin general?\nWhich features have the\nmost overall influence?]
    EXP --> LOCAL[Local Explainability\nWhy did the model make\nTHIS decision for\nTHIS specific case?\nOften required for compliance]

Explainability Tools

SHAP (SHapley Additive exPlanations)

SHAP decomposes a prediction and attributes each value to each input feature.

import shap
import joblib
import pandas as pd

# Load model
model = joblib.load("model_v2.pkl")

# Initialize SHAP explainer
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_val)

# --- GLOBAL Explainability ---
# Which features have the most influence overall?
shap.summary_plot(shap_values, X_val, feature_names=feature_names, plot_type="bar")

# --- LOCAL Explainability ---
# Explanation for a specific candidate (index 0)
shap.force_plot(
    explainer.expected_value,
    shap_values[0],
    X_val.iloc[0],
    feature_names=feature_names
)

LIME (Local Interpretable Model-agnostic Explanations)

LIME builds local interpretable models around individual predictions.

from lime.lime_tabular import LimeTabularExplainer

explainer_lime = LimeTabularExplainer(
    training_data=X_train.values,
    feature_names=feature_names,
    class_names=["Not retained", "Retained"],
    mode="classification"
)

# Explanation for a specific candidate
explanation = explainer_lime.explain_instance(
    data_row=X_val.iloc[0].values,
    predict_fn=model.predict_proba,
    num_features=10
)
explanation.show_in_notebook()

Bias — When the Model Produces Unfair Results

A model is biased if it produces systematically unfair results for certain groups.

Sources of bias:

graph TD
    BIAS[Sources of Bias] --> H[Historical inequalities\nin training data]
    BIAS --> S[Sensitive attributes\nthat influence the outcome\ne.g.: gender → performance]
    BIAS --> P[Proxy features\nNeutral in appearance\nbut correlated with\nsensitive attributes]

Types of bias:

Fairness Metrics

$$\text{Demographic Parity} = P(\hat{Y}=1 | A=0) = P(\hat{Y}=1 | A=1)$$

$$\text{Equal Opportunity} = P(\hat{Y}=1 | Y=1, A=0) = P(\hat{Y}=1 | Y=1, A=1)$$

from fairlearn.metrics import demographic_parity_difference, equalized_odds_difference
from fairlearn.reductions import ExponentiatedGradient, DemographicParity

# Evaluate current bias
dp_diff = demographic_parity_difference(
    y_true=y_val,
    y_pred=y_pred,
    sensitive_features=gender_val
)
print(f"Demographic parity gap: {dp_diff:.4f}")
# 0.0 = perfect, high values indicate bias

# Bias mitigation during training (in-processing)
mitigator = ExponentiatedGradient(
    estimator=GradientBoostingClassifier(),
    constraints=DemographicParity()
)
mitigator.fit(X_train, y_train, sensitive_features=gender_train)
y_pred_fair = mitigator.predict(X_val)

Techniques to improve fairness:

graph LR
    subgraph PRE[Pre-processing]
        R[Rebalancing\nthe data]
        A[Removing\nsensitive attributes]
    end
    subgraph IN[In-processing]
        C[Adding fairness\nconstraints during training]
    end
    subgraph POST[Post-processing]
        ADJ[Adjusting decision\nthresholds by group]
    end

    PRE --> IN --> POST

Explainability and bias require constant attention, particularly when models are updated or deployed in new environments.

2.4 Handling Drift and Degradation

The Challenge of Degradation Over Time

A model that performs well today will not necessarily perform well tomorrow. Without early detection, the consequences can be severe, especially in critical systems.

Data Drift vs. Concept Drift

graph TD
    DRIFT[Two Types of Drift] --> DD[Data Drift\nThe distribution of input\ndata changes over time\nEx: new resume formats,\nnew terminology]
    DRIFT --> CD[Concept Drift\nThe relationship between inputs and outputs\nchanges over time\nEx: skills that predict\nsuccess today ≠ in 1 year]

Consequences of Not Monitoring

Time ──────────────────────────────────────►

Performance
│
│ ██████████████████
│              ████████
│                   █████████
│                        ██████████
│                                 ██████████████
└─────────────────────────────────────────────►
Deployment  Week 4   Month 2    Month 4    Month 6
                                    ↑
                            "Someone noticed
                             something is
                             wrong..."

Response Strategies

flowchart TD
    MON[Continuous Monitoring\nPrediction distribution\nInput statistics\nModel accuracy] --> ALERT{Drift detected?}

    ALERT -->|No| MON
    ALERT -->|Yes| ACTION

    subgraph ACTION[Possible Actions]
        RETRAIN[Periodic\nretraining\nKnown schedule]
        TRIGGER[Alert-triggered\nretraining]
        HUMAN[Human review\nof prediction batches]
        FEEDBACK[Feedback loops\nNew ground-truth data]
    end

    ACTION --> NEWMODEL[New model\ntrained and evaluated]
    NEWMODEL --> COMPARE[Comparison\nwith previous version]
    COMPARE --> DEPLOY[Deployment\nof new version]
    DEPLOY --> MON

Feedback Loops

A feedback loop is the process of collecting new ground-truth data after a prediction has been made.

# Example: collecting and using feedback
import pandas as pd
from datetime import datetime

# Feedback table schema
feedback_schema = {
    "candidate_id": "string",
    "prediction_date": "datetime",
    "model_version": "string",
    "predicted_score": "float",          # Score predicted by the model
    "actual_retained_12m": "int",        # Reality: retained after 12 months? (0 or 1)
    "feedback_date": "datetime"          # Date when feedback was collected
}

# Using feedback for retraining
def prepare_retraining_data(feedback_df, original_features_df, min_samples=500):
    """
    Combines original features with feedback labels
    to create a new training dataset.
    """
    labeled_data = feedback_df.merge(
        original_features_df,
        on="candidate_id",
        how="inner"
    )

    if len(labeled_data) < min_samples:
        raise ValueError(f"Not enough labeled data: {len(labeled_data)} < {min_samples}")

    return labeled_data.dropna(subset=["actual_retained_12m"])

Recommended Retraining Frequency

# Example of an automated retraining pipeline
import mlflow
from datetime import datetime

def retraining_pipeline(drift_detected: bool, scheduled: bool = False):
    """
    Retraining pipeline triggered by drift or scheduling.
    """
    trigger = "drift_alert" if drift_detected else "scheduled"
    print(f"[{datetime.now()}] Retraining triggered: {trigger}")

    with mlflow.start_run(run_name=f"retrain_{trigger}_{datetime.now().strftime('%Y%m%d')}"):
        # 1. Collect fresh data
        fresh_data = load_fresh_training_data(lookback_months=3)
        mlflow.log_param("dataset_size", len(fresh_data))
        mlflow.log_param("trigger", trigger)

        # 2. Preprocessing (same steps as original version)
        X_new, y_new = preprocess(fresh_data)

        # 3. Training
        new_model = train_model(X_new, y_new)

        # 4. Evaluation and comparison with production model
        metrics = evaluate_model(new_model, X_test, y_test)
        mlflow.log_metrics(metrics)

        # 5. Deploy if improvement confirmed
        if metrics["f1_score"] > get_production_model_f1():
            deploy_model(new_model, version=get_next_version())
            mlflow.set_tag("deployed", "true")
            print(f"✅ New model deployed with F1={metrics['f1_score']:.4f}")
        else:
            print(f"⚠️ Current model retained. New F1={metrics['f1_score']:.4f}")

Drift is inevitable. Teams that succeed in machine learning engineering don’t just build good models — they build systems that adapt and evolve with the world.

Running Case Study: TalentFlow

TalentFlow is a fictional company building an intelligent recruitment assistant to help large enterprises evaluate and rank candidates.

journey
    title TalentFlow Application Journey
    section Data
      Client defines "good hire": 5: Client
      Collect resumes + interview notes: 4: TalentFlow
      Cleaning and anonymization: 4: TalentFlow
    section Model
      Feature engineering on resumes: 4: TalentFlow
      Model training: 5: TalentFlow
      Fairness validation: 4: TalentFlow
    section Deployment
      Deploy via REST API: 5: TalentFlow
      Real-time scoring: 5: Candidate
      SHAP explanation to recruiter: 4: Recruiter
    section Monitoring
      Drift monitoring: 4: TalentFlow
      Feedback after 12 months: 3: Client
      Retraining if needed: 4: TalentFlow

TalentFlow challenges summary by module:

Key Tools Summary

Tools by Category

mindmap
  root((ML Engineering Tools))
    Data Pipelines
      Apache Airflow
      dbt
      Prefect
      Dagster
    ML Frameworks
      Scikit-learn
      XGBoost / LightGBM
      TensorFlow / PyTorch
    Experiment Tracking
      MLflow
      Weights & Biases
      Neptune.ai
    Deployment
      Flask / FastAPI
      Docker
      Kubernetes
      AWS SageMaker
      Azure ML
      Google Vertex AI
    Monitoring
      Evidently AI
      WhyLabs
      Grafana + Prometheus
    Explainability
      SHAP
      LIME
    Fairness
      Fairlearn
      AIF360
    Feature Engineering
      Pandas
      Scikit-learn
      FeatureTools

Cheat Sheet — Essential Formulas

Notes generated from the “Foundations of Machine Learning Engineering” course by Maaike van Putten (Deck version 2024.04.e)

Search Terms

foundations · machine · engineering · ml · fundamentals · data · science · model · explainability · tools · inference · monitoring · performance · preparation · bias · collection · degradation · deployment · drift · essential · expectations · experiment · explanations · fairness