Deep Learning Frameworks and Model Implementation

Table of Contents

1. Course Overview
2. Module 1 — Why Frameworks?
   • 2.1 The Covertype Dataset
   • 2.2 Frameworks vs Manual Implementation
   • 2.3 PyTorch vs TensorFlow
   • 2.4 Hardware Acceleration — CPU and GPU
3. Module 2 — Building and Training Your First Model
   • 3.1 Defining a Network: Sequential API vs Functional API
   • 3.2 Preparing Data Correctly
   • 3.3 Inside the Training Loop
   • 3.4 Validation and Stopping Criteria
   • 3.5 Checkpoints, Logging and Experiment Tracking
4. Module 3 — Production-Ready Practices
   • 4.1 Mixed-Precision Training
   • 4.2 Structuring Code for Reuse
   • 4.3 Saving and Loading Models
   • 4.4 Versioning and Reproducibility
5. Quick Reference — Project Files

1. Course Overview

flowchart LR
    A([Start:\nunderstanding\nnetworks]) --> B[Module 1\nWhy\nframeworks?]
    B --> C[Module 2\nBuild &\ntrain]
    C --> D[Module 3\nProduction\npractices]
    D --> E([Result:\nfast, maintainable,\nreproducible\nmodel])

    style A fill:#f0f4ff,stroke:#7c9ef5
    style B fill:#e8f4e8,stroke:#5aab5a
    style C fill:#fff4e0,stroke:#e8a820
    style D fill:#fce8e8,stroke:#d45050
    style E fill:#f0f4ff,stroke:#7c9ef5

Most deep learning tutorials go too deep into theory, or at the opposite extreme, hand you a working script without ever explaining the decisions behind it. Neither approach prepares you to build models independently. This course teaches you to think like a practitioner:

Module 1 — Why Frameworks?

2.1 The Covertype Dataset

The problem solved throughout the course: predicting the forest cover type of a parcel of land in Roosevelt National Forest (Colorado) from cartographic data.

Dataset : UCI Forest Cover Type
Source   : Jock Blackhart & Dennis Dean, Colorado State University (1998)
Access   : sklearn.datasets.fetch_covtype()

Dataset characteristics

Breakdown of the 54 features:

10 continuous features (numeric)
├── Elevation
├── Slope
├── Aspect (orientation)
├── Horizontal distance to water
├── Vertical distance to water
├── Horizontal distance to roads
├── Horizontal distance to fire ignition points
└── 3 hillshade measurements at different times of day

44 binary features (one-hot encoded)
├── 4 wilderness area indicators
└── 40 soil type indicators

The 7 classes:

Class 0 — Spruce/Fir         (36 %)
Class 1 — Lodgepole Pine     (49 %)
Class 2 — Ponderosa Pine
Class 3 — Cottonwood/Willow
Class 4 — Aspen
Class 5 — Douglas-fir
Class 6 — Krummholz

pie title Class Distribution (approximate)
    "Lodgepole Pine" : 49
    "Spruce/Fir" : 36
    "Others (5 classes)" : 15

Loading the dataset:

import numpy as np
import pandas as pd
from sklearn.datasets import fetch_covtype

data = fetch_covtype()

# data.data  → feature matrix
# data.target → class labels (1 to 7)

X, y = data.data, data.target - 1  # shift to 0-6 for TensorFlow

print("Feature matrix shape:", X.shape)   # (581012, 54)
print("Labels shape:", y.shape)            # (581012,)

# Exploration in a DataFrame
df = pd.DataFrame(X, columns=data.feature_names)
df["target"] = y
df.describe()

Class distribution:

unique, counts = np.unique(y, return_counts=True)
class_names = ["Spruce/Fir", "Lodgepole Pine", "Ponderosa Pine",
               "Cottonwood/Willow", "Aspen", "Douglas-fir", "Krummholz"]

print("Class distribution:")
for cls, name, count in zip(unique, class_names, counts):
    print(f"  Class {cls} ({name}): {count:,} samples ({count/len(y)*100:.1f}%)")

2.2 Frameworks vs Manual Implementation

What a 100% manual implementation involves

Building a neural network with only Python and NumPy requires:

flowchart TD
    A[Manually initialize\nweights and biases] --> B[Write the forward pass\nmatrix × matrix,\nactivation functions]
    B --> C[Compute the loss]
    C --> D[Backpropagation\nmanually!\nDerive every operation]
    D --> E{Change\narchitecture?}
    E -- Yes --> D
    E -- No --> F[Write the optimizer\nby hand]
    F --> G[Debug sign errors\nin gradients 😰]

    style D fill:#fce8e8,stroke:#d45050
    style G fill:#fce8e8,stroke:#d45050

Problem: A single sign error in the gradient computation and the network silently stops learning. Hours of mathematical debugging instead of architectural experimentation.

What frameworks provide

mindmap
  root((Framework))
    Automatic Differentiation
      PyTorch → AutoGrad
      TensorFlow → GradientTape
    Pre-built Layers
      Dense
      Convolutional
      Normalization
    Ready-made Optimizers
      Adam
      SGD
    Loss Functions
    GPU Utilities
      No manual memory management
    Model Serialization

Concrete comparison:

2.3 PyTorch vs TensorFlow

quadrantChart
    title PyTorch vs TensorFlow
    x-axis "Deployment ecosystem →"
    y-axis "Flexibility / Debugging →"
    quadrant-1 TF 2.x (modern)
    quadrant-2 PyTorch
    quadrant-3 TF 1.x (legacy)
    quadrant-4 Keras standalone
    PyTorch: [0.35, 0.85]
    TensorFlow 2: [0.75, 0.65]
    Keras: [0.70, 0.55]

Flexibility and debugging

Example: automatic differentiation

PyTorch — AutoGrad:

import torch

x = torch.tensor([2.0], requires_grad=True)
y = x ** 3 + 2 * x
y.backward()
print(x.grad)  # dy/dx = 3x² + 2 = 14

TensorFlow — GradientTape:

import tensorflow as tf

x = tf.Variable([2.0])
with tf.GradientTape() as tape:
    y = x ** 3 + 2 * x
grad = tape.gradient(y, x)
print(grad)  # [14.]

Ecosystem and deployment

Course choice: TensorFlow with Keras. The Keras API is clean, the abstractions are well-designed, and all concepts, patterns and decisions translate directly to PyTorch.

Device selection (hardware abstraction)

# PyTorch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)

# TensorFlow
with tf.device('/GPU:0'):
    # operations here run on the first GPU
    pass

2.4 Hardware Acceleration — CPU and GPU

CPU vs GPU comparison

┌─────────────────────────────────────┬───────────────────────────────────────┐
│              CPU                    │               GPU                     │
├─────────────────────────────────────┼───────────────────────────────────────┤
│  General purpose                    │  Specialized (originally: 3D rendering)│
│  8 – 32 powerful cores              │  3,000 – 7,000 simple small cores     │
│  Complex logic & branching          │  Thousands of operations in parallel  │
│  OS, I/O, memory, etc.              │  Massive matrix multiplications       │
│  Efficient sequential execution     │  Massive parallelism                  │
└─────────────────────────────────────┴───────────────────────────────────────┘

Why neural networks leverage GPUs so well

flowchart LR
    A[Forward pass\nX × W layer\nafter layer] --> C{Can be\nparallelized?}
    B[Backpropagation\nmatrix operations\non gradients] --> C
    C -- Yes ✅ --> D[GPU\nThousands of simultaneous\noperations]
    C -- No ❌ --> E[CPU\nSequential execution]
    D --> F[⚡ 10x – 100x\nfaster]

    style D fill:#e8f4e8,stroke:#5aab5a
    style F fill:#e8f4e8,stroke:#5aab5a

Example time gains:

What happens under the hood

CPU RAM  ─────→  GPU VRAM  (transfer via PCIe)
                    │
             Matrix operations
             in parallel on GPU cores
                    │
             Gradients computed
                    │
             Weights updated

Key considerations:

• A GPU’s VRAM is limited (8 – 80 GB depending on model). Exceeding this limit → out-of-memory error.
• CPU → GPU transfer is not free: an inefficient data pipeline will keep the GPU waiting.
• This is why building an efficient data pipeline is as important as the model itself.

Frameworks abstract all of this: no CUDA code to write.

Module 2 — Building and Training Your First Model

3.1 Defining a Network: Sequential API vs Functional API

The model to build:

• Input: 54 features
• Output: 7 forest cover classes
• Architecture: stacked Dense layers with ReLU activations
• Output layer: 7 neurons with softmax

Sequential API

Simple and readable. Limitations: single input, single output, no branches or skip connections.

from tensorflow import keras

sequential_model = keras.Sequential([
    keras.layers.Input(shape=(54,)),
    keras.layers.Dense(256, activation="relu"),
    keras.layers.Dense(128, activation="relu"),
    keras.layers.Dense(64, activation="relu"),
    keras.layers.Dense(7, activation="softmax")
], name="covertype_sequential")

sequential_model.summary()

flowchart TD
    I["Input (54)"] --> D1["Dense 256 · ReLU"]
    D1 --> D2["Dense 128 · ReLU"]
    D2 --> D3["Dense 64 · ReLU"]
    D3 --> O["Dense 7 · Softmax"]

    style I fill:#dbeafe,stroke:#3b82f6
    style O fill:#dcfce7,stroke:#22c55e

Functional API

More explicit. Supports non-linear architectures: branches, skip connections, multiple inputs/outputs.

from tensorflow import keras

inputs = keras.Input(shape=(54,), name="covertype_input")

x = keras.layers.Dense(256, activation="relu")(inputs)
x = keras.layers.Dense(128, activation="relu")(x)
x = keras.layers.Dense(64, activation="relu")(x)

outputs = keras.layers.Dense(7, activation="softmax", name="covertype_output")(x)

model = keras.Model(inputs=inputs, outputs=outputs, name="covertype_classifier")

model.summary()

When to use which API?

Linear network (one input → one output)         →  Sequential API ✅
Non-linear network (branches, skip, multi-I/O)  →  Functional API ✅

Course choice: Functional API — it scales to complex situations without rewriting.

3.2 Preparing Data Correctly

Complete pipeline

flowchart LR
    A[fetch_covtype] --> B[Labels 1-7\n→ 0-6]
    B --> C[Stratified\ntrain_test_split]
    C --> D[StandardScaler\ncontinuous features\nonly]
    D --> E[tf.data.Dataset\nbatching + prefetch]

    style A fill:#dbeafe,stroke:#3b82f6
    style E fill:#dcfce7,stroke:#22c55e

Step 1 — Split the datasets:

from sklearn.model_selection import train_test_split

# Stratified split to preserve class distribution
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.15, random_state=7, stratify=y
)

X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.15, random_state=7, stratify=y_temp
)

print(f"Train : {X_train.shape}")   # (~418k, 54)
print(f"Val   : {X_val.shape}")     # (~74k, 54)
print(f"Test  : {X_test.shape}")    # (~87k, 54)

Step 2 — Normalization (continuous features only):

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()

# fit_transform on train, transform on val and test
X_train[:, :10] = scaler.fit_transform(X_train[:, :10])
X_val[:, :10]   = scaler.transform(X_val[:, :10])
X_test[:, :10]  = scaler.transform(X_test[:, :10])
# The 44 binary features (columns 10-53) are left unchanged

Why normalize only the first 10?
The continuous features (elevation ~3000, distances ~1000) have very different scales from binary features (0 or 1). Normalization prevents large values from dominating the gradient.

Step 3 — tf.data pipeline:

import tensorflow as tf

BATCH_SIZE = 1024

train_dataset = (tf.data.Dataset
                 .from_tensor_slices((X_train, y_train))
                 .shuffle(buffer_size=10000)    # random shuffle each epoch
                 .batch(BATCH_SIZE)
                 .prefetch(tf.data.AUTOTUNE))   # preloads during GPU computation

val_dataset = (tf.data.Dataset
               .from_tensor_slices((X_val, y_val))
               .batch(BATCH_SIZE)
               .prefetch(tf.data.AUTOTUNE))

test_dataset = (tf.data.Dataset
                .from_tensor_slices((X_test, y_test))
                .batch(BATCH_SIZE)
                .prefetch(tf.data.AUTOTUNE))

Why prefetch(tf.data.AUTOTUNE)?

Without prefetch : GPU waits → CPU prepares → GPU computes → GPU waits → …
With prefetch    : CPU prepares batch N+1 while GPU computes batch N

3.3 Inside the Training Loop

Every training run repeats the same cycle:

flowchart LR
    A[Data batch] --> B[Forward pass\npredictions]
    B --> C[Loss computation]
    C --> D[Backward pass\ngradients]
    D --> E[Weight update]
    E --> A

    style B fill:#dbeafe,stroke:#3b82f6
    style C fill:#fff4e0,stroke:#e8a820
    style D fill:#fce8e8,stroke:#d45050
    style E fill:#dcfce7,stroke:#22c55e

• Epoch = one full pass over the entire training set
• With 400k samples and batch_size=1024 → ~390 steps per epoch

Explicit training loop (low level)

Useful for understanding what model.fit() does under the hood:

loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

train_acc_metric = tf.keras.metrics.SparseCategoricalAccuracy()
val_acc_metric   = tf.keras.metrics.SparseCategoricalAccuracy()

NUM_EPOCHS = 20

for epoch in range(NUM_EPOCHS):
    print(f"\nEpoch {epoch+1}/{NUM_EPOCHS}")

    # --- Training ---
    for step, (X_batch, y_batch) in enumerate(train_dataset):
        with tf.GradientTape() as tape:
            predictions = model(X_batch, training=True)
            loss = loss_fn(y_batch, predictions)

        gradients = tape.gradient(loss, model.trainable_weights)
        optimizer.apply_gradients(zip(gradients, model.trainable_weights))
        train_acc_metric.update_state(y_batch, predictions)

        if step % 100 == 0:
            print(f"  Step {step}: loss = {loss:.4f}")

    train_acc = train_acc_metric.result()
    train_acc_metric.reset_state()

    # --- Validation ---
    for X_batch, y_batch in val_dataset:
        predictions = model(X_batch, training=False)
        val_acc_metric.update_state(y_batch, predictions)

    val_acc = val_acc_metric.result()
    val_acc_metric.reset_state()

    print(f"  Train accuracy: {train_acc:.4f} | Val accuracy: {val_acc:.4f}")

training=True vs training=False: some layers (Dropout, BatchNorm) behave differently during training and inference.

tf.GradientTape: records all operations within its context to enable automatic gradient computation.

model.fit() — simplified interface

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(),
    metrics=["accuracy"]
)

history = model.fit(
    train_dataset,
    epochs=20,
    validation_data=val_dataset,
    verbose=1
)

Visualizing training curves:

import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

ax1.plot(history.history["loss"],     label="Train loss")
ax1.plot(history.history["val_loss"], label="Val loss")
ax1.set_title("Loss per epoch")
ax1.set_xlabel("Epoch")
ax1.set_ylabel("Loss")
ax1.legend()

ax2.plot(history.history["accuracy"],     label="Train accuracy")
ax2.plot(history.history["val_accuracy"], label="Val accuracy")
ax2.set_title("Accuracy per epoch")
ax2.set_xlabel("Epoch")
ax2.set_ylabel("Accuracy")
ax2.legend()

plt.tight_layout()
plt.show()

What to look for in these curves:

Train loss ↘ and Val loss ↘ in parallel → healthy training
Train loss ↘ but Val loss ↗             → overfitting
Train loss stalls from the start        → learning rate too low
Train loss oscillates strongly          → learning rate too high

3.4 Validation and Stopping Criteria

The global metrics trap

With 49% Lodgepole Pine and 36% Spruce/Fir, a model that only learns to predict these two classes would still achieve ~85% accuracy — an impressive number but a useless model for minority classes.

Class-sensitive metrics:

flowchart LR
    P["Precision\nOf all positive predictions,\nhow many are correct?"] 
    R["Recall\nOf all true positives,\nhow many were detected?"]
    F["F1 Score\nHarmonic mean\nof Precision and Recall"]
    P --- F
    R --- F

$$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN} \qquad F_1 = 2 \cdot \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Keras Callbacks

Callbacks are objects that hook into the training loop at specific moments (end of batch, end of epoch, etc.) and execute an action based on what they observe. They are passed to model.fit().

flowchart TD
    FIT[model.fit] --> CB{Callbacks}
    CB --> ES[EarlyStopping\nAutomatic stop\nif no improvement]
    CB --> RL[ReduceLROnPlateau\nReduces learning rate\nif progress stalls]
    CB --> MC[ModelCheckpoint\nSaves the\nbest model]
    CB --> TB[TensorBoard\nReal-time\nvisualization]
    CB --> WB[WandbMetricsLogger\nExperiment tracking]

callbacks = [
    tf.keras.callbacks.EarlyStopping(
        monitor="val_loss",
        patience=5,           # stop if no improvement for 5 epochs
        min_delta=0.001,
        restore_best_weights=True,
        verbose=1
    ),
    tf.keras.callbacks.ReduceLROnPlateau(
        monitor="val_loss",
        factor=0.5,           # divides LR by 2
        patience=3,
        min_lr=1e-6,
        verbose=1
    )
]

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=0.001),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(),
    metrics=["accuracy"]
)

history = model.fit(
    train_dataset,
    epochs=100,
    validation_data=val_dataset,
    callbacks=callbacks,
    verbose=1
)

Per-class evaluation:

from sklearn.metrics import classification_report
import numpy as np

y_pred = []
y_true = []

for X_batch, y_batch in test_dataset:
    predictions = model(X_batch, training=False)
    y_pred.extend(np.argmax(predictions.numpy(), axis=1))
    y_true.extend(y_batch.numpy())

class_names = ["Spruce/Fir", "Lodgepole Pine", "Ponderosa Pine",
               "Cottonwood/Willow", "Aspen", "Douglas-fir", "Krummholz"]

print(classification_report(y_true, y_pred, target_names=class_names))

3.5 Checkpoints, Logging and Experiment Tracking

Tool overview

Setting up wandb

# Installation
# pip install wandb

import wandb
from wandb.integration.keras import WandbMetricsLogger

wandb.login()   # requires an account at wandb.ai

Centralized hyperparameter configuration:

config = {
    "learning_rate": 0.001,
    "batch_size": 1024,
    "epochs": 100,
    "architecture": [256, 128, 64],
    "optimizer": "adam",
    "min_delta": 0.001,
    "early_stopping_patience": 5,
    "reduce_lr_patience": 3,
    "reduce_lr_factor": 0.5
}

Initializing the run:

run = wandb.init(
    project="covertype-classifier",  # groups all experiments
    config=config,
    name="baseline-run"              # readable label for this specific run
)

Complete callbacks:

callbacks = [
    tf.keras.callbacks.EarlyStopping(
        monitor='val_loss',
        patience=config['early_stopping_patience'],
        min_delta=config['min_delta'],
        restore_best_weights=True,
        verbose=1
    ),
    tf.keras.callbacks.ReduceLROnPlateau(
        monitor='val_loss',
        factor=config['reduce_lr_factor'],
        patience=config['reduce_lr_patience'],
        min_lr=1e-6,
        verbose=1
    ),
    tf.keras.callbacks.ModelCheckpoint(
        filepath=f"checkpoints/{run.name}/best_model.keras",
        monitor="val_loss",
        save_best_only=True,
        verbose=1
    ),
    tf.keras.callbacks.TensorBoard(
        log_dir=f"logs/{run.name}",
        histogram_freq=1
    ),
    WandbMetricsLogger(log_freq="epoch")
]

Training with all callbacks:

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=config['learning_rate']),
    loss=tf.keras.losses.SparseCategoricalCrossentropy(),
    metrics=['accuracy']
)

history = model.fit(
    train_dataset,
    epochs=config['epochs'],
    validation_data=val_dataset,
    callbacks=callbacks,
    verbose=1
)

wandb.finish()

Loading and evaluating the best model:

loaded_model = tf.keras.models.load_model(
    f"checkpoints/{run.name}/best_model.keras"
)

loss, accuracy = loaded_model.evaluate(test_dataset, verbose=0)
print(f"Test loss: {loss:.4f}, Test accuracy: {accuracy:.4f}")

Module 3 — Production-Ready Practices

4.1 Mixed-Precision Training

The precision problem

By default, TensorFlow stores and operates on numbers in float32 (32 bits). Modern GPUs have dedicated Tensor Cores for float16 (16 bits) matrix multiplications — faster and less memory-intensive.

float32  →  extended range, stable, but slower
float16  →  Tensor Cores, 2x–3x faster, but reduced range (underflow/overflow)

flowchart LR
    A[Matrix multiplications\nforward & backward] -- float16 ✅ --> B[Tensor Cores\nGPU\n2x – 3x faster]
    C[Model weights\nstored in memory] -- float32 ✅ --> D[Numerical\nstability]
    E[Gradients\nsmall values] -- Loss Scaling ✅ --> F[Prevents\nunderflow]

    style B fill:#dcfce7,stroke:#22c55e
    style D fill:#dbeafe,stroke:#3b82f6
    style F fill:#fff4e0,stroke:#e8a820

What mixed precision does:

Enabling mixed precision

from tensorflow.keras import mixed_precision

# Must be placed BEFORE building the model
mixed_precision.set_global_policy('mixed_float16')

Redefining the model with mixed precision:

from tensorflow import keras

inputs = keras.Input(shape=(54,), name='covertype_input')

x = inputs
for units in [256, 128, 64]:
    x = keras.layers.Dense(units, activation='relu')(x)

# Final Dense layer without activation
x = keras.layers.Dense(7)(x)

# Softmax explicitly in float32 to avoid numerical instabilities
outputs = keras.layers.Activation(
    'softmax', dtype='float32', name='covertype_output'
)(x)

model = keras.Model(inputs=inputs, outputs=outputs, name='covertype_classifier')

Important: the final softmax layer must be explicitly declared as dtype='float32'. If the model computes in float16 and the output remains float16, the loss could overflow. By separating the activation from the Dense layer and forcing float32, numerical stability is preserved.

4.2 Structuring Code for Reuse

From notebook to modular project

A notebook is excellent for experimentation, but difficult to maintain and share. Organizing around the natural breakpoints of a deep learning project gives:

graph TD
    subgraph "covertype/ project"
        CFG[config.py\nHyperparameters]
        DAT[data.py\nLoading & preprocessing]
        MOD[model.py\nModel definition]
        TRN[train.py\nTraining loop]
        EVL[evaluate.py\nEvaluation]
    end

    CFG --> DAT
    CFG --> MOD
    CFG --> TRN
    TRN --> DAT
    TRN --> MOD
    EVL --> DAT
    EVL --> CFG

    style CFG fill:#fef9c3,stroke:#ca8a04
    style DAT fill:#dbeafe,stroke:#3b82f6
    style MOD fill:#f3e8ff,stroke:#a855f7
    style TRN fill:#dcfce7,stroke:#22c55e
    style EVL fill:#fce8e8,stroke:#d45050

Benefits:

• Changes are isolated: modifying the loss only affects train.py
• When something breaks, you know exactly where to look
• Reuse: data.py can be imported into any other script

config.py — Centralized configuration

config = {
    "learning_rate": 0.001,
    "batch_size": 1024,
    "epochs": 100,
    "architecture": [256, 128, 64],
    "optimizer": "adam",
    "min_delta": 0.001,
    "early_stopping_patience": 5,
    "reduce_lr_patience": 3,
    "reduce_lr_factor": 0.5,
    "checkpoint_path": "checkpoints/best_model.keras",
    "log_dir": "logs/training",
    "random_state": 42,
    "test_size": 0.15,
    "val_size": 0.15,
}

A single place to change a hyperparameter. No more hunting through training code.

data.py — Loading and preprocessing

import numpy as np
from sklearn.datasets import fetch_covtype
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import tensorflow as tf
from config import config

def load_and_preprocess():
    data = fetch_covtype()
    X, y = data.data, data.target - 1

    X_temp, X_test, y_temp, y_test = train_test_split(
        X, y,
        test_size=config['test_size'],
        random_state=config['random_state'],
        stratify=y
    )
    X_train, X_val, y_train, y_val = train_test_split(
        X_temp, y_temp,
        test_size=config['val_size'],
        random_state=config['random_state'],
        stratify=y_temp
    )

    scaler = StandardScaler()
    X_train[:, :10] = scaler.fit_transform(X_train[:, :10])
    X_val[:, :10]   = scaler.transform(X_val[:, :10])
    X_test[:, :10]  = scaler.transform(X_test[:, :10])

    return X_train, X_val, X_test, y_train, y_val, y_test, scaler

def make_datasets(X_train, X_val, X_test, y_train, y_val, y_test):
    batch_size = config['batch_size']

    train_dataset = (tf.data.Dataset
        .from_tensor_slices((X_train, y_train))
        .shuffle(buffer_size=10000)
        .batch(batch_size)
        .prefetch(tf.data.AUTOTUNE))

    val_dataset = (tf.data.Dataset
        .from_tensor_slices((X_val, y_val))
        .batch(batch_size)
        .prefetch(tf.data.AUTOTUNE))

    test_dataset = (tf.data.Dataset
        .from_tensor_slices((X_test, y_test))
        .batch(batch_size)
        .prefetch(tf.data.AUTOTUNE))

    return train_dataset, val_dataset, test_dataset

model.py — Model definition

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import mixed_precision
from config import config

def build_model():
    mixed_precision.set_global_policy('mixed_float16')

    inputs = keras.Input(shape=(54,), name='covertype_input')

    x = inputs
    for units in config['architecture']:
        x = keras.layers.Dense(units, activation='relu')(x)

    x = keras.layers.Dense(7)(x)
    outputs = keras.layers.Activation(
        'softmax', dtype='float32', name='covertype_output'
    )(x)

    return keras.Model(inputs=inputs, outputs=outputs, name='covertype_classifier')

train.py — Training logic

import os
import random
import numpy as np
import tensorflow as tf
import wandb
from wandb.integration.keras import WandbMetricsLogger

from config import config
from data import load_and_preprocess, make_datasets
from model import build_model

def set_seeds(seed=7):
    """Fixes all random sources for reproducibility."""
    os.environ['PYTHONHASHSEED'] = str(seed)  # Python hash randomization
    random.seed(seed)                          # built-in random module
    np.random.seed(seed)                       # NumPy
    tf.random.set_seed(seed)                   # TensorFlow (weights, dropout, etc.)

def get_callbacks():
    return [
        tf.keras.callbacks.EarlyStopping(
            monitor='val_loss',
            patience=config['early_stopping_patience'],
            min_delta=config['min_delta'],
            restore_best_weights=True,
            verbose=1
        ),
        tf.keras.callbacks.ReduceLROnPlateau(
            monitor='val_loss',
            factor=config['reduce_lr_factor'],
            patience=config['reduce_lr_patience'],
            min_lr=1e-6,
            verbose=1
        ),
        tf.keras.callbacks.ModelCheckpoint(
            filepath=config['checkpoint_path'],
            monitor='val_loss',
            save_best_only=True,
            verbose=1
        ),
        tf.keras.callbacks.TensorBoard(
            log_dir=config['log_dir'],
            histogram_freq=1
        ),
        WandbMetricsLogger(log_freq='epoch'),
    ]

def train():
    set_seeds(config['random_state'])   # ← MUST be the very first call

    wandb.init(
        project="covertype-classifier",
        config=config,
        name="baseline-run"
    )

    X_train, X_val, X_test, y_train, y_val, y_test, _ = load_and_preprocess()
    train_dataset, val_dataset, _ = make_datasets(
        X_train, X_val, X_test, y_train, y_val, y_test
    )

    model = build_model()
    model.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=config['learning_rate']),
        loss=tf.keras.losses.SparseCategoricalCrossentropy(),
        metrics=['accuracy']
    )

    model.fit(
        train_dataset,
        epochs=config['epochs'],
        validation_data=val_dataset,
        callbacks=get_callbacks(),
        verbose=1
    )

    wandb.finish()

if __name__ == "__main__":
    train()

evaluate.py — Evaluation

import numpy as np
import tensorflow as tf
from sklearn.metrics import classification_report
from config import config
from data import load_and_preprocess, make_datasets

CLASS_NAMES = ['Spruce/Fir', 'Lodgepole Pine', 'Ponderosa Pine',
               'Cottonwood/Willow', 'Aspen', 'Douglas-fir', 'Krummholz']

def evaluate():
    model = tf.keras.models.load_model(config['checkpoint_path'])

    X_train, X_val, X_test, y_train, y_val, y_test, _ = load_and_preprocess()
    _, _, test_dataset = make_datasets(
        X_train, X_val, X_test, y_train, y_val, y_test
    )

    y_pred, y_true = [], []
    for X_batch, y_batch in test_dataset:
        predictions = model(X_batch, training=False)
        y_pred.extend(np.argmax(predictions.numpy(), axis=1))
        y_true.extend(y_batch.numpy())

    print(classification_report(y_true, y_pred, target_names=CLASS_NAMES))

if __name__ == "__main__":
    evaluate()

4.3 Saving and Loading Models

Two things to preserve

Complete model = Architecture (JSON) + Weights (numerical arrays) + Optimizer state

flowchart LR
    subgraph "What is saved"
        A[Architecture\n→ JSON config]
        B[Weights\n→ numerical arrays]
        C[Optimizer state\n→ resume training]
    end

    subgraph ".keras format (recommended)"
        D[ZIP archive\nself-contained]
    end

    subgraph ".h5 format (legacy)"
        E[HDF5\nModern Keras\nlimitations]
    end

    A --> D
    B --> D
    C --> D
    A --> E
    B --> E

Format comparison

Saving the complete model

# Keras format (recommended)
model.save("covertype_model.keras")

# Load in a fresh context
loaded_model = tf.keras.models.load_model("covertype_model.keras")

Saving weights only

# Weights only
model.save_weights("covertype_weights.keras")

# Loading: rebuild the architecture first, then load weights
from model import build_model

new_model = build_model()
new_model.load_weights("covertype_weights.keras")

4.4 Versioning and Reproducibility

Sources of randomness in a deep learning pipeline

flowchart TD
    A[Randomness\nin the pipeline] --> B[Weight initialization\nDifferent starting points\n→ different trajectories]
    A --> C[Data shuffling\nBatch order\n→ different updates]
    A --> D[Train/val/test split\nDifferent data\n→ invalid comparisons]

    B --> E{Solution:\nfix the seeds}
    C --> E
    D --> E

    style E fill:#dcfce7,stroke:#22c55e

These random sources span three separate libraries:

Python built-in  →  random.seed()
NumPy            →  np.random.seed()
TensorFlow       →  tf.random.set_seed()
Python hashing   →  os.environ['PYTHONHASHSEED']

Setting seeds correctly

import os
import random
import numpy as np
import tensorflow as tf

def set_seeds(seed=7):
    """
    Must be called BEFORE any import that might trigger
    random operations.
    """
    os.environ['PYTHONHASHSEED'] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    tf.random.set_seed(seed)

# In train.py — FIRST line in train()
def train():
    set_seeds(42)  # ← before data loading, before model building
    # ...

Why order matters: randomization in Python is procedural. At the point where you import NumPy or TensorFlow, they initialize their own internal random state. Seeds must be set as early as possible.

What reproducibility guarantees

Fixed seeds + Logged configuration  →  any experiment can be
                                        recreated exactly

With wandb, every run automatically logs:

• All hyperparameters (config)
• Metrics epoch by epoch
• The run name to find the corresponding checkpoint

5. Quick Reference — Project Files

Project structure

covertype/
├── config.py       ← hyperparameters (single source of truth)
├── data.py         ← loading, split, normalization, tf.data
├── model.py        ← architecture + mixed precision
├── train.py        ← set_seeds, callbacks, wandb, model.fit
├── evaluate.py     ← per-class classification_report
└── checkpoints/
    └── best_model.keras

Summary of technical choices

Complete data flow

flowchart TD
    A["fetch_covtype()\n581,012 samples\n54 features, 7 classes"] 
    --> B["Labels - 1\n(1-7 → 0-6)"]
    --> C["Stratified\ntrain_test_split\n70% / 15% / 15%"]
    --> D["StandardScaler\non features [:10]"]
    --> E["tf.data.Dataset\nshuffle → batch(1024) → prefetch"]
    --> F["build_model()\nmixed_float16\n54 → 256 → 128 → 64 → 7"]
    --> G["model.fit()\nEarlyStopping + ReduceLR\n+ Checkpoint + TensorBoard + wandb"]
    --> H["evaluate()\nper-class\nclassification_report"]

    style A fill:#dbeafe,stroke:#3b82f6
    style F fill:#f3e8ff,stroke:#a855f7
    style G fill:#dcfce7,stroke:#22c55e
    style H fill:#fce8e8,stroke:#d45050

Search Terms

deep · frameworks · model · neural · networks · machine · data · science · api · saving · comparison · correctly · cpu · dataset · functional · gpu · hardware · loading · loop · manual · pipeline · precision · reproducibility · sequential