Amazon SageMaker

Table of Contents

1. Introduction to Amazon SageMaker
   • 1.1 The Problem: Limitations of the Traditional Approach
   • 1.2 Machine Learning — Overview
   • 1.3 Jupyter Notebooks — The Essential Tool
   • 1.4 SageMaker Notebook Instances
   • 1.5 SageMaker Studio
2. Training Models
   • 2.1 Built-in Algorithms
   • 2.2 Training with XGBoost (Built-in Algorithm)
   • 2.3 Hyperparameter Optimization (Hyperparameter Tuning)
   • 2.4 Custom Algorithms with Docker and ECR
3. Model Evaluation
   • 3.1 Why Evaluate Continuously?
   • 3.2 Metrics for Classification Models
   • 3.3 Metrics for Regression Models
   • 3.4 Evaluation with SageMaker Canvas
   • 3.5 Evaluation with the SageMaker SDK (Python)
   • 3.6 Tips for Continuous Evaluation
4. Deploying Models
   • 4.1 SageMaker Deployment Pipeline
   • 4.2 Inference Types
   • 4.3 Deployment Options
   • 4.4 Demo: Real-Time Deployment with SageMaker Canvas
   • 4.5 Updating a Deployment and Autoscaling
   • 4.6 Cost Optimization

1. Introduction to Amazon SageMaker

1.1 The Problem: Limitations of the Traditional Approach

The classic approach to solving problems with a computer always follows the same pattern: a human thinks about the problem, a programmer translates the solution into code, and software produces a result. This method runs into limitations in two situations:

┌─────────────────────────────────────────────────────────────────┐
│            TRADITIONAL APPROACH                                 │
│                                                                 │
│  Human thinks    →  Programmer writes  →  Software produces    │
│  about problem      the code               a result            │
│                                                                 │
│  LIMITATIONS:                                                   │
│  • Large amounts of data are difficult to analyze               │
│  • Complex, counter-intuitive patterns between data points      │
└─────────────────────────────────────────────────────────────────┘

Concrete use case — Carved Rock Fitness:
A regional gym chain wanted to analyze employee absenteeism. Their HR department had a large dataset, but the patterns in the data were too complex for manual analysis, and on-premises infrastructure was insufficient for intensive computation.

1.2 Machine Learning — Overview

Machine Learning (ML) consists of delegating to a computer the statistical analysis of large amounts of data to detect patterns that are difficult for humans to identify.

flowchart LR
    A[Labeled\nDataset] --> B[ML\nAlgorithm]
    B --> C[Mathematical\nModel]
    C --> D[Prediction\nApplication]
    D --> E[Inferences\non new data]

    style A fill:#2196F3,color:#fff
    style B fill:#FF9800,color:#fff
    style C fill:#4CAF50,color:#fff
    style D fill:#9C27B0,color:#fff
    style E fill:#F44336,color:#fff

Essential terminology:

Machine Learning challenges:

┌──────────────────────────────────────────┐
│  CHALLENGE 1: Compute Power              │
│  • Specialized GPU requirements          │
│  • Machine clusters needed               │
│  ← Solution: AWS SageMaker (cloud)       │
├──────────────────────────────────────────┤
│  CHALLENGE 2: Data Quality               │
│  • Missing data, errors                  │
│  • Intensive data preparation            │
│  ← Solution: Data Scientists + tools    │
└──────────────────────────────────────────┘

1.3 Jupyter Notebooks — The Essential Tool

A Jupyter Notebook is an interactive document that combines human-language documentation and executable code in the same web interface. It was developed by the Project Jupyter open-source project.

flowchart TD
    JN[Jupyter Notebook]
    JN --> CC[Code Cells\nPython / R / Scala\nExecutable code]
    JN --> MC[Markdown Cells\nHuman documentation\nTitles, text, lists]

    CC --> EX[Run\ntraining jobs]
    MC --> DOC[Explain the\nreasoning]

    style JN fill:#F5A623,color:#000
    style CC fill:#4A90E2,color:#fff
    style MC fill:#7ED321,color:#fff

Key advantages:

• Intuitive web interface
• Mixes documentation and code in a single file (.ipynb)
• Supports Python, R, Scala
• Two cell types: Code cells (instructions) and Markdown cells (documentation)

Example Markdown cell:

# Carved Rock Fitness
## Absenteeism Analysis

*Data* provided by the HR department.  
**Objective**: Identify absenteeism patterns.

1.4 SageMaker Notebook Instances

A SageMaker Notebook Instance is a pre-configured virtual machine in the AWS cloud, including the Jupyter Lab environment.

flowchart LR
    subgraph AWS Cloud
        NI[SageMaker\nNotebook Instance\nPre-configured VM]
        S3[(S3 Bucket\nData)]
        SM[SageMaker\nServices]
        NI <--> S3
        NI <--> SM
    end
    U[User\nBrowser] --> NI

    style NI fill:#FF9900,color:#fff
    style S3 fill:#569A31,color:#fff
    style SM fill:#232F3E,color:#fff

Characteristics:

• Virtual machine in AWS, accessible via browser
• Pre-configured with Jupyter Lab and common ML libraries
• Direct access to data in S3
• Billed only when the instance is running (pay-as-you-go model)
• Can be stopped to save costs

Creation steps in the AWS console:

1. Navigate to the SageMaker service
2. Select Notebook Instances → Create notebook instance
3. Choose a name and an EC2 instance type (e.g.: ml.t3.medium for free tier)
4. Assign an IAM role with S3 and SageMaker permissions
5. (Optional) Associate a Git repository
6. Click Create → Wait for InService status

1.5 SageMaker Studio

SageMaker Studio is the first fully browser-based Machine Learning IDE (Integrated Development Environment), designed to replace classic Notebook Instances.

flowchart TD
    SS[SageMaker Studio\nComplete ML IDE]
    SS --> JL[JupyterLab\nJupyter Notebooks]
    SS --> CE[Code Editor\nbased on VS Code]
    SS --> RS[RStudio\nfor R]
    SS --> JS[JumpStart\nGenAI Models]
    SS --> DW[Data Wrangler\nData Preparation]
    SS --> MM[Model Monitor\nDeployment Tracking]

    subgraph Separate Infrastructure
        CALC[Compute\nEC2 Instances]
        STOCK[Storage\nS3]
    end

    SS -.->|separate| CALC
    SS -.->|separate| STOCK

    style SS fill:#232F3E,color:#fff
    style JL fill:#F5A623,color:#000
    style CE fill:#0078D4,color:#fff
    style RS fill:#276DC3,color:#fff
    style JS fill:#4CAF50,color:#fff

Comparison: Notebook Instance vs. Studio

SageMaker JumpStart provides access to pre-trained models from providers including:

• AI21 Labs
• Stability AI
• PyTorch Hub
• And many others

SageMaker Spaces: In Studio, Jupyter environments run in Spaces that isolate storage and compute.

2. Training Models

flowchart LR
    A[Raw Data\nS3] --> B[Data\nPreparation]
    B --> C{Algorithm\ntype}
    C -->|Built-in| D[Built-in\nAlgorithm\nContainer]
    C -->|Custom| E[Docker Image\nECR Repository]
    D --> F[SageMaker\nTraining Job]
    E --> F
    F --> G[Model\nArtifact S3]
    G --> H[Evaluation]

    style A fill:#569A31,color:#fff
    style F fill:#FF9900,color:#fff
    style G fill:#4CAF50,color:#fff

2.1 Built-in Algorithms

SageMaker offers 17 optimized, scalable, ready-to-use built-in algorithms covering several domains:

┌─────────────────────────────────────────────────────────────────┐
│           SAGEMAKER BUILT-IN ALGORITHMS                         │
├──────────────────┬──────────────────────────────────────────────┤
│ Tabular data     │ XGBoost, Linear Learner, K-Nearest Neighbors │
├──────────────────┼──────────────────────────────────────────────┤
│ Text (NLP)       │ BlazingText, Sequence-to-Sequence            │
├──────────────────┼──────────────────────────────────────────────┤
│ Time series      │ DeepAR Forecasting                           │
├──────────────────┼──────────────────────────────────────────────┤
│ Unsupervised     │ K-Means, PCA, IP Insights                    │
├──────────────────┼──────────────────────────────────────────────┤
│ Computer vision  │ Image Classification, Object Detection        │
└──────────────────┴──────────────────────────────────────────────┘

Each algorithm has a downloadable whitepaper in the SageMaker Developer Guide.

2.2 Training with XGBoost (Built-in Algorithm)

The following example uses the XGBoost built-in algorithm to train a classification model on the MNIST dataset (handwritten digit recognition).

Initial Setup and Data Preparation

# Installation and imports
!pip install --upgrade sagemaker

import os
import boto3
import copy
import time
from time import gmtime, strftime, sleep
import sagemaker
from sagemaker import get_execution_role
from sagemaker.inputs import TrainingInput
from sagemaker.image_uris import retrieve
from pprint import pprint
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# SageMaker configuration
role = get_execution_role()
region = boto3.Session().region_name
sess = sagemaker.Session()

# S3 configuration
bucket_name = "training-models-on-amazon-sagemaker"
prefix = 'mnist/xgboost'
bucket_path = f"s3://{bucket_name}/{prefix}"

# SageMaker client initialization
sm = boto3.client('sagemaker')

# Function to save datasets in libsvm format
def save_as_libsvm(X, y, file_name):
    with open(file_name, 'w') as f:
        for i in range(len(X)):
            line = str(int(y[i])) + ' '
            line += ' '.join([f'{j + 1}:{X[i][j]}' for j in range(len(X[i]))])
            f.write(line + '\n')

# Upload to S3 with bucket verification
def upload_to_s3(local_file, s3_path):
    s3_client = boto3.client('s3')
    try:
        s3_client.head_bucket(Bucket=bucket_name)
    except:
        s3_client.create_bucket(
            Bucket=bucket_name,
            CreateBucketConfiguration={'LocationConstraint': region}
        )
    s3_client.upload_file(local_file, bucket_name, s3_path)
    print(f"Uploaded {local_file} to s3://{bucket_name}/{s3_path}")

# Prepare MNIST data
digits = datasets.load_digits()
X = StandardScaler().fit_transform(digits.data)
y = digits.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Save in libsvm format (expected by built-in XGBoost)
save_as_libsvm(X_train, y_train, 'train.libsvm')
save_as_libsvm(X_test, y_test, 'validation.libsvm')

# Upload to S3
upload_to_s3('train.libsvm', f'{prefix}/data/train.libsvm')
upload_to_s3('validation.libsvm', f'{prefix}/data/validation.libsvm')

Launching the Training Job

# Select the built-in XGBoost container image
container = retrieve(framework="xgboost", region=region, version="1.7-1")

# Define the Estimator
xgb_estimator = sagemaker.estimator.Estimator(
    image_uri=container,
    role=role,
    instance_count=1,
    instance_type="ml.m5.4xlarge",
    output_path=f's3://{bucket_name}/{prefix}/training-jobs',
    hyperparameters={
        "max_depth": "5",
        "eta": "0.2",
        "gamma": "4",
        "min_child_weight": "6",
        "verbosity": "0",
        "objective": "multi:softmax",      # Multi-class classification
        "num_class": "10",                 # 10 digits (0-9)
        "num_round": "10"
    }
)

# Launch training
xgb_estimator.fit({
    'train': TrainingInput(
        f'{bucket_path}/data/train.libsvm',
        content_type="libsvm"
    ),
    'validation': TrainingInput(
        f'{bucket_path}/data/validation.libsvm',
        content_type="libsvm"
    )
})

sequenceDiagram
    participant NB as Jupyter Notebook
    participant SM as SageMaker
    participant EC2 as EC2 Instance
    participant S3 as S3 Bucket

    NB->>S3: Upload data (libsvm)
    NB->>SM: Create Training Job (Estimator.fit)
    SM->>EC2: Provision ml.m5.4xlarge instance
    EC2->>S3: Download training data
    EC2->>EC2: Run XGBoost algorithm
    EC2->>S3: Save model artifact (.tar.gz)
    SM->>NB: Progress logs
    NB->>S3: Verify saved model

2.3 Hyperparameter Optimization (Hyperparameter Tuning)

Hyperparameters are values set before training (batch size, learning rate, tree depth, etc.). SageMaker offers several tuning strategies:

flowchart TD
    HT[Hyperparameter\nTuning Job]
    HT --> GS[Grid Search\nBrute force over all\ncombinations]
    HT --> RS[Random Search\nRandom selection\nwithin defined ranges]
    HT --> BO[Bayesian Optimization\nProbabilistic model based\non previous evaluations]
    HT --> HB[Hyperband\nDynamically allocates\nresources]
    HT --> ES[Early Stopping\nStops when performance\nno longer improves]

    style GS fill:#E53935,color:#fff
    style RS fill:#FB8C00,color:#fff
    style BO fill:#43A047,color:#fff
    style HB fill:#1E88E5,color:#fff
    style ES fill:#8E24AA,color:#fff

Code — Bayesian Tuning Job for XGBoost

def tune_hyperparameters():
    tuning_job_name = f'xgboost-{strftime("%Y-%m-%d-%H-%M-%S", gmtime())}'

    # Tuning job configuration (Bayesian strategy)
    tuning_job_config = {
        "ParameterRanges": {
            "CategoricalParameterRanges": [],
            "ContinuousParameterRanges": [
                {"MaxValue": "0.5", "MinValue": "0.1", "Name": "eta"},
                {"MaxValue": "5",   "MinValue": "0",   "Name": "gamma"},
                {"MaxValue": "120", "MinValue": "0",   "Name": "min_child_weight"},
                {"MaxValue": "1",   "MinValue": "0.5", "Name": "subsample"},
                {"MaxValue": "2",   "MinValue": "0",   "Name": "alpha"},
            ],
            "IntegerParameterRanges": [
                {"MaxValue": "10", "MinValue": "0", "Name": "max_depth"},
                {"MaxValue": "50", "MinValue": "1", "Name": "num_round"},
            ],
        },
        "ResourceLimits": {
            "MaxNumberOfTrainingJobs": 6,
            "MaxParallelTrainingJobs": 2,
        },
        "Strategy": "Bayesian",
        "HyperParameterTuningJobObjective": {
            "MetricName": "validation:merror",
            "Type": "Minimize"            # Minimize error rate
        },
    }

    training_job_definition = copy.deepcopy(training_parameters)
    del training_job_definition["HyperParameters"]
    training_job_definition["OutputDataConfig"]["S3OutputPath"] = \
        f"{bucket_path}/tuning-jobs"
    training_job_definition["StaticHyperParameters"] = {
        "objective": "multi:softmax",
        "verbosity": "2",
        "num_class": "10",
    }

    # Launch tuning job
    sm.create_hyper_parameter_tuning_job(
        HyperParameterTuningJobName=tuning_job_name,
        HyperParameterTuningJobConfig=tuning_job_config,
        TrainingJobDefinition=training_job_definition,
    )

    # Track progress
    status = sm.describe_hyper_parameter_tuning_job(
        HyperParameterTuningJobName=tuning_job_name
    )["HyperParameterTuningJobStatus"]

    while status not in ("Completed", "Failed"):
        sleep(60)
        status = sm.describe_hyper_parameter_tuning_job(
            HyperParameterTuningJobName=tuning_job_name
        )["HyperParameterTuningJobStatus"]
        print(status)

    # Analyze results
    tuning_result = sm.describe_hyper_parameter_tuning_job(
        HyperParameterTuningJobName=tuning_job_name
    )
    best_job = tuning_result['BestTrainingJob']
    print(f"Best job: {best_job['TrainingJobName']}")
    print(f"Best value: "
          f"{best_job['FinalHyperParameterTuningJobObjectiveMetric']['Value']}")
    print("\nBest hyperparameters:")
    pprint(best_job['TunedHyperParameters'])

tune_hyperparameters()

2.4 Custom Algorithms with Docker and ECR

When built-in algorithms are not sufficient, SageMaker allows using any algorithm or framework via a Docker container hosted in Amazon ECR (Elastic Container Registry).

flowchart TD
    subgraph Local Development / Notebook Instance
        A[Write the training\nscript train.py]
        B[Create the\nDockerfile]
        C[Build the\nDocker image]
    end
    subgraph AWS
        D[(ECR Repository\nmy-custom-algorithm)]
        E[SageMaker\nTraining Job]
        F[(S3 Bucket\nModel Artifact)]
    end

    A --> B --> C
    C -->|docker push| D
    D --> E
    E --> F

    style D fill:#FF6B35,color:#fff
    style E fill:#FF9900,color:#fff
    style F fill:#569A31,color:#fff

Step 1 — Write the PyTorch Training Script

%%writefile train.py
import torch
import torch.optim as optim
import torch.nn as nn
from torchvision import datasets, transforms
from torch.utils.data import DataLoader

# Simple neural network definition
class SimpleNN(nn.Module):
    def __init__(self):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(28 * 28, 128)   # Input layer: 784 → 128
        self.fc2 = nn.Linear(128, 10)         # Output layer: 128 → 10 classes

    def forward(self, x):
        x = x.view(-1, 28 * 28)              # Flatten 28x28 image to 784 vector
        x = torch.relu(self.fc1(x))          # ReLU activation on 1st layer
        x = self.fc2(x)                       # Output (logits for 10 classes)
        return x

def train():
    transform = transforms.Compose([transforms.ToTensor()])
    train_dataset = datasets.MNIST(
        root='/opt/ml/input/data/train',
        train=True,
        transform=transform,
        download=True
    )
    train_loader = DataLoader(
        dataset=train_dataset, batch_size=64, shuffle=True
    )

    model = SimpleNN()
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    criterion = nn.CrossEntropyLoss()

    for epoch in range(10):
        for batch_idx, (data, target) in enumerate(train_loader):
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
        print(f'Epoch {epoch+1} complete')

    # Save model to SageMaker standard path
    torch.save(model.state_dict(), '/opt/ml/model/model.pth')

if __name__ == '__main__':
    train()

Step 2 — Create the Dockerfile

FROM python:3.8

# Install dependencies
RUN pip install torch torchvision

# Working directory
WORKDIR /app

# Copy training script into the container
COPY train.py .

# Entry command
ENTRYPOINT ["python", "train.py"]

Step 3 — Build and Push to ECR

# ECR authentication
!$(aws ecr get-login --no-include-email --region us-east-2)

# Create ECR repository
!aws ecr create-repository --repository-name my-custom-algorithm

sts = boto3.client("sts")
account_id = sts.get_caller_identity()["Account"]
region = boto3.Session().region_name

repository_name = "my-custom-algorithm"
ecr_uri = f"{account_id}.dkr.ecr.{region}.amazonaws.com/{repository_name}"

!docker build -t my-custom-algorithm .
!docker tag my-custom-algorithm:latest {ecr_uri}:latest
!docker push {ecr_uri}:latest

Step 4 — Launch the Training Job with the Custom Algorithm

from sagemaker.estimator import Estimator

bucket_name = "training-models-on-amazon-sagemaker"
prefix = 'mnist/simpleNN'
bucket_path = f"s3://{bucket_name}/{prefix}"

account_id = boto3.client("sts").get_caller_identity()["Account"]
region = boto3.Session().region_name
repository_name = "my-custom-algorithm"
role = sagemaker.get_execution_role()

# Estimator pointing to our custom ECR image
custom_estimator = Estimator(
    image_uri=f"{account_id}.dkr.ecr.{region}.amazonaws.com/{repository_name}:latest",
    role=role,
    instance_count=1,
    instance_type='ml.m5.large',
    output_path=bucket_path,
)

# Launch and track
training_job_name = f'simpleNN-{strftime("%Y-%m-%d-%H-%M-%S", gmtime())}'
custom_estimator.fit(job_name=training_job_name)

# Retrieve the S3 path of the saved model
sm_client = boto3.client('sagemaker')
training_info = sm_client.describe_training_job(
    TrainingJobName=training_job_name
)
model_s3_uri = training_info['ModelArtifacts']['S3ModelArtifacts']
print(f"Model saved at: {model_s3_uri}")

3. Model Evaluation

3.1 Why Evaluate Continuously?

ML model evaluation is not a one-time operation — it is a continuous process integrated into the complete lifecycle:

flowchart LR
    A[Data\nCollection] --> B[Model\nTraining]
    B --> C[Model\nEvaluation]
    C -->|Performing well| D[Deployment]
    C -->|Insufficient| B
    D --> E[Inference\nMonitoring]
    E --> F[Drift\nDetection]
    F --> A

    style C fill:#FF9800,color:#fff
    style D fill:#4CAF50,color:#fff
    style F fill:#F44336,color:#fff

Why evaluation is critical:

• Models are used to make decisions — they must be reliable
• An underperforming model generates losses or erroneous decisions
• Real-world data changes over time (model drift)
• Continuous evaluation ensures alignment with business objectives

3.2 Metrics for Classification Models

The Confusion Matrix

For a binary classification model (e.g.: diabetic / not diabetic):

                    ┌─────────────────────────────────────┐
                    │      ACTUAL VALUES                   │
                    │   Positive (1)   Negative (0)        │
         ┌──────────┼──────────────────────────────────────┤
PREDICTED│Positive(1)│  True Positive   False Positive     │
         │          │  (TP) ✅         (FP) ❌             │
         ├──────────┼──────────────────────────────────────┤
         │Negative(0)│  False Negative  True Negative      │
         │          │  (FN) ❌         (TN) ✅             │
         └──────────┴──────────────────────────────────────┘

Derived Metrics from the Confusion Matrix

Accuracy

$$\text{Accuracy} = \frac{TP + TN}{TP + FP + FN + TN}$$

Percentage of correct predictions. Watch out: 97.8% accuracy can mask a useless model if the positive class is rare (2.2% of the dataset).

Precision

$$\text{Precision} = \frac{TP}{TP + FP}$$

Among all positive predictions, how many are actually positive? Reduces false alarms.

Recall (Sensitivity)

$$\text{Recall} = \frac{TP}{TP + FN}$$

Among all actually positive cases, how many did the model identify? Critical for fire detection, medical diagnostics.

F1 Score

$$\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}}$$

Harmonic mean between precision and recall. For multi-class problems, calculate F1 per class then average.

AUC — Area Under the Curve (ROC)

ROC Curve:
     1.0 │ ╭───────────
         │╭╯
True     │╱
Positive │╱
Rate     │╱
     0.0 └──────────────
         0.0           1.0
         False Positive Rate

AUC = 1.0 → Perfect model
AUC = 0.5 → No better than random

AUC measures the model’s ability to distinguish classes at all thresholds. Particularly useful for spam detection.

3.3 Metrics for Regression Models

For models predicting a numerical value (e.g.: house price):

Mean Squared Error (MSE)

$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$

Heavily penalizes large errors (squared). Always positive. Lower = better.

Apple example (weight in grams): actual = [150, 160, 170], predicted = [148, 162, 172] → MSE = (2²+2²+2²)/3 = 4

Root Mean Squared Error (RMSE)

$$\text{RMSE} = \sqrt{\text{MSE}}$$

MSE in the same unit as the target variable. More interpretable.

R² (Coefficient of Determination)

$$R^2 = 1 - \frac{\sum(y_i - \hat{y}_i)^2}{\sum(y_i - \bar{y})^2}$$

Proportion of variance explained by the model. R²=1 → perfect. R²=0 → model just predicts the mean.

Example: R²=0.95 for house surface area → 95% of price variability is explained by surface area.

Mean Absolute Error (MAE)

$$\text{MAE} = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y}_i|$$

Same unit as the target variable. More intuitive. Less sensitive to outliers than MSE.

Apple example: |2|+|2|+|2| / 3 = 2 grams

Mean Absolute Percentage Error (MAPE)

$$\text{MAPE} = \frac{100%}{n}\sum_{i=1}^{n}\left|\frac{y_i - \hat{y}_i}{y_i}\right|$$

Expresses error as a percentage. Easily interpretable.

Apple example: ((2/150)+(2/160)+(2/170))/3 × 100 ≈ 1.25%

Visual summary of regression metrics:

┌──────────────────┬───────────────┬─────────────────────────────────┐
│ Metric           │ Optimal       │ Main usage                       │
├──────────────────┼───────────────┼─────────────────────────────────┤
│ MSE              │ → 0           │ Penalizes large errors           │
│ RMSE             │ → 0           │ Same unit as target              │
│ R²               │ → 1           │ Overall performance view         │
│ MAE              │ → 0           │ Mean absolute error              │
│ MAPE             │ → 0%          │ Percentage error                 │
└──────────────────┴───────────────┴─────────────────────────────────┘

3.4 Evaluation with SageMaker Canvas

SageMaker Canvas enables no-code model evaluation through a graphical interface.

Example — Classification model for industrial equipment:
Maintenance dataset predicting 3 classes: No Failure, Power Failure, Overstrain Failure.

"Analyze" tab → "Advanced Metrics" in SageMaker Canvas:

┌─────────────────────────────────────────────────────────────┐
│  MAINTENANCE MODEL METRICS                                  │
├─────────────────────────────────────────────────────────────┤
│  Accuracy             : 98.5%                               │
│  Average F1 Score     : calculated per class                │
│  Precision (per class): variable                            │
│  Recall (per class)   : ⚠️ relatively low                  │
│  AUC                  : N/A (multi-class)                   │
└─────────────────────────────────────────────────────────────┘

Steps in SageMaker Canvas:

1. Load the trained model
2. Go to the Analyze tab
3. Click Advanced metrics
4. Analyze Accuracy, Precision, Recall, F1, AUC

3.5 Evaluation with the SageMaker SDK (Python)

For regression models, use a Batch Transform Job rather than a real-time endpoint.

from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
import numpy as np

# Compute regression metrics
# (after downloading predictions from S3)

# Extract actual labels
true_labels = test_data[target].values

# Mean Squared Error
mse = mean_squared_error(true_labels, predictions)
print(f"MSE  : {mse:.4f}")

# Root Mean Squared Error
rmse = np.sqrt(mse)
print(f"RMSE : {rmse:.4f}")

# R² (Coefficient of determination)
r2 = r2_score(true_labels, predictions)
print(f"R²   : {r2:.4f}")       # Close to 1 → good model

# Mean Absolute Error
mae = mean_absolute_error(true_labels, predictions)
print(f"MAE  : {mae:.4f}")

Example interpretation (house price prediction):

3.6 Tips for Continuous Evaluation

Model Drift: phenomenon where data distribution changes over time, degrading model performance.

flowchart TD
    A[Deployed Model] --> B[Continuous monitoring\nSageMaker Model Monitor]
    B --> C{Drift\ndetected?}
    C -->|No| B
    C -->|Yes| D[Analyze metrics\nSageMaker Clarify]
    D --> E[Retrain with\nnew data]
    E --> A

    style C fill:#FF9800,color:#fff
    style E fill:#4CAF50,color:#fff

Practical recommendations:

1. Automate evaluations — Configure periodic pipelines
2. Monitor metrics continuously — Use SageMaker Model Monitor and SageMaker Clarify
3. Retrain regularly — Update training data with newly collected data
4. Detect drift early — Compare evaluation metrics over time

Drift example: A model trained on maintenance data for a specific type of equipment may underperform when new equipment is introduced in the factory.

4. Deploying Models

4.1 SageMaker Deployment Pipeline

flowchart LR
    subgraph Preparation
        TD[Training Data\nRaw data]
        CE[Clean & Enrich\nSageMaker Data Wrangler]
        PT[Processed Training Data]
        TD --> CE --> PT
    end

    subgraph Training
        TR[Train\nSageMaker Training Job]
        TM[Trained Model\nS3 artifact]
        PT --> TR --> TM
    end

    subgraph Deployment
        DP[Deploy\nEC2 Instances]
        DM[Deployed Model]
        EP[Endpoint\nAccess URL]
        TM --> DP --> DM --> EP
    end

    subgraph Usage
        U[Users\nApplications]
        EP --> U
    end

    style TD fill:#2196F3,color:#fff
    style TR fill:#FF9900,color:#fff
    style TM fill:#4CAF50,color:#fff
    style EP fill:#9C27B0,color:#fff

An endpoint is an AWS resource that:

• Exposes the deployed model via a URL
• Handles all incoming requests
• Routes requests to the right model
• Returns predictions

4.2 Inference Types

flowchart TD
    INF[SageMaker\nInference Types]
    INF --> RT[Real-time Inference\nImmediate response\nModel always active\nEx.: fraud detection]
    INF --> SL[Serverless Inference\nActive only on demand\nUnpredictable traffic\nPay-per-use]
    INF --> AS[Async Inference\nBackground processing\nResult sent later\nEx.: long tasks]

    style RT fill:#F44336,color:#fff
    style SL fill:#2196F3,color:#fff
    style AS fill:#FF9800,color:#fff

4.3 Deployment Options

flowchart TD
    OPT[Deployment Options]
    OPT --> NC[Low-code / No-code\nSageMaker JumpStart\nor SageMaker Canvas\nQuick deployment without code]
    OPT --> SDK[SageMaker Python SDK\nFull control\nMaximum flexibility]
    OPT --> CFN[AWS CloudFormation\nInfrastructure as Code\nEnterprise, scalable, reproducible]

    style NC fill:#4CAF50,color:#fff
    style SDK fill:#2196F3,color:#fff
    style CFN fill:#FF9800,color:#fff

4.4 Demo: Real-Time Deployment with SageMaker Canvas

Scenario: Auto claims analysis model (ClaimsAnalyzer) — predicts whether a claim will be approved or rejected.

Steps in SageMaker Canvas:

1. Go to SageMaker → Canvas (from the SageMaker domain)
2. Load the "auto claims" dataset
3. Create a model: name = "ClaimsAnalyzer"
4. Select target column = "Claim Status"
5. Exclude columns: Policy Holder ID, Claim ID, Claim Amount
6. Launch training (result: ~71% accuracy)
7. Test with "Single Prediction":
   - Accident date: July 15
   - Claim date: July 20
   - Vehicle: 2017 Ford Escape
   - Amount: $8,000
   → Result: Rejected ✅

Deployment configuration:

Click "Create Deployment Model":
├── Name: claims-analyzer-deployment
├── Instance type: ml.t2.medium  (smallest available)
└── Instance count: 1

→ Status: InService ✅
→ Inference type: Real-time
→ Endpoint URL: available on the details page

Advanced configuration from the AWS console:

SageMaker → Inference → Endpoints
├── Available operational metrics:
│   ├── CPU Utilization
│   ├── Memory Utilization
│   ├── Disk Utilization
│   ├── GPU Utilization
│   └── GPU Memory Utilization
└── Invocation metrics:
    ├── 4xx / 5xx Errors
    ├── Invocations per Instance
    └── Model Latency

4.5 Updating a Deployment and Autoscaling

Problem: T2 instances don’t support autoscaling → create a new endpoint configuration.

sequenceDiagram
    participant U as User
    participant SM as SageMaker Console
    participant EP as Endpoint
    participant EC2 as EC2 Instances

    U->>SM: Change endpoint configuration
    U->>SM: Create "v2-autoscaling-configuration"
    SM->>SM: Select ml.m5.large
    SM->>EP: Update endpoint
    EP-->>SM: InService ✅

    U->>SM: Configure autoscaling
    SM->>EP: Apply scaling policy
    Note over EC2: Min: 1 instance / Max: 2 instances
    Note over EC2: Trigger: > 100 invocations/instance

    U->>EP: Normal traffic (< 100 req/instance)
    EP->>EC2: 1 active instance
    U->>EP: Traffic spike (> 100 req/instance)
    EP->>EC2: Scale out → 2 instances
    U->>EP: Traffic returns to normal
    EC2->>EP: Scale in → 1 instance (after cooldown)

Autoscaling configuration in the console:

4.6 Cost Optimization

Amazon SageMaker Neo

SageMaker Neo optimizes models for a specific target platform, reducing resource requirements.

flowchart LR
    A[Build the model\nMXNet/TF/PyTorch/XGBoost]
    B[Train the model]
    C[Choose the target platform\nCloud / Edge Device]
    D[Optimize with\nSageMaker Neo]
    E[Deploy\nOptimized Model]

    A --> B --> C --> D --> E

    style D fill:#FF9900,color:#fff

SageMaker Neo benefits:

• Faster model on the target platform
• Fewer resources needed for inference
• Reduced inference costs
• Supports: Apache MXNet, TensorFlow, PyTorch, XGBoost

Autoscaling

Autoscaling automatically adjusts the number of EC2 instances based on demand:

Low traffic   → Fewer instances → Cost savings
Traffic spike → More instances  → Performance

Cost optimization strategies summary:

┌──────────────────────────────────────────────────────────────┐
│            SAGEMAKER COST OPTIMIZATION                       │
├──────────────────────────────────────────────────────────────┤
│  1. SageMaker Neo                                            │
│     • Optimizes model for the target platform                │
│     • Reduces resource consumption at inference              │
├──────────────────────────────────────────────────────────────┤
│  2. Autoscaling                                              │
│     • Dynamically adjusts instance count                     │
│     • Scales out during traffic spikes                       │
│     • Scales in during quiet periods                         │
├──────────────────────────────────────────────────────────────┤
│  3. Serverless Inference                                     │
│     • Pay-per-use: cost only when invocations occur          │
│     • Ideal for unpredictable or infrequent traffic          │
├──────────────────────────────────────────────────────────────┤
│  4. Stop unused Notebook Instances                           │
│     • No charges when the instance is stopped                │
│     • SageMaker Studio: free interface (zero cost)           │
└──────────────────────────────────────────────────────────────┘

General Summary

mindmap
  root((Amazon\nSageMaker))
    Introduction
      Machine Learning
        Dataset / Features / Labels
        Training Job → Model
        Inferences / Predictions
      Jupyter Notebooks
        Code Cells
        Markdown Cells
      Notebook Instances
        Pre-configured VM
        Pay-as-you-go
      SageMaker Studio
        Complete ML IDE
        Free interface
        JumpStart GenAI
    Training
      Built-in Algorithms
        XGBoost
        17 algorithms available
        Optimized and scalable
      Hyperparameter Tuning
        Grid Search
        Random Search
        Bayesian Optimization
        Hyperband
        Early Stopping
      Custom algorithms
        Docker + ECR
        PyTorch / TF / Custom
    Evaluation
      Classification
        Confusion Matrix TP/FP/FN/TN
        Accuracy / Precision / Recall
        F1 Score / AUC
      Regression
        MSE / RMSE
        R² / MAE / MAPE
      Tools
        SageMaker Canvas no-code
        Python SDK sklearn
        Model Monitor Clarify
    Deployment
      Inference types
        Real-time
        Serverless
        Async
      Options
        Canvas no-code
        Python SDK
        CloudFormation
      Optimization
        SageMaker Neo
        Autoscaling
        Serverless inference

References and Resources

• Official Amazon SageMaker Documentation
• SageMaker Built-in Algorithms — Developer Guide
• SageMaker Model Monitor
• SageMaker Clarify

Search Terms

amazon · sagemaker · aws · ai · machine · web · services · deployment · evaluation · models · job · metrics · algorithm · algorithms · autoscaling · built-in · canvas · confusion · custom · ecr · matrix · optimization · tuning · xgboost