Deploying Machine Learning Solutions

Table of Contents

1. Module 1 – Understanding Factors That Impact Deployed Models
   • Overview and Prerequisites
   • The Classic Machine Learning Workflow
   • New Realities of Deployed Models
   • Overfitting
   • Training-Serving Skew
   • Concept Drift
   • Coordinated Adversaries
   • Deployment Options
2. Module 2 – Deploying ML Models with Flask
   • Serializing Model Parameters
   • Demo – Serialization with JSON
   • Demo – Serialization with Pickle and Joblib
   • Demo – Checkpointing and Resuming Training
   • Demo – Serializing Preprocessors
   • Demo – Serializing scikit-learn Pipelines
   • Flask – Architecture and Deployment
3. Module 3 – Deploying to Serverless Environments
   • Event-Driven Serverless Compute
   • Serverless Execution Flow
   • Demo – Building Classification Models
   • Demo – Google Cloud Function
4. Module 4 – Deploying to Google AI Platform
   • Google AI Platform Overview
   • Models and Versions
   • Supported Prediction Types
   • gcloud Commands for Deployment
   • Monitoring with Stackdriver
   • Data Labeling Service
5. Module 5 – Deploying Deep Learning Models to AWS SageMaker
   • Amazon SageMaker Overview
   • SageMaker Training Workflow
   • Demo – Configuring the SageMaker Session
   • Demo – TensorFlow Estimator for Distributed Training
   • TensorFlow Training Script (mnist.py)
   • CNN MNIST Architecture
   • Demo – Deploying the Model for Predictions
   • Production Variants and A/B Testing
   • Demo – AWS CloudTrail for Auditing
6. Comparative Summary Table

Module 1 – Understanding Factors That Impact Deployed Models

Overview and Prerequisites

A machine learning model that performs well during development does not guarantee equivalent results once deployed in production. The main causes of performance degradation are:

• Overfitting: the model has memorized the training data too closely
• Training-serving skew: divergence between training and serving pipelines
• Concept drift: the relationship between features and labels changes over time
• Coordinated adversaries: malicious actors attempting to deceive the model

Model development does not end at deployment. Active maintenance is an integral part of the ML model lifecycle.

Recommended prerequisites:

• Python 3 (intermediate proficiency)
• scikit-learn (training, pipelines, metrics)
• Deep learning frameworks (TensorFlow / Keras)
• Basic cloud computing knowledge (AWS, GCP, or Azure)

The Classic Machine Learning Workflow

The traditional ML workflow follows a linear sequence from data collection to deployment. However, this sequence is cyclical in practice: deployment inevitably triggers a new retraining cycle.

flowchart LR
    A[Raw Data] --> B["Data\nPreprocessing"]
    B --> C["Algorithm\nSelection"]
    C --> D["Model\nTraining"]
    D --> E[Evaluation]
    E -->|Insufficient results| C
    E -->|Acceptable results| F[Deployment]
    F --> G[Retraining]
    G --> B

This lifecycle illustrates that deployment is not an endpoint but a step in a continuous model improvement process.

New Realities of Deployed Models

Unlike traditional software, ML models have fundamentally different properties that require a specialized maintenance approach.

ML Models vs. Traditional Software

Retraining Frequency by Domain

The optimal retraining frequency depends directly on how quickly the underlying data evolves:

• Real-time: cybersecurity, weather forecasting, news feeds – patterns change constantly
• Daily: fraud detection, content recommendations – fresh data every day
• Weekly: e-commerce, demand forecasting – slower but regular cycles
• Monthly or on-demand: real estate pricing models, credit scoring – relative stability

Geographic Localization of Models

Cultural, linguistic, and behavioral differences across geographic regions can make a model effective in one country and useless in another. Localization involves training or fine-tuning models on data representative of each target market.

Overfitting

Overfitting occurs when a model learns the details and noise of the training data at the expense of its ability to generalize to new data. The model performs excellently on training data but fails on unseen data.

Training vs. Test Comparison

An ideal model strikes a balance between the two: an acceptable training error and a similar test error (good generalization).

Techniques to Mitigate Overfitting

• Regularization (L1/Lasso, L2/Ridge): penalizes large coefficients to constrain model complexity
• Cross-validation (k-fold): evaluates the model on different data partitions for a robust estimate of real performance
• Dropout (deep learning): randomly disables neurons during training, forcing the network to learn redundant and robust representations
• Ensemble learning: combines predictions from multiple models (Random Forest, Gradient Boosting) to reduce variance

Training-Serving Skew

Training-serving skew refers to the divergence that can appear between the data preprocessing pipeline used during training and the one used during serving (production inference). Even a slight difference in how features are normalized, encoded, or transformed can produce incorrect predictions.

Common Causes

• Batch data used in training vs. streaming data in production
• Different versions of preprocessing libraries
• Different processing order between steps
• Missing values handled differently

Lambda Architecture

The Lambda architecture combines two layers to handle both historical (batch) and real-time (streaming) data:

graph LR
    A[Data Sources] --> B["Streaming Pipeline\nSpeed Layer"]
    A --> C["Batch Pipeline\nBatch Layer"]
    B --> D[Serving Layer]
    C --> D
    style B fill:#f9a825,color:#000
    style C fill:#1565c0,color:#fff
    style D fill:#2e7d32,color:#fff

• Speed Layer: processes data in real time with low latency
• Batch Layer: processes large volumes of historical data with high accuracy
• Serving Layer: merges both views to respond to queries

Kappa Architecture

The Kappa architecture simplifies Lambda by using a single pipeline to process both batch and streaming data. This ensures preprocessing is identical between training and serving.

graph LR
    A[Batch Data Sources] --> C["Single Pipeline\nKappa"]
    B[Streaming Data Sources] --> C
    C --> D[Serving Layer]
    style C fill:#6a1b9a,color:#fff
    style D fill:#2e7d32,color:#fff

Best practice: using the same preprocessing pipeline for training and serving eliminates training-serving skew. scikit-learn Pipeline objects serialized with Joblib are an elegant solution.

Concept Drift

Concept drift is the phenomenon where the statistical relationship between input features and output labels changes over time. The model, trained on historical data, gradually becomes ill-suited to new distributions.

Visual Example: Shifting Decision Boundary

Consider an SVM classifier that learns to separate two classes (positive / negative). Over time, new data points appear on the wrong side of the original boundary:

Before drift (t=0)            After drift (t=N)
                                
  ● ● ●  |  ○ ○ ○               ● ●  | ● ○ ○ ○
  ● ●    |    ○ ○               ● ●  |   ● ○ ○
  ●      |      ○                ●   |     ● ○
         |                           |
  ←──────→ decision                  ←────→ decision
           boundary (t=0)                   boundary (t=N)
                                            (must be updated)

The new ● points appear to the right of the original boundary — the model would classify them incorrectly without retraining.

Types of Concept Drift

• Sudden (sudden drift): abrupt change, for example during an economic crisis
• Gradual (gradual drift): slow progressive evolution of user behaviors
• Recurring (recurring drift): seasonal patterns (holiday shopping, summer behaviors)

Solutions

1. Continuous monitoring: track model performance metrics in production (accuracy, F1, prediction distribution)
2. Retraining on new data: regularly incorporate recent data into the training process
3. Full redevelopment: sometimes, when drift is too severe, it is necessary to completely rethink the model and its features

Coordinated Adversaries

Coordinated adversaries represent a category of threats specific to ML models deployed in hostile environments. These are malicious actors (hackers, fraudsters, competitors) who deliberately manipulate model inputs to obtain predictions favorable to their objectives.

Characteristics of Adversarial Attacks

• Non-repetitive: fraudulent patterns constantly change to avoid detection
• Sophisticated: exploit inherent flaws in ML models (e.g., adversarial examples for neural networks)
• Costly: late detection = financial losses or reputational damage

Concrete Examples

• Fraud detection: transactions designed to resemble legitimate behavior
• Spam filters: emails that bypass classifiers through text obfuscation
• Image recognition: slightly perturbed images that fool CNN networks

Recommended Solution: Human Minders

ML models alone are insufficient against coordinated adversaries, as they cannot detect patterns they have never seen. The solution is to combine machine learning with human oversight:

• Human analysts (human minders) review ambiguous or suspicious cases
• New attack techniques detected by humans feed the model retraining process
• A closed-loop feedback process continuously improves system robustness

Deployment Options

Several deployment strategies exist, each offering a different tradeoff between control, ease of implementation, and operational cost.

graph TD
    A[HTTP Endpoint] --> B["Web Framework\nFlask / Django"]
    A --> C["Serverless\nAWS Lambda / GCP Cloud Functions / Azure Functions"]
    A --> D["Cloud ML Platform\nGoogle AI Platform / AWS SageMaker"]
    B --> E[VM / Kubernetes Cluster]
    style A fill:#37474f,color:#fff
    style B fill:#1565c0,color:#fff
    style C fill:#e65100,color:#fff
    style D fill:#1b5e20,color:#fff
    style E fill:#4a148c,color:#fff

Module 2 – Deploying ML Models with Flask

Before serving an ML model via an HTTP API, it is necessary to serialize the trained model to disk, then deserialize it when the server starts. This section covers different serialization methods and complete model deployment via Flask.

Serializing Model Parameters

Serialization converts a Python object (trained model) into a persistent representation on disk. Three main approaches exist:

Recommendation: for scikit-learn models, Joblib is the preferred method as it is optimized for NumPy arrays which make up most of these models’ parameters.

Demo – Serialization with JSON

JSON serialization is useful for manually inspecting the parameters of a simple model (linear regression). However, it requires knowledge of the model’s internal structure to reconstruct it.

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
import json

# Load and prepare data (vehicle pricing dataset)
data = pd.read_csv('datasets/VehiclePrice_Dataset.csv')
data.drop(['vehicle_id', 'symboling', 'VehicleName'], axis=1, inplace=True)
data = pd.get_dummies(data)

X = data.drop('price', axis=1)
y = data['price']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train the model
regressor = LinearRegression()
regressor.fit(X_train, y_train)

print(f"R² training: {regressor.score(X_train, y_train):.2f}")  # ~0.94
print(f"R² test:     {r2_score(y_test, regressor.predict(X_test)):.2f}")  # ~0.89

# Serialize to JSON
model_params = {
    'coef': regressor.coef_.tolist(),
    'intercept': regressor.intercept_.tolist()
}
json_str = json.dumps(model_params)
with open('models/regressor_params.txt', 'w') as f:
    f.write(json_str)

# Deserialize from JSON
with open('models/regressor_params.txt', 'r') as f:
    params = json.load(f)

restored_model = LinearRegression()
restored_model.coef_ = np.array(params['coef'])
restored_model.intercept_ = np.array(params['intercept'])
print(f"R² restored JSON model: {r2_score(y_test, restored_model.predict(X_test)):.2f}")  # 0.89

The model deserialized from JSON produces exactly the same predictions as the original (R² = 0.89), confirming serialization fidelity.

Demo – Serialization with Pickle and Joblib

Pickle and Joblib allow serializing the model object in its entirety, without needing to know its internal structure.

import pickle
import joblib

# Pickle
with open('models/regressor.pkl', 'wb') as f:
    pickle.dump(regressor, f)

# Deserialize with Pickle
with open('models/regressor.pkl', 'rb') as f:
    pickle_model = pickle.load(f)

print(f"R² Pickle: {r2_score(y_test, pickle_model.predict(X_test)):.2f}")  # 0.89

# Joblib (better for sparse NumPy arrays)
joblib.dump(regressor, 'models/regressor.joblib')

# Deserialize with Joblib
joblib_model = joblib.load('models/regressor.joblib')
print(f"R² Joblib: {r2_score(y_test, joblib_model.predict(X_test)):.2f}")  # 0.89

All three methods produce identical results (R² = 0.89). The choice depends on context: JSON for human inspection, Pickle for universal compatibility, Joblib for performance with large models.

Demo – Checkpointing and Resuming Training

Checkpointing saves the state of a model at a given point, then resumes training from where it left off without starting over. This is particularly useful for Random Forests with warm_start=True.

from sklearn.ensemble import RandomForestRegressor

# Initial training on file1
df1 = pd.read_csv('datasets/vehicles_batch1.csv')
X_train1, y_train1 = df1.drop('price', axis=1), df1['price']

forest = RandomForestRegressor(n_estimators=5, warm_start=True)
forest.fit(X_train1, y_train1)
print(f"R² initial test: {r2_score(y_test, forest.predict(X_test)):.2f}")  # ~0.78

# Save the checkpoint
checkpoint = {
    'model': forest,
    'sklearn_version': sklearn.__version__,
    'test_r2': r2_score(y_test, forest.predict(X_test))
}
joblib.dump(checkpoint, 'models/forest_checkpoint.joblib')

# Resume on file2 (warm_start=True + increase n_estimators)
loaded = joblib.load('models/forest_checkpoint.joblib')
forest_loaded = loaded['model']
forest_loaded.n_estimators = 15  # Must increase for warm_start

df2 = pd.read_csv('datasets/vehicles_batch2.csv')
X_train2, y_train2 = df2.drop('price', axis=1), df2['price']
forest_loaded.fit(X_train2, y_train2)
print(f"R² after retraining: {r2_score(y_test, forest_loaded.predict(X_test)):.2f}")  # ~0.88

Important: with warm_start=True, you must increase n_estimators at each resume. If the value stays the same or decreases, scikit-learn will raise an error. Here we go from 5 to 15 trees.

The checkpoint serialization includes the scikit-learn version used, which facilitates debugging in case of incompatibility when reloading in a different environment.

Demo – Serializing Preprocessors

In an NLP pipeline, the preprocessor (here a TfidfVectorizer) is as important as the model itself. It must be serialized together with the model to ensure the same transformations are applied during serving.

from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
import sklearn

# Sentiment analysis dataset (~7000 movie reviews)
df = pd.read_csv('datasets/sentiment_data.csv')
X = df['ReviewText']
y = df['Sentiment']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# TF-IDF vectorization
vectorizer = TfidfVectorizer(max_features=15)
X_train_tfidf = vectorizer.fit_transform(X_train)
X_test_tfidf = vectorizer.transform(X_test)

# Train the classifier
clf = LinearSVC()
clf.fit(X_train_tfidf, y_train)
print(f"Accuracy: {accuracy_score(y_test, clf.predict(X_test_tfidf)):.2f}")  # 0.89

# Serialize both the model AND the vectorizer
checkpoint = {
    'model': clf,
    'vectorizer': vectorizer,
    'sklearn_version': sklearn.__version__,
    'accuracy': 0.89
}
joblib.dump(checkpoint, 'models/sentiment_checkpoint.joblib')

# Deserialize
reloaded = joblib.load('models/sentiment_checkpoint.joblib')
restored_vect = reloaded['vectorizer']
restored_clf = reloaded['model']
print(f"Reloaded accuracy: {accuracy_score(y_test, restored_clf.predict(restored_vect.transform(X_test))):.2f}")

Key point: if only the model is serialized without the vectorizer, deployment will be impossible — the vectorizer contains the vocabulary learned from training data.

Demo – Serializing scikit-learn Pipelines

The most elegant solution is to encapsulate the preprocessor and model in a single scikit-learn Pipeline object. This ensures both steps are always applied in the correct order and greatly simplifies serialization.

from sklearn.pipeline import Pipeline

# Full pipeline: vectorization + classification
text_pipeline = Pipeline([
    ('tfidf', TfidfVectorizer(max_features=15)),
    ('clf', LinearSVC())
])
text_pipeline.fit(X_train, y_train)
print(f"Pipeline accuracy: {accuracy_score(y_test, text_pipeline.predict(X_test)):.2f}")  # 0.89

# Save the entire pipeline
joblib.dump({'sentiment_pipeline': text_pipeline}, 'models/sentiment_pipeline.joblib')

# Reload and use directly for prediction
reloaded_pipe = joblib.load('models/sentiment_pipeline.joblib')
print(f"Reloaded pipeline accuracy: {accuracy_score(y_test, reloaded_pipe['sentiment_pipeline'].predict(X_test)):.2f}")

A serialized scikit-learn Pipeline is the ideal format for deployment: a single line of code to load and predict, with no risk of missing a preprocessing step.

Flask – Architecture and Deployment

Flask Server Architecture in Production

Flask alone is not sufficient for a production deployment. A complete stack includes:

HTTP Client Request
        │
   [nginx - Reverse Proxy]
        │  (SSL management, load balancing, static files)
        │
   [gunicorn - WSGI Server]
        │  (Python workers, concurrency management)
        │
   [Flask Application]
        │  (HTTP routing, business logic)
        │
   [Joblib → Scikit-learn Pipeline]
        │  (model deserialization at startup)
        │
   JSON Response

• nginx: reverse proxy that handles SSL connections, distributes load and serves static files efficiently
• gunicorn: WSGI server that launches multiple Python workers to handle concurrency
• Flask: lightweight web framework that defines routes and API logic

Demo – Deployment with Flask (server.py)

import numpy as np
from flask import Flask, request, jsonify
import joblib

app = Flask(__name__)

# Load model at startup
pipeline_data = joblib.load(open('models/sentiment_pipeline.joblib', 'rb'))
sentiment_clf = pipeline_data['sentiment_pipeline']

@app.route('/api/predict', methods=['POST'])
def predict():
    payload = request.get_json(force=True)
    prediction = sentiment_clf.predict(payload["reviews"])
    input_info = "Input:" + str(payload["reviews"])
    result = "Predicted class: " + str(prediction)
    return jsonify(input_info, result)

if __name__ == '__main__':
    app.run(port=8080, debug=True)

The model is loaded once at server startup (outside the predict function). This avoids reloading the file on every request, which would be very costly in terms of performance.

Python Client (client.py)

import requests

url = 'http://localhost:8080/api/predict'
response = requests.post(url, json={"reviews": ['I absolutely loved this film']})
print(response.json())

curl Requests

# Single review
curl -XPOST http://localhost:8080/api/predict \
  -H 'Content-Type: application/json' \
  -d '{"reviews": ["I absolutely loved this film"]}'

# Multiple reviews in one request
curl -XPOST http://localhost:8080/api/predict \
  -H 'Content-Type: application/json' \
  -d '{"reviews": ["I absolutely loved this film", "I found the film disappointing"]}'

The scikit-learn Pipeline natively accepts a list of texts as input and returns an array of predictions, enabling batch inference without modifying the server code.

Module 3 – Deploying to Serverless Environments

The serverless paradigm represents a major evolution in how applications are deployed. Rather than managing persistent servers, functions are deployed that execute on demand in response to events.

Event-Driven Serverless Compute

The Infrastructure Abstraction Spectrum

From bare metal to serverless, each additional level of abstraction reduces operational overhead but decreases control:

graph LR
    A[Bare Metal] --> B[Virtual Machine]
    B --> C["Container Cluster\nKubernetes"]
    C --> D[Hosted Application]
    D --> E["Serverless Function\nAWS Lambda\nGCP Cloud Functions\nAzure Functions"]
    
    style A fill:#f44336,color:#fff
    style B fill:#ff9800,color:#000
    style C fill:#ffc107,color:#000
    style D fill:#8bc34a,color:#000
    style E fill:#4caf50,color:#fff

Serverless Platforms by Cloud Provider

Advantages of the Serverless Model for ML

• Automatic scaling: from 0 to thousands of instances in seconds
• Pay-per-use: billed by execution, not by instance hour
• Reduced maintenance: no OS to patch, no server to manage
• Native integration: HTTP triggers, message queues, storage events

Limitations to Consider

• Cold starts: first invocation is slow (runtime + model loading)
• Memory limits: typically 1–10 GB depending on the platform
• Timeout: maximum execution duration is limited (15 min for Lambda)
• Package size: large models must be stored on Cloud Storage

Serverless Execution Flow

The typical flow of an ML inference via a Serverless function involves several steps, including downloading the model from Cloud Storage:

sequenceDiagram
    participant Client
    participant CP as Cloud Platform
    participant SF as Serverless Function
    participant CS as Cloud Storage

    Client->>CP: Event (e.g., HTTP POST)
    CP->>SF: Triggers the function
    SF->>CS: Download model.pkl
    CS-->>SF: Serialized model
    SF->>SF: Deserialize + Predict
    SF-->>Client: JSON Response

Downloading the model from Cloud Storage on every invocation creates a latency bottleneck. In production, it is recommended to implement an in-memory cache or use /tmp (available on most Serverless platforms) to avoid repeated re-downloads.

Demo – Building Classification Models

Three sentiment classification pipelines are built and serialized to be dynamically selected when the Cloud Function is invoked:

from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline
import pickle

# Dataset: ~7000 movie reviews
df = pd.read_csv('datasets/sentiment_data.csv')
X_train, X_test, y_train, y_test = train_test_split(df['ReviewText'], df['Sentiment'], test_size=0.2)

# Model 1: Logistic Regression (~89% accuracy)
lr_pipeline = Pipeline([('tfidf', TfidfVectorizer()), ('clf', LogisticRegression())])
lr_pipeline.fit(X_train, y_train)
pickle.dump(lr_pipeline, open('models/logistic_reg/model.pkl', 'wb'))

# Model 2: Decision Tree (~88.5% accuracy)
dt_pipeline = Pipeline([('tfidf', TfidfVectorizer()), ('clf', DecisionTreeClassifier(max_depth=10))])
dt_pipeline.fit(X_train, y_train)
pickle.dump(dt_pipeline, open('models/decision_tree/model.pkl', 'wb'))

# Model 3: LinearSVC (~89% accuracy)
svc_pipeline = Pipeline([('tfidf', TfidfVectorizer()), ('clf', LinearSVC())])
svc_pipeline.fit(X_train, y_train)
pickle.dump(svc_pipeline, open('models/linear_svc/model.pkl', 'wb'))

All three models achieve similar performance (~89% accuracy) on this dataset, but may behave differently on edge cases. Exposing multiple models through a single Cloud Function allows users to compare their predictions.

Demo – Google Cloud Function (main.py)

The Cloud Function receives the request, downloads the requested model from Cloud Storage, and returns the prediction:

import requests
import pickle
from google.cloud import storage

def classify_sentiment(request):
    
    if request.method == 'GET':
        return "Sentiment Classifier Ready"
    
    if request.method == 'POST':
        
        storage_client = storage.Client()
        bucket = storage_client.get_bucket('ml-sentiment-models')
        payload = request.get_json()
        
        if payload['model'] == ['DecisionTree']:
            blob = bucket.blob('models/decision_tree/model.pkl')
            blob.download_to_filename('/tmp/model.pkl')
            model = pickle.load(open('/tmp/model.pkl', 'rb'))
        elif payload['model'] == ['LinearSVC']:
            blob = bucket.blob('models/linear_svc/model.pkl')
            blob.download_to_filename('/tmp/model.pkl')
            model = pickle.load(open('/tmp/model.pkl', 'rb'))
        else:
            blob = bucket.blob('models/logistic_reg/model.pkl')
            blob.download_to_filename('/tmp/model.pkl')
            model = pickle.load(open('/tmp/model.pkl', 'rb'))
        
        reviews = payload['reviews']
        output = model.predict(reviews)
        result = 'Input:' + str(reviews) + '\nPredicted class: ' + str(output)
            
    return result

requirements.txt for Cloud Functions

requests==2.22.0
scikit-learn==0.21.3
google-cloud-storage==1.17.0

Dependency versions must exactly match those used when serializing the models. A version mismatch can make a .pkl file unreadable.

Invocation via curl

# Positive review with DecisionTree
curl -XPOST https://us-central1-ml-deploy-demo.cloudfunctions.net/classify_sentiment \
  -H 'Content-Type: application/json' \
  -d '{"model": ["DecisionTree"], "reviews": ["I absolutely loved this film"]}'

# With LinearSVC
curl -XPOST https://us-central1-ml-deploy-demo.cloudfunctions.net/classify_sentiment \
  -H 'Content-Type: application/json' \
  -d '{"model": ["LinearSVC"], "reviews": ["I absolutely loved this film"]}'

# Multiple reviews simultaneously
curl -XPOST https://us-central1-ml-deploy-demo.cloudfunctions.net/classify_sentiment \
  -H 'Content-Type: application/json' \
  -d '{"model": ["LinearSVC"], "reviews": ["I loved this film", "I disliked this film"]}'

Module 4 – Deploying to Google AI Platform

Google AI Platform (formerly Cloud ML Engine) is Google’s managed solution for model deployment, distributed training, and ML model lifecycle management. It integrates natively with the Google Cloud ecosystem.

Google AI Platform Overview

graph TD
    A[Google AI Platform] --> B["Cloud ML Engine\nDistributed Training\nBatch/Online Prediction"]
    A --> C["Data Labeling Service\nHuman Annotation"]
    A --> D["JupyterLab Notebooks\nPrototyping"]
    A --> E["Deep Learning VM Images\nGPU Support"]
    B --> F["Supported Frameworks:\nTensorFlow, Keras\nscikit-learn, XGBoost"]
    style A fill:#1a73e8,color:#fff
    style B fill:#34a853,color:#fff
    style C fill:#fbbc04,color:#000
    style D fill:#ea4335,color:#fff
    style E fill:#9c27b0,color:#fff

Google AI Platform offers a complete environment for the ML lifecycle:

• Cloud ML Engine: distributed training on GPU/TPU and model serving (batch and online)
• Data Labeling Service: human data annotation platform to create ground truth labels
• JupyterLab Notebooks: integrated prototyping environment with access to Google Cloud resources
• Deep Learning VM Images: pre-configured images with TensorFlow, PyTorch, CUDA for GPU training

Models and Versions

Google AI Platform organizes deployed models in a Model → Versions hierarchy. Each model can have multiple versions in production simultaneously, with one designated as the default.

graph LR
    M["Model\nsentiment_analysis_model"] --> V1["Version v1\nDecisionTree Classifier"]
    M --> V2["Version v2\nLinearSVC Classifier"]
    V1 -->|"default ✓"| P[Prediction Service]
    V2 --> P
    style M fill:#1a73e8,color:#fff
    style V1 fill:#34a853,color:#fff
    style V2 fill:#fbbc04,color:#000
    style P fill:#ea4335,color:#fff

This architecture enables:

• Versioning: complete traceability of each deployed model
• A/B comparisons: testing a new version against the stable version
• Instant rollback: returning to a previous version by changing the default version

Supported Prediction Types

• Online Prediction: real-time response, low latency, individual or small batch requests
• Batch Prediction: asynchronous processing of large data volumes, results written to Cloud Storage

gcloud Commands for Deployment

Inspecting a Deployed Version

# Environment variables
MODEL_NAME="sentiment_analysis_model"
VERSION_NAME="v1"

# Describe a deployed version
gcloud ai-platform versions describe $VERSION_NAME \
  --model $MODEL_NAME

# Prediction via gcloud CLI
INPUT_DATA_FILE="input.json"
gcloud ai-platform predict \
  --model $MODEL_NAME \
  --version $VERSION_NAME \
  --json-instances $INPUT_DATA_FILE

Format of input.json file

"These thriller films were really disappointing"

Deploying a New Version v2 (LinearSVC)

MODEL_DIR="gs://ml-sentiment-models/models/linear_svc/"
VERSION_NAME="v2"
MODEL_NAME="sentiment_analysis_model"
FRAMEWORK="scikit-learn"

gcloud ai-platform versions create $VERSION_NAME \
  --model $MODEL_NAME \
  --origin $MODEL_DIR \
  --runtime-version=1.13 \
  --framework $FRAMEWORK \
  --python-version=3.5

--origin points to the Google Cloud Storage directory containing the model.pkl file (for scikit-learn). The platform automatically detects the model file.

Prediction via REST API (curl)

# Direct REST API call
curl -X POST \
  -d '{"instances": ["These thriller films were really disappointing", "I loved this documentary"]}' \
  -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
  https://ml.googleapis.com/v1/projects/ml-deploy-demo/models/sentiment_analysis_model/versions/v2:predict

JSON Request Format for GCP AI Platform

{
  "instances": ["These thriller films were really disappointing"]
}

The Google AI Platform REST API conforms to RESTful standards and can be integrated into any application capable of sending authenticated HTTP requests.

Monitoring with Stackdriver

Google Cloud Stackdriver (now Google Cloud Operations Suite) provides native monitoring for models deployed on AI Platform.

Available Metrics

• Predictions/second: throughput of the prediction service
• Errors: error rate by HTTP code (4xx, 5xx)
• Latency: response time P50, P95, P99
• Response codes: distribution of HTTP response codes

Configuration

Customizable dashboards (e.g., sentiment_metrics_dashboard) allow creating automatic alerts when metrics deviate from normal values:

• Alert if P95 latency exceeds 500ms
• Alert if error rate exceeds 1%
• Alert if prediction volume drops sharply (possible outage)

Data Labeling Service

The Data Labeling Service closes the model lifecycle loop by allowing humans to annotate model predictions to create new labeled training data.

flowchart TD
    A[Model Prediction] -->|"10% sampling"| B["BigQuery Table\nevaluation_table"]
    B --> C["Human Reviewers\nGround Truth Labeling"]
    C --> D[Improved Training Data]
    D --> E[Model Retraining]
    E --> A
    style A fill:#1a73e8,color:#fff
    style B fill:#34a853,color:#fff
    style C fill:#fbbc04,color:#000
    style D fill:#ea4335,color:#fff
    style E fill:#9c27b0,color:#fff

BigQuery Evaluation Table Structure

ml-deploy-demo.dataset.evaluation_table
├── review_text      (review text)
├── predicted_class  (model prediction)
└── ground_truth     (correct label provided by human reviewers)

This workflow enables:

1. Capturing a representative sample of production predictions
2. Having human annotators validate these predictions
3. Using cases where the model was wrong to improve training data
4. Retraining the model with newly annotated data

Module 5 – Deploying Deep Learning Models to AWS SageMaker

Amazon SageMaker is AWS’s managed ML platform, designed to cover the entire ML lifecycle: from Jupyter prototyping to production deployment with monitoring and A/B testing.

Amazon SageMaker Overview

graph TD
    A[Amazon SageMaker] --> B["Notebook Instances\nJupyter on AWS"]
    A --> C["Distributed Training\nTensorFlow, PyTorch, MXNet\nscikit-learn"]
    A --> D["Deployment & Hosting\nAmazon Hosting Services"]
    A --> E["Built-in Algorithms\nReady-to-use algorithms"]
    A --> F["Ground Truth\nData Labeling"]
    C --> G["Estimator APIs\nHigh-level APIs"]
    D --> H["HTTP Endpoint\nProduction Variants"]
    style A fill:#ff9900,color:#000
    style B fill:#232f3e,color:#fff
    style C fill:#232f3e,color:#fff
    style D fill:#232f3e,color:#fff
    style E fill:#232f3e,color:#fff
    style F fill:#232f3e,color:#fff

Key Components

• Notebook Instances: Jupyter environments hosted on EC2, with direct access to SageMaker APIs
• Training Jobs: distributed training on dedicated EC2 instances, with GPU support (P3, G4)
• Hosting Services: model deployment on managed HTTPS endpoints with auto-scaling
• Built-in Algorithms: optimized and distributed algorithms (XGBoost, K-Means, PCA, etc.) ready to use
• Ground Truth: annotation platform similar to GCP’s Data Labeling Service

SageMaker Training Workflow

sequenceDiagram
    participant Notebook as SageMaker Notebook
    participant SageMaker
    participant EC2 as EC2 Instances
    participant S3

    Notebook->>SageMaker: mnist_estimator.fit(training_data_uri)
    SageMaker->>EC2: Spin up ML instances (ml.p3.2xlarge x2)
    SageMaker->>S3: Download training script
    EC2->>S3: Download training data
    EC2->>EC2: Distributed TensorFlow training
    EC2->>S3: Upload model artifacts
    S3-->>Notebook: Training complete

The SageMaker workflow is fully managed: the developer focuses on the Python training script, while SageMaker handles provisioning instances, distributing training, and storing model artifacts on S3.

Demo – Configuring the SageMaker Session

import os
import sagemaker
from sagemaker import get_execution_role

sm_session = sagemaker.Session()
execution_role = get_execution_role()

region = sm_session.boto_session.region_name
print(region)  # us-east-2

# URI of MNIST training data on S3
training_data_uri = 's3://sagemaker-sample-data-{}/tensorflow/mnist'.format(region)

• get_execution_role() retrieves the IAM role attached to the Notebook, which defines access permissions to AWS resources
• The SageMaker session handles authentication and region transparently

Demo – TensorFlow Estimator for Distributed Training

The SageMaker Estimator abstracts all the complexity of distributed training with just a few configuration parameters:

from sagemaker.tensorflow import TensorFlow

digit_estimator = TensorFlow(
    entry_point='digits.py',             # Training script
    role=execution_role,
    train_instance_count=2,              # 2 machines for distributed training
    train_instance_type='ml.p3.2xlarge', # GPU instance
    framework_version='1.12',
    py_version='py3',
    distributions={'parameter_server': {'enabled': True}}  # Distributed parameter server
)

# Launch training
digit_estimator.fit(training_data_uri)

The parameter_server option enables TensorFlow’s parameter server architecture for gradient synchronization between workers during distributed training. SageMaker automatically configures inter-instance communication.

TensorFlow Training Script (mnist.py)

import tensorflow as tf
import numpy as np

def build_cnn_model(features, labels, mode):
    """Defines the CNN for MNIST digit classification (digits 0-9)."""
    
    # Input Layer: 28x28 images → 4D tensor
    input_layer = tf.reshape(features["x"], [-1, 28, 28, 1])
    
    # Convolutional layer 1 + pooling
    conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, kernel_size=[5, 5],
                             padding="same", activation=tf.nn.relu)
    pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)
    
    # Convolutional layer 2 + pooling
    conv2 = tf.layers.conv2d(inputs=pool1, filters=64, kernel_size=[5, 5],
                             padding="same", activation=tf.nn.relu)
    pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)
    
    # Flatten + Dense layer + Dropout
    pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])
    dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
    dropout = tf.layers.dropout(inputs=dense, rate=0.4, training=(mode == tf.estimator.ModeKeys.TRAIN))
    
    # Logits layer (output: 10 classes)
    logits = tf.layers.dense(inputs=dropout, units=10)
    
    predictions = {
        "classes": tf.argmax(input=logits, axis=1),
        "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
    }
    
    if mode == tf.estimator.ModeKeys.PREDICT:
        return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)
    
    # Loss for training and evaluation
    loss = tf.losses.sparse_softmax_cross_entropy(labels=labels, logits=logits)
    
    if mode == tf.estimator.ModeKeys.TRAIN:
        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.001)
        train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
        return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
    
    # EVAL mode
    eval_metric_ops = {"accuracy": tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])}
    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, eval_metric_ops=eval_metric_ops)

def serving_input_fn():
    """Input function for SageMaker serving."""
    inputs = {'x': tf.placeholder(tf.float32, [None, 784])}
    return tf.estimator.export.ServingInputReceiver(inputs, inputs)

def main(argv):
    # Load data, distributed training and model export
    # tf.estimator.train_and_evaluate(...)
    # estimator.export_savedmodel(model_dir, serving_input_fn)
    pass

The serving_input_fn function is essential: it defines the data format expected by the model during serving. SageMaker uses it to create the exported TensorFlow SavedModel.

CNN MNIST Architecture

The convolutional network for MNIST follows a classic architecture with two conv+pool blocks followed by fully-connected layers:

Input [28x28 pixels, 1 channel]
    │
[Conv2D + ReLU] ─── 32 filters, 5x5 kernel, same padding
    │              → output: [28x28x32]
[MaxPooling 2x2] ── strides=2
    │              → output: [14x14x32]
    │
[Conv2D + ReLU] ─── 64 filters, 5x5 kernel, same padding
    │              → output: [14x14x64]
[MaxPooling 2x2] ── strides=2
    │              → output: [7x7x64]
    │
[Flatten] ────────── 7 × 7 × 64 = 3136 neurons
    │
[Dense: 1024 + ReLU]
    │
[Dropout: 40%] ───── (TRAIN mode only)
    │
[Logits: 10 classes] (one per digit 0-9)
    │
[Softmax → Probabilities] → predicted class = argmax

Demo – Deploying the Model for Predictions

# Deploy after training
predictor = digit_estimator.deploy(
    initial_instance_count=1, 
    instance_type='ml.p3.2xlarge',
    endpoint_name='digit-classification-2019-09-03-05-58-51-313'
)

# Download test data
import numpy as np
test_images = np.load('test_images.npy')
test_labels = np.load('test_labels.npy')

# Predictions on the first 50 examples
predictions = predictor.predict(test_images[:50])

# Example result:
# {'predictions': [
#   {'probabilities': [..., 0.724196, ...], 'classes': 7},
#   {'probabilities': [..., 0.997957, ...], 'classes': 3},
#   ...
# ]}

The predictor returned by deploy() encapsulates HTTP calls to the SageMaker endpoint. Each prediction includes:

• classes: the predicted class (digit 0-9)
• probabilities: the softmax probability vector (10 values summing to 1)

SageMaker Instance Types

Production Variants and A/B Testing

SageMaker supports Production Variants, which allow routing traffic to different model versions simultaneously. This is the foundation of A/B testing in production:

graph TD
    E[HTTP Endpoint] -->|"95% of traffic"| V1["Variant v1\nStable Model"]
    E -->|"5% of traffic"| V2["Variant v2\nNew Model"]
    V2 -->|"Performance validated ✓"| E2["100% to v2\n(full migration)"]
    style E fill:#ff9900,color:#000
    style V1 fill:#232f3e,color:#fff
    style V2 fill:#1a73e8,color:#fff
    style E2 fill:#4caf50,color:#fff

This approach enables:

1. Risk limitation: only 5% of users initially exposed to the new model
2. Real data collection: comparing performance of both variants on real traffic
3. Progressive migration: gradually increasing the new variant’s percentage
4. Immediate rollback: redirecting 100% of traffic to v1 if v2 degrades

Demo – AWS CloudTrail for Auditing

AWS CloudTrail records all AWS API calls, providing complete traceability of access to ML deployment resources.

Trail Configuration

• Name: digit-model-trail
• Storage bucket: digit-model-trail-logs
• Scope: monitors data events (R/W access) to the SageMaker S3 bucket

CloudTrail Event Structure

{
  "eventType": "AwsConsoleSignIn",
  "userIdentity": {"type": "Root"},
  "sourceIPAddress": "signin.amazonaws.com",
  "eventTime": "2019-09-03T...",
  "awsRegion": "us-west-2"
}

Log Collection Flow

flowchart LR
    A["AWS Resource Access\n(S3, SageMaker, EC2)"] --> B["CloudTrail Trail\ndigit-model-trail"]
    B --> C["S3 Bucket:\ndigit-model-trail-logs"]
    C --> D["AWSLogs/\nyyyy/mm/dd/\nevents.json.gz"]
    D --> E["Audit & Compliance\n(forensic analysis)"]
    style A fill:#ff9900,color:#000
    style B fill:#232f3e,color:#fff
    style C fill:#569a31,color:#fff
    style D fill:#232f3e,color:#fff
    style E fill:#1a73e8,color:#fff

CloudTrail is essential for:

• Security auditing: identifying who accessed models and when
• Regulatory compliance: GDPR, HIPAA, SOC2
• Forensics: reconstructing events in case of a security incident
• Anomaly detection: identifying unusual access to ML resources

Comparative Summary Table

After studying the four deployment options, here is a comparison table to guide choices based on project constraints and objectives:

Recommendations by Use Case

• Prototype / Proof of Concept → Flask on VM or Serverless: fast implementation, minimal cost
• High availability production, existing GCP stack → Google AI Platform: native integration, Stackdriver monitoring
• High availability production, existing AWS stack → SageMaker: managed endpoints, A/B testing, CloudTrail
• Multi-cloud or on-premise → Flask + Kubernetes: maximum portability, no vendor lock-in
• Low traffic, event-based scaling → Serverless: cost-effective outside peaks, zero infrastructure management

Search Terms

deploying · machine · ml · platforms · deployment · data · science · models · architecture · sagemaker · serverless · flask · cloud · curl · deployed · google · model · platform · serializing · types · aws · cloudtrail · concept · configuration