Machine Learning in the Enterprise

Case study approach — the fictional XYZ team (data analysts, data scientists, developers, ML engineers)

Table of Contents

1. Introduction
2. Understanding the Enterprise ML Workflow
3. Enterprise Data Management
   • 3.1 Feature Store
   • 3.2 Data Catalog
   • 3.3 Dataplex
   • 3.4 Analytics Hub
   • 3.5 Data Preprocessing Options
   • 3.6 Dataprep
4. ML Science and Custom Training
   • 4.1 The Art and Science of ML — Learning Rate and Batch Size
   • 4.2 Accelerating Training — Distributed Training
   • 4.3 When to Use Custom Training
   • 4.4 Training Prerequisites and Dependencies
   • 4.5 Training Custom Models with Vertex AI
5. Vertex Vizier — Hyperparameter Tuning
6. Predictions and Model Monitoring with Vertex AI
   • 6.1 Batch and Online Predictions
   • 6.2 Model Monitoring
7. Vertex AI Pipelines
8. Best Practices for ML Development
   • 8.1 Model Deployment and Serving
   • 8.2 Model Monitoring
   • 8.3 Pipeline Best Practices
   • 8.4 Artifact Organization
9. Vertex AI Tools Summary

1. Introduction

This course takes a case study approach: a fictional team of data analysts, data scientists, software developers, and ML engineers — called the XYZ team — works across multiple ML/AI projects.

Course Objectives

By the end of this course, you will be able to:

• Describe data management, governance, and preprocessing options
• Identify when to use Vertex AutoML, BigQuery ML, and custom training
• Implement Vertex Vizier Hyperparameter Tuning
• Explain how to create batch and online predictions
• Configure model monitoring
• Build pipelines with Vertex AI

Prerequisites — XYZ Team Skills

All team members are familiar with AutoML and BigQuery ML, but not yet with custom training on Vertex AI.

2. Understanding the Enterprise ML Workflow

Overview

Enterprise ML development typically involves two primary activities:

flowchart TD
    A[Measurable Success Criteria] --> B[Experimentation\nData Science Workflow]
    A --> C[Training Ops\nOperationalization]

    B --> B1[Feature Dataset Store]
    B1 --> B2[Experimentation Platform]
    B2 --> B3[Problem Refinement\nData Selection\nExploration\nFeature Engineering]
    B3 --> B4[Model Prototyping\nAlgo · Training · HP Tuning · Evaluation]
    B4 --> B5{One-off case?}

    B5 -->|Yes| D[Model Registry]
    B5 -->|No| C

    C --> C1[Automated Training Pipeline]
    C1 --> C2[Inference Scripts\nPreprocessing · Post-processing]
    C2 --> D

    D --> E[Model Deployment]

    E --> F[Production Environment]
    F --> G[Online inference\nReal-time REST API]
    F --> H[Streaming inference\nNear real-time]
    F --> I[Batch inference\nOffline / ETL]
    F --> J[Embedded inference\nIoT / Mobile]

    F --> K[Explainable AI\nFeature attribution]
    F --> L[Model Monitoring]

Key Components of the Enterprise ML Workflow

graph LR
    subgraph Sources
        S1[Data Engineering Pipelines]
    end

    subgraph Storage
        S2[Feature Dataset Store]
        S3[Source Code Repository\nGit / notebooks / scripts]
        S4[ML Metadata & Artifact Store\nParams · Evaluations · Definitions]
    end

    subgraph Vertex AI
        V1[Vertex AI Training]
        V2[Model Registry]
        V3[Vertex AI Prediction]
        V4[Vertex AI Monitoring]
        V5[Vertex AI Experiments]
        V6[Vertex ML Metadata]
        V7[Explainable AI]
    end

    S1 --> S2
    S2 --> V1
    S3 --> V1
    V1 --> S4
    V1 --> V2
    V2 --> V3
    V3 --> V4

Inference Types

3. Enterprise Data Management

3.1 Feature Store

Common Feature Management Challenges

The XYZ team faces three major challenges:

graph TD
    P1[Features difficult\nto share and reuse]
    P2[Low-latency production\nserving]
    P3[Training-Serving Skew\ndifferences between train and prod]

    P1 --> SOL[Vertex AI Feature Store]
    P2 --> SOL
    P3 --> SOL

What Is Vertex AI Feature Store?

Feature Store is a fully managed solution that:

• Manages and scales the underlying infrastructure (storage and compute)
• Provides a centralized feature repository with search and discovery APIs
• Computes feature values once and reuses them for both training and serving
• Supports batch and real-time (stream) feature ingestion

Vertex AI Feature Store Architecture

graph LR
    subgraph APIs
        A1[Batch Import API\nFeature value ingestion]
        A2[Online Serving API\nLow latency · Small batches]
        A3[Batch Serving API\nHigh throughput · Large volumes]
    end

    subgraph Storage
        OS[Online Store\nRecent data]
        OFS[Offline Store\nFull history]
    end

    subgraph Usages
        U1[Online Predictions\nReal-time]
        U2[Model Training]
        U3[Batch Predictions\nOffline]
    end

    A1 --> OS
    A1 --> OFS
    A2 --> OS --> U1
    A3 --> OFS --> U2
    A3 --> OFS --> U3

Feature Store Workflow

sequenceDiagram
    participant DS as Data Scientist
    participant FS as Feature Registry
    participant OST as Online Store
    participant OFS as Offline Store
    participant MOD as Production Model

    DS->>FS: Register feature (Discovery API)
    DS->>OFS: Ingest by batch (Batch Import API)
    DS->>OST: Ingest in real-time (Stream API)

    DS->>OFS: Retrieve batch for training\n(Point-in-time lookup)
    Note over DS,OFS: Prevents data leakage

    MOD->>OST: Online serving request (low latency)
    OST-->>MOD: Current feature values

Entity View in Feature Store

The entity view also shows the graphical distribution of values per feature, along with min, median, max, and zero counts.

3.2 Data Catalog

Enterprise Data Asset Management Challenges

mindmap
  root((Data Challenges))
    Discovery
      Unknown or unfindable data
      Undocumented data
      Stale documentation
    Understanding
      Data freshness and validation
      Duplicate datasets
      Relationships between datasets
      Ownership and transformations
    Accessibility
      No self-service
      Long access request processes
      Producer bottlenecks

What Data Catalog Can Index

Natively supported Google Cloud sources:

• BigQuery: datasets, tables, and views
• Pub/Sub: topics
• Dataproc Metastore: services, databases, and tables

Additional capabilities:

• APIs to create and manage entries for custom resource types
• Add custom metadata tags to cataloged assets

3.3 Dataplex

Overview

Dataplex is an intelligent data fabric that lets organizations manage, monitor, and govern their data centrally across:

• Data lakes
• Data warehouses
• Data marts

Dataplex Logical Structure

graph TD
    subgraph Organization
        L1[Lake: Retail]
        L2[Lake: Sales]
        L3[Lake: Finance]
    end

    L1 --> Z1[Zone: Landing\nRaw data]
    L1 --> Z2[Zone: Curated Analytics\nAnalysis-ready data]
    L1 --> Z3[Zone: Curated Data Science\nML-ready data]

    Z1 --> A1[Asset: Cloud Storage Bucket]
    Z2 --> A2[Asset: BigQuery Dataset]
    Z3 --> A3[Asset: Cloud Storage\nMulti-project]

Dataplex Key Capabilities

Automatic Metadata Pipeline

When Parquet files are written to a Cloud Storage bucket, Dataplex:

1. Automatically extracts metadata
2. Detects the tabular schema (including Hive partitions)
3. Runs data quality checks
4. Makes the data queryable in BigQuery as an external table
5. Publishes to BigQuery, Dataproc Metastore, and Data Catalog

3.4 Analytics Hub

Problems with Traditional Data Sharing

graph LR
    subgraph Traditional
        T1[Batch extraction\nflat files] --> T2[Transmission] --> T3[Ingestion\ndatabases]
    end

    T1 --> P1[Expensive pipelines]
    T2 --> P2[Stale data]
    T3 --> P3[Multiple copies]
    T3 --> P4[Governance bypassed]

Analytics Hub — Modern Solution

graph TD
    subgraph Data Ecosystem
        PE[Public exchanges\nWorld Bank...]
        IE[Sector exchanges\nHealth · Retail...]
        CE[Commercial exchanges\nLogistics · Energy...]
        GE[Google Exchanges\nPatents · Web Analytics · Trends]
    end

    subgraph Roles
        PUB[Data Publisher\nPublishes shared datasets]
        ADM[Exchange Administrator\nManages listings]
        SUB[Data Subscriber\nSubscribes to datasets]
    end

    subgraph Components
        PP[Publisher Project\nShared Datasets]
        SP[Subscriber Project\nLinked Datasets]
        EX[Exchange\nData collections]
    end

    PUB --> PP --> EX
    ADM --> EX
    EX --> SP --> SUB

Benefits:

• Real-time sharing without multiple copies
• Built-in governance and traceability
• Data monetization possible (subscriptions and entitlements)

3.5 Data Preprocessing Options

Decision Tree for Preprocessing

flowchart TD
    START[What type of data?] --> TAB{Tabular data?}
    TAB -->|Yes| BQ[BigQuery\nSQL transformation\nMaterialized table]
    TAB -->|No| UNST{Volume of\nunstructured data?}

    UNST -->|Large volume| DF[Dataflow\nApache Beam\nConversion to TFRecord]
    UNST -->|Spark investment| DP[Dataproc\nHadoop + Spark ETL]
    UNST -->|Small in-memory dataset| PY[Python script\nOne-off]

    START --> STREAM{Streaming or\nnon-SQL?}
    STREAM -->|Yes| DFP[Dataflow + pandas]

    START --> TF{TensorFlow framework?}
    TF -->|Yes| TFX[TensorFlow Extended\nTensorFlow Transform]

Preprocessing Tools Comparison

# Example: Transforming tabular data with BigQuery
# Create a materialized view after transformation
CREATE OR REPLACE TABLE `project.dataset.processed_features` AS
SELECT
    customer_id,
    SAFE_DIVIDE(purchase_amount, num_transactions) AS avg_purchase,
    DATE_DIFF(CURRENT_DATE(), last_purchase_date, DAY) AS days_since_purchase,
    CASE WHEN age BETWEEN 18 AND 35 THEN 'young'
         WHEN age BETWEEN 36 AND 55 THEN 'middle'
         ELSE 'senior' END AS age_group
FROM `project.dataset.raw_customers`
WHERE purchase_amount IS NOT NULL;

# Example: Dataflow pipeline with Apache Beam for TFRecord conversion
import apache_beam as beam
import tensorflow as tf

def convert_to_tfrecord(element):
    feature = {
        'input_val': tf.train.Feature(float_list=tf.train.FloatList(value=[element['input_val']])),
        'target': tf.train.Feature(int64_list=tf.train.Int64List(value=[element['target']]))
    }
    example = tf.train.Example(features=tf.train.Features(feature=feature))
    return example.SerializeToString()

with beam.Pipeline() as pipeline:
    (pipeline
     | 'ReadFromBigQuery' >> beam.io.ReadFromBigQuery(query='SELECT * FROM dataset.events')
     | 'ConvertToTFRecord' >> beam.Map(convert_to_tfrecord)
     | 'WriteTFRecord' >> beam.io.WriteToTFRecord('gs://bucket/output/data'))

# Example: TensorFlow Transform (TFX) for preprocessing
import tensorflow_transform as tft

def preprocessing_fn(inputs):
    """Transformation function defined once, reused for both training and serving."""
    outputs = {}
    outputs['normalized_age'] = tft.scale_to_z_score(inputs['age'])
    outputs['vocab_review'] = tft.compute_and_apply_vocabulary(inputs['review'])
    outputs['label'] = inputs['label']
    return outputs

3.6 Dataprep

Data Lifecycle in Dataprep

graph LR
    IN[Ingestion\nGoogle Cloud\nBigQuery\nCloud Storage\nLocal] --> DIS[Discover\nAnomalies\nCorrelations]
    DIS --> CL[Cleanse\nDetect and fix\ncorrupted data]
    CL --> ST[Structure\nChange format\nPivot rows/cols\nExtract JSON]
    ST --> EN[Enrich\nCombine sources\nDerive new values]
    EN --> VAL[Validate\nConformance to\nrequired dataset]
    VAL --> OUT[Analyze\nVisualization\nML Models]

Dataprep Architecture

graph TD
    subgraph Connection Sources
        C1[Upload/Download\nLocal machine]
        C2[Cloud Storage\nRead/Write]
        C3[BigQuery\nRelational content]
    end

    subgraph Dataprep
        FL[Flows\nSequences of Recipes]
        RC[Recipes\nTransformation steps]
        WR[Wranglers\nOperation library]
    end

    subgraph Execution
        DF[Dataflow Jobs\nUnder the hood]
    end

    subgraph Outputs
        BQ[BigQuery\nCleaned data]
    end

    C1 --> FL
    C2 --> FL
    C3 --> FL
    FL --> RC --> WR
    FL --> DF --> BQ

Dataprep Capabilities

• Automatically detects schemas, data types, possible joins, and anomalies
• Supports structured and unstructured data: CSV, JSON, relational tables
• Handles any size: megabytes to petabytes
• Detects more than 17 different data types
• Intelligent suggestions for predictive transformations
• Built on Dataflow → auto-scalable and schedulable

Available Wrangler Types

Dataprep Wranglers (main categories):
├── Filter / Select
├── Rename / Move columns
├── Split / Merge columns
├── Extract patterns (regex)
├── Enrich (lookup, join)
├── Aggregate (group by, pivot)
├── Format (dates, numbers, strings)
└── Validate (assertions, business rules)

4. ML Science and Custom Training

4.1 The Art and Science of ML — Learning Rate and Batch Size

Learning Rate

The learning rate controls the step size in the weight space.

graph LR
    LR_HIGH[Learning Rate too high\nex: 0.1] --> BOUNCE[Oscillations\nRisk of missing optimum]
    LR_LOW[Learning Rate too low\nex: 0.0001] --> SLOW[Very slow convergence\n3000+ epochs]
    LR_OK[Optimal learning rate\nex: 0.001 to 0.01] --> CONV[Fast and stable\nconvergence]

The default value for TensorFlow’s linear regressor is 0.2 or $\frac{1}{\sqrt{\text{number of features}}}$

Batch Size

The batch size controls how many samples the gradient is computed over.

Practical rules:

General rule: batch size between 40 and 100
Recommended maximum: 500
Recent research: mini-batch between 2 and 32 for best performance
Large batch size → use a smaller learning rate

On CIFAR-10, CIFAR-100, and ImageNet datasets, the best results were achieved with a batch size between m=2 and m=32.

# Configuring hyperparameters in TensorFlow
import tensorflow as tf

# Using tf.keras
network = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dense(1)
])

opt = tf.keras.optimizers.Adam(learning_rate=0.001)

network.compile(
    optimizer=opt,
    loss='mse',
    metrics=['mae']
)

training_history = network.fit(
    train_dataset,
    batch_size=32,       # Batch size
    epochs=100,
    validation_data=val_dataset
)

Why Is Distributed Computing Necessary?

The exponential growth of modern models:

AlexNet (2013)       : < 0.01 petaFLOPs/s·day
ResNet (2015)        : ~1 petaFLOP/s·day
Neural Architecture  : ~100 petaFLOPs/s·day
Search (Google, 2017)

→ 1000x increase in ~4 years

• The X axis of the error curve is logarithmic: doubling data size linearly reduces error rate
• More complex models help, but more data helps even more

4.2 Accelerating Training — Distributed Training

Options for Accelerating Training

graph TD
    OPT[How to accelerate training?]
    OPT --> D1[Use accelerators\nTPUs · GPUs]
    OPT --> D2[Optimize the\ndata input pipeline]
    OPT --> D3[Distributed Training\nParallelization across multiple devices]

Training Infrastructure Evolution

Level 1: Multi-core CPU machine
          → TensorFlow handles scaling automatically

Level 2: Machine + GPU/TPU (1 device)
          → Significant speedup

Level 3: Machine + multiple GPUs/TPUs
          → AllReduce recommended

Level 4: Multiple machines + multiple devices
          → Parameter Server or AllReduce
          → Up to hundreds of devices

Data Parallelism — Primary Architecture

graph TD
    DATA[Training dataset]
    DATA --> |Subset 1| W1[Worker 1\nSame model]
    DATA --> |Subset 2| W2[Worker 2\nSame model]
    DATA --> |Subset N| WN[Worker N\nSame model]

    W1 --> |Gradients| AGG[Gradient aggregation\nParameter update]
    W2 --> |Gradients| AGG
    WN --> |Gradients| AGG

    AGG --> |Updated parameters| W1
    AGG --> |Updated parameters| W2
    AGG --> |Updated parameters| WN

Comparison: Parameter Server vs AllReduce

graph LR
    subgraph Async Parameter Server
        PS[Parameter Server\nStores parameters]
        AW1[Worker 1] --> |Gradients| PS
        AW2[Worker 2] --> |Gradients| PS
        PS --> |Recent parameters| AW1
        PS --> |Recent parameters| AW2
    end

    subgraph Synchronous AllReduce
        SW1[Worker 1\nModel copy]
        SW2[Worker 2\nModel copy]
        SW1 <--> |Peer-to-peer\ncommunication| SW2
    end

Model Parallelism

When the model is too large to fit in memory on a single device:

• Split the model into smaller parts
• Each device processes the same training samples
• Example: different layers on different devices

4.3 When to Use Custom Training

AutoML vs Custom Training Comparison

graph TD
    USE{Use case} -->|Matches AutoML offerings| AUTO[Vertex AutoML]
    USE -->|Does not match| CUSTOM[Custom Training]
    USE -->|Mixed data\nimages + tabular| CUSTOM
    USE -->|Architecture control\nframework required| CUSTOM
    USE -->|Quick prototype\nbaseline| AUTO
    USE -->|Clustering, custom algo| CUSTOM

Advantages of Vertex AI Custom Training Service

• Automatic provisioning of compute resources (deprovisioned at completion)
• Modular architecture: code in a container = portable unit
• Parameters passed as arguments: data location, hyperparameters → no redeployment
• Reproducibility: each job is tracked with inputs, outputs, container image
• Logs in Cloud Logging, real-time monitoring
• Distributed training across multiple nodes in parallel

4.4 Training Prerequisites and Dependencies

Forms of Training Code Accepted by Vertex AI

graph TD
    CODE[Training code] --> OPT1[Python application\nwith Prebuilt Container]
    CODE --> OPT2[Custom Container Image\nDocker]

    OPT1 --> PR[PyTorch · scikit-learn\nTensorFlow · XGBoost]
    OPT2 --> CR[Any framework\nAny language]

    PR --> GCS[Export to\nCloud Storage]
    CR --> GCS

Training Project Structure

trainer/
├── __init__.py
├── task.py          # Main entry point
├── model.py         # Model definition
└── utils.py         # Utilities

requirements.txt     # pip dependencies
setup.py             # Python package configuration

gcloud Command for Local Execution

# Local execution to test before submitting to Vertex AI
gcloud beta ai custom-jobs local-run \
    --base-image-uri=BASE_IMAGE_URI \
    --working-dir=WORKING_DIRECTORY \
    --script=SCRIPT_PATH \
    --output-image-uri=OUTPUT_IMAGE_NAME \
    -- \
    --my-arg=value        # Arguments passed to the script

# Example with local GPU (Linux only)
gcloud beta ai custom-jobs local-run \
    --base-image-uri=gcr.io/cloud-aiplatform/training/tf-gpu.2-8:latest \
    --working-dir=./trainer \
    --script=task.py \
    --output-image-uri=my-training-image:latest \
    --gpu \
    -- \
    --epochs=10 \
    --batch-size=32

Python Dependency Management

# Option 1: requirements.txt in the working directory
# (processed automatically by the local-run command)
# requirements.txt:
tensorflow==2.8.0
pandas==1.4.0
scikit-learn==1.0.2

# Option 2: Via --requirements flag
gcloud beta ai custom-jobs local-run \
    --requirements=tensorflow==2.8.0,pandas==1.4.0

# Option 3: Via --extra-packages flag (local wheels or source dists)
gcloud beta ai custom-jobs local-run \
    --extra-packages=./my_lib-1.0.0-py3-none-any.whl

# Option 4: Via --extra-dirs flag (additional directories)
gcloud beta ai custom-jobs local-run \
    --extra-dirs=configs,data_utils

Authentication and Credentials

# Default: Application Default Credentials (ADC) mounted in the container

# To use a specific service account:
gcloud beta ai custom-jobs local-run \
    --service-account-key-file=/path/to/key.json \
    ...

Push to Artifact Registry

# Tag the image with the Artifact Registry repository
docker tag my-training-image:latest \
    us-central1-docker.pkg.dev/my-project/my-repo/my-training-image:latest

# Push the image
docker push \
    us-central1-docker.pkg.dev/my-project/my-repo/my-training-image:latest

# Note: maximum artifact size = 5 TB
# Note: if upload > 60 min: use auth other than access token (expires in 60 min)

4.5 Training Custom Models with Vertex AI

Complete Example: Keras Model with Cloud Build

# trainer/task.py — Minimal training example with TensorFlow/Keras
import tensorflow as tf
import os
import argparse

def get_args():
    parser = argparse.ArgumentParser()
    parser.add_argument('--epochs', type=int, default=10)
    parser.add_argument('--batch-size', type=int, default=32)
    parser.add_argument('--learning-rate', type=float, default=0.001)
    parser.add_argument('--model-dir', type=str,
                        default=os.environ.get('AIP_MODEL_DIR', './model'))
    return parser.parse_args()

def build_network():
    """Build a Sequential Keras model."""
    network = tf.keras.Sequential([
        tf.keras.layers.Dense(128, activation='relu', input_shape=(10,)),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(1)
    ])
    return network

def main():
    args = get_args()

    # Load data from Cloud Storage or BigQuery
    # (never store data alongside training code)
    (x_train, y_train), (x_val, y_val) = tf.keras.datasets.boston_housing.load_data()

    # Build and compile the model
    network = build_network()
    network.compile(
        optimizer=tf.keras.optimizers.Adam(learning_rate=args.learning_rate),
        loss='mse',
        metrics=['mae']
    )

    # Train
    network.fit(
        x_train, y_train,
        batch_size=args.batch_size,
        epochs=args.epochs,
        validation_data=(x_val, y_val)
    )

    # Evaluate
    loss, mae = network.evaluate(x_val, y_val)
    print(f"Validation Loss: {loss:.4f}, MAE: {mae:.4f}")

    # Save to Cloud Storage
    network.save(args.model_dir)
    print(f"Model saved to: {args.model_dir}")

if __name__ == '__main__':
    main()

Cloud Build Specification (cloudbuild.yaml)

# cloudbuild.yaml — Build and push the training image
steps:
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - 'build'
      - '-t'
      - '${_IMAGE_URI}'
      - '.'
  - name: 'gcr.io/cloud-builders/docker'
    args:
      - 'push'
      - '${_IMAGE_URI}'

substitutions:
  _IMAGE_URI: 'us-central1-docker.pkg.dev/${PROJECT_ID}/training-repo/my-trainer:latest'

images:
  - '${_IMAGE_URI}'

Submitting a Custom Training Job

# Submitting a custom training job from a notebook
from google.cloud import aiplatform

# Initialize
aiplatform.init(project='my-project', location='us-central1',
                staging_bucket='gs://my-staging-bucket')

# Path to the training script
script_path = 'trainer/task.py'

# URI of the prebuilt container
container_uri = 'us-docker.pkg.dev/vertex-ai/training/tf-cpu.2-8:latest'

# Create and submit the job
job = aiplatform.CustomTrainingJob(
    display_name='my-custom-training-job',
    script_path=script_path,
    container_uri=container_uri,
    requirements=['scikit-learn>=1.0'],
    model_serving_container_image_uri='us-docker.pkg.dev/vertex-ai/prediction/tf2-cpu.2-8:latest'
)

model = job.run(
    args=['--epochs=20', '--batch-size=64'],
    replica_count=1,
    machine_type='n1-standard-4',
    accelerator_type='NVIDIA_TESLA_T4',
    accelerator_count=1,
)

workerPoolSpecs Configuration for Distributed Training

# Worker pool configuration for distributed training
worker_pool_specs = [
    {
        "machineSpec": {
            "machineType": "n1-standard-8",
            "acceleratorType": "NVIDIA_TESLA_V100",
            "acceleratorCount": 4,
        },
        "replicaCount": 1,
        "containerSpec": {
            "imageUri": "us-central1-docker.pkg.dev/my-project/repo/trainer:latest",
            "args": ["--epochs=50", "--batch-size=128"],
        },
    }
]

Summary of 3 Methods for Building a Model

graph TD
    GOAL[Build an ML model]

    GOAL --> AUTO[AutoML\nPoint-and-click\nNo code required\nDeep learning via NAS\nEdge deployment possible]
    GOAL --> BQL[BigQuery ML\nSQL only\nData in BigQuery\nPre-built models]
    GOAL --> VTX[Vertex AI Custom Training\nCustom code\nPrebuilt algorithms\nMaximum flexibility]

    AUTO --> PRED[Vertex AI Prediction Service]
    BQL --> PRED
    VTX --> PRED

5. Vertex Vizier — Hyperparameter Tuning

Defining Hyperparameters

Hyperparameters are user-defined parameters that guide the learning process and control the trade-off between training accuracy and generalizability.

Examples of hyperparameters:

• Optimizer (Adam, SGD, RMSProp)
• Number of epochs
• Regularization parameters (L1, L2, Dropout)
• Number of hidden layers in a neural network
• Size of each layer

Hyperparameter Tuning Methods

graph TD
    METHODS[Hyperparameter Tuning Methods]

    METHODS --> GS[Grid Search\nExhaustive grid\nAll combinations]
    METHODS --> RS[Random Search\nRandom selection\nof combinations]
    METHODS --> BO[Bayesian Optimization\nUses past evaluations]

    GS --> GS_PROS[Exhaustive\nNo combination missed]
    GS --> GS_CONS[Very slow\nNo learning from past results]

    RS --> RS_PROS[Faster than\nGrid Search]
    RS --> RS_CONS[Misses some combinations\nNo learning]

    BO --> BO_PROS[Fewer iterations\nExploits past experience\nSkips unpromising regions]
    BO --> BO_CONS[More complex]

Methods Comparison

Vertex AI Vizier

Vertex AI Vizier is a black-box optimization service that:

• Supports Bayesian Optimization for hyperparameter tuning
• Minimizes the number of evaluations needed to find the optimum
• Skips regions of the parameter space deemed unpromising
• Integrates Cloud ML HyperTune for reporting results

Configuring a Hyperparameter Tuning Job

# config.yaml — HP tuning job configuration for Vertex AI
trainingInput:
  hyperparameters:
    goal: MAXIMIZE
    hyperparameterMetricTag: accuracy
    maxTrials: 20
    maxParallelTrials: 5
    enableTrialEarlyStopping: True
    params:
      - parameterName: learning_rate
        type: DOUBLE
        minValue: 0.0001
        maxValue: 0.1
        scaleType: UNIT_LOG_SCALE
      - parameterName: batch_size
        type: INTEGER
        minValue: 16
        maxValue: 256
        scaleType: UNIT_LINEAR_SCALE
      - parameterName: num_hidden_layers
        type: INTEGER
        minValue: 1
        maxValue: 5
      - parameterName: hidden_units
        type: DISCRETE
        discreteValues: [64, 128, 256, 512]

# trainer/task.py — Reporting metrics for HyperTune
import hypertune

def main():
    args = get_args()

    # ... model training ...

    # Report metric to Vertex AI Vizier
    hpt = hypertune.HyperTune()
    hpt.report_hyperparameter_tuning_metric(
        hyperparameter_metric_tag='accuracy',
        metric_value=val_accuracy,
        global_step=args.epochs
    )

# Submitting a HP tuning job
from google.cloud import aiplatform

job = aiplatform.HyperparameterTuningJob(
    display_name='hp-tuning-job',
    custom_job=custom_job,
    metric_spec={'accuracy': 'maximize'},
    parameter_spec={
        'learning_rate': aiplatform.hyperparameter_tuning.DoubleParameterSpec(
            min=0.0001, max=0.1, scale='log'
        ),
        'batch_size': aiplatform.hyperparameter_tuning.IntegerParameterSpec(
            min=16, max=256, scale='linear'
        ),
    },
    max_trial_count=20,
    parallel_trial_count=5,
    search_algorithm=None,  # None = Bayesian Optimization by default
)

job.run()

6. Predictions and Model Monitoring with Vertex AI

6.1 Batch and Online Predictions

Overview of Prediction Types

graph TD
    MOD[Trained and validated\nmodel]

    MOD --> BATCH[Batch Prediction\nAsynchronous\nMultiple simultaneous requests]
    MOD --> ONLINE[Online Prediction\nSynchronous / Real-time\nREST API]

    BATCH --> BO1[Input: CSV or BigQuery]
    BATCH --> BO2[Output: CSV or BigQuery]
    BATCH --> BO3[Use case: large-scale processing\nnot time-sensitive]

    ONLINE --> OO1[Deploy to endpoint]
    ONLINE --> OO2[1 request per API call]
    ONLINE --> OO3[Use case: real-time applications]

Containers for Predictions

graph LR
    CONT[Container type]

    CONT --> PRE[Prebuilt Container\nMinimal config\nHTTP prediction server\nOrganized by framework/version]
    CONT --> CUST[Custom Container\nCustom Docker image\nMust expose HTTP server\nLiveness + health checks]

    PRE --> FW[TensorFlow · PyTorch\nscikit-learn · XGBoost]
    CUST --> ANY[Any framework\nAny language]

Example: Batch Prediction with AutoML

# Create a batch prediction job
from google.cloud import aiplatform

batch_prediction_job = aiplatform.BatchPredictionJob.create(
    job_display_name='my-batch-prediction',
    model_name='projects/my-project/locations/us-central1/models/my-model',

    # Input data source
    gcs_source='gs://my-bucket/input/batch_input.jsonl',
    # Alternatively: bigquery_source='bq://project.dataset.table'

    # Prediction destination
    gcs_destination_prefix='gs://my-bucket/output/',
    # Alternatively: bigquery_destination_prefix='bq://project.dataset'

    # Resource configuration
    machine_type='n1-standard-4',
    starting_replica_count=1,
    max_replica_count=5,
)

batch_prediction_job.wait()
print(f"Status: {batch_prediction_job.state}")

Example: Online Prediction with Endpoint

# Deployment and online prediction
from google.cloud import aiplatform

# Retrieve the registered model
model = aiplatform.Model('projects/my-project/locations/us-central1/models/my-model')

# Create an endpoint
endpoint = model.deploy(
    deployed_model_display_name='my-deployed-model',
    machine_type='n1-standard-4',
    min_replica_count=2,    # Min 2 for high availability (SLA)
    max_replica_count=10,   # Autoscaling up to 10
    accelerator_type='NVIDIA_TESLA_T4',
    accelerator_count=1,
)

# Make a prediction
instances = [
    {"feature1": 0.5, "feature2": 1.2, "feature3": 3.4},
]

prediction = endpoint.predict(instances=instances)
print(prediction.predictions)

Custom Container for Prediction (Requirements)

# The custom container must expose these HTTP endpoints:
#
# GET  /health   → Return 200 if the server is healthy
# GET  /          → Liveness check
# POST /predict   → Accept prediction requests

# Minimal example with Flask
from flask import Flask, request, jsonify
import tensorflow as tf

app = Flask(__name__)
model = tf.saved_model.load('/model')

@app.route('/health', methods=['GET'])
def health():
    return jsonify({"status": "healthy"}), 200

@app.route('/predict', methods=['POST'])
def predict():
    data = request.json
    instances = data.get('instances', [])
    predictions = model(instances)
    return jsonify({"predictions": predictions.numpy().tolist()})

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

6.2 Model Monitoring

Model Monitoring Configuration Process

flowchart TD
    START[Navigate to Endpoints]
    START --> SEL[Select an endpoint]
    SEL --> SETTINGS[Click SETTINGS]
    SETTINGS --> ENABLE[Enable Model Monitoring\ntoggle]
    ENABLE --> NAME[Name the monitoring job]
    NAME --> WINDOW[Define the time window\nin hours]
    WINDOW --> FREQ[Define monitoring frequency\nex: every 6h]
    FREQ --> EMAIL[Provide an email address\nfor alerts]
    EMAIL --> THRESH[Configure alert thresholds\nskew + drift]
    THRESH --> ACTIVE[Monitoring active]

    ACTIVE --> SKEW[Skew Detection\nTraining vs Serving]
    ACTIVE --> DRIFT[Drift Detection\nEvolution of production data]
    ACTIVE --> ATTR[Feature Attribution\nMonitoring feature importance]

Skew Detection vs Drift Detection

Note: Model monitoring works for structured data (numeric and categorical features), but not for unstructured data (images, text).

Configuring Monitoring via the API

# Configure model monitoring for an endpoint
from google.cloud import aiplatform_v1

monitoring_config = {
    "alertConfig": {
        "emailAlertConfig": {
            "userEmails": ["ml-team@company.com"]
        }
    },
    "objectiveConfigs": [
        {
            "trainingDataset": {
                "dataFormat": "csv",
                "gcsSource": {
                    "uris": ["gs://my-bucket/training_data/"]
                },
                "targetField": "label"
            },
            "trainingPredictionSkewDetectionConfig": {
                "skewThresholds": {
                    "feature1": {"value": 0.3},
                    "feature2": {"value": 0.2},
                }
            },
            "predictionDriftDetectionConfig": {
                "driftThresholds": {
                    "feature1": {"value": 0.3},
                }
            }
        }
    ],
    "loggingSamplingStrategy": {
        "randomSampleConfig": {"sampleRate": 0.1}  # 10% of requests
    },
    "monitorInterval": {"seconds": 3600},  # Every hour
    "statsAnomaliesBase": {
        "minAnomaly_count": 1,
        "recent_time_interval": {"seconds": 3600}
    }
}

7. Vertex AI Pipelines

Definition

An ML pipeline is a sequence of code-based steps that automates the complete ML workflow:

Data Extraction → Preprocessing → Training → Evaluation → Deployment → Retraining

Supported SDKs

Vertex AI Pipeline Architecture

flowchart LR
    subgraph Pipeline Components
        C1[1. Data\nExtraction]
        C2[2. Preprocessing\n& Validation]
        C3[3. Feature\nEngineering]
        C4[4. Model\nTraining]
        C5[5. Model\nEvaluation]
        C6[6. Model\nDeployment]
        C7[7. Monitoring\n& Retraining]
    end

    C1 --> C2 --> C3 --> C4 --> C5 --> C6 --> C7

    subgraph Vertex AI Services
        NB[Notebooks]
        VT[Vertex Training]
        VP[Vertex Prediction]
        AS[Artifact Store]
    end

    C4 --> VT
    C6 --> VP
    C3 --> NB
    C5 --> AS

Kubeflow Pipelines Example

# pipeline.py — Pipeline example with Kubeflow Pipelines SDK
import kfp
from kfp import dsl
from kfp.v2 import compiler
from google.cloud import aiplatform
from google_cloud_pipeline_components import aiplatform as gcc_aip

# Environment variables
PROJECT_ID = 'my-project'
REGION = 'us-central1'
PIPELINE_ROOT = 'gs://my-bucket/pipeline_root'

@dsl.pipeline(
    name='vertex-ai-training-pipeline',
    description='Pipeline for training and deploying an AutoML model',
    pipeline_root=PIPELINE_ROOT
)
def pipeline(project: str = PROJECT_ID, region: str = REGION):

    # Step 1: Create the dataset
    dataset_create_op = gcc_aip.TabularDatasetCreateOp(
        display_name='my-tabular-dataset',
        gcs_source='gs://my-bucket/data/train.csv',
        project=project,
        location=region
    )

    # Step 2: Train the model
    # (takes the dataset from the previous step as input)
    training_job_run_op = gcc_aip.AutoMLTabularTrainingJobRunOp(
        display_name='my-automl-training-job',
        optimization_prediction_type='regression',
        dataset=dataset_create_op.outputs['dataset'],
        target_column='price',
        project=project,
        location=region
    )

    # Step 3: Create the endpoint
    endpoint_create_op = gcc_aip.EndpointCreateOp(
        display_name='my-endpoint',
        project=project,
        location=region
    )

    # Step 4: Deploy the model to the endpoint
    model_deploy_op = gcc_aip.ModelDeployOp(
        model=training_job_run_op.outputs['model'],
        endpoint=endpoint_create_op.outputs['endpoint'],
        dedicated_resources_machine_type='n1-standard-4',
        dedicated_resources_min_replica_count=1,
        dedicated_resources_max_replica_count=5,
    )

# Compile the pipeline
compiler.Compiler().compile(
    pipeline_func=pipeline,
    package_path='pipeline.json'
)

# Submit the pipeline
aiplatform.init(project=PROJECT_ID, location=REGION)
pipeline_job = aiplatform.PipelineJob(
    display_name='my-pipeline-run',
    template_path='pipeline.json',
    pipeline_root=PIPELINE_ROOT,
)
pipeline_job.run()

Example: Simple 3-Step Text Pipeline

# intro_pipeline.py — Introduction pipeline with text input
import kfp
from kfp.v2 import dsl
from kfp.v2.dsl import component, Output, Input, Artifact

@component(base_image='python:3.9')
def preprocess_op(text: str, output_text: Output[Artifact]):
    """Step 1: Text preprocessing."""
    processed = text.strip().lower()
    with open(output_text.path, 'w') as f:
        f.write(processed)

@component(base_image='python:3.9')
def train_op(input_text: Input[Artifact], model_output: Output[Artifact]):
    """Step 2: Simulated training."""
    with open(input_text.path, 'r') as f:
        text = f.read()
    # ... training logic ...
    with open(model_output.path, 'w') as f:
        f.write(f"model_trained_on: {text}")

@component(base_image='python:3.9')
def evaluate_op(model: Input[Artifact]) -> float:
    """Step 3: Evaluation."""
    with open(model.path, 'r') as f:
        content = f.read()
    return 0.95  # Simulated score

@dsl.pipeline(name='intro-pipeline')
def intro_pipeline(text_input: str = "Hello Vertex AI Pipelines!"):
    preprocess_task = preprocess_op(text=text_input)
    train_task = train_op(input_text=preprocess_task.outputs['output_text'])
    evaluate_task = evaluate_op(model=train_task.outputs['model_output'])

Scheduling with Cloud Scheduler

# Schedule pipeline execution with Cloud Scheduler
gcloud scheduler jobs create http my-pipeline-schedule \
    --schedule="0 2 * * 1" \
    --uri="https://us-central1-aiplatform.googleapis.com/v1/projects/my-project/locations/us-central1/pipelineJobs" \
    --message-body-from-file=pipeline_job_payload.json \
    --oauth-service-account-email=my-sa@my-project.iam.gserviceaccount.com \
    --location=us-central1

8. Best Practices for ML Development

8.1 Model Deployment and Serving

Deployment Checklist

graph TD
    DEP[Model Deployment]

    DEP --> HW[1. Choose appropriate hardware\nCPU/GPU VM types]
    DEP --> INP[2. Plan model inputs\nBatch vs Online]
    DEP --> SCALE[3. Enable autoscaling\nMin + Max nodes]
    DEP --> PERF[4. Define performance criteria\nBusiness metrics]

    HW --> CPU[CPU VMs\nn1-standard-*\nn1-highmem-*]
    HW --> GPU[GPU VMs\nNVIDIA T4 · V100 · A100]

    INP --> BATCH_IN[Batch: fetch from data lake\nor Feature Store Batch API]
    INP --> ONLINE_IN[Online: instances sent\nvia REST API]

    SCALE --> SLA[Autoscaling min=2 nodes\nfor high availability SLA]

    PERF --> METRICS[Business metrics:\nROC AUC · RMSE · F1\naligned with objective]

Retraining Assessment

Before putting the model into service, assess retraining needs:

Questions to ask:
How often does data change?
Must the model adapt to new trends?
What is the cost of a degraded model in production?
Can retraining be automated with a pipeline?

8.2 Model Monitoring

Monitoring Best Practices

graph TD
    MON[Model Monitoring Best Practices]

    MON --> SKD[Skew Detection\nTop priority if training\ndata is available]
    MON --> FIN[Fine-tune alert thresholds\nBased on use case +\ndomain expertise]
    MON --> ATTR[Feature Attributions\nDetect data drift or skew\nby feature importance]
    MON --> OUT[Outlier tracking\nIsolate anomalies]

    SKD --> BEST[Best indicator\nof model degradation]
    FIN --> ALERT[Email alerts\nwhen thresholds exceeded]
    ATTR --> EXAI[Vertex Explainable AI\nintegrated]
    OUT --> QUAL[Prediction quality\nmaintained]

Minimum monitoring frequency: 1 hour

Supported data types: Structured data only (numeric and categorical features)

8.3 Pipeline Best Practices

Why Use Pipelines?

• Automation: automatic training and deployment
• Monitoring: end-to-end governance of the ML workflow
• Reproducibility: each run is tracked with its artifacts
• Serverless orchestration: no infrastructure to manage

The 7 Components of the ML Workflow in a Pipeline

1. Data Extraction
2. Data Preprocessing
3. Feature Engineering
4. Model Training
5. Model Evaluation
6. Model Deployment
7. Monitoring & Retraining

Metadata Management Best Practices

graph LR
    RUN[Pipeline Run]

    RUN --> META[Vertex ML Metadata\nArtifacts + Metadata]
    META --> ASSESS[Assess accuracy\nWhy is this run so precise?]
    META --> COMP[Compare pipelines\nWhich run has the best perf?]
    META --> DEBUG[Debug\nWhat caused the error?]
    META --> AUDIT[Audit\nWho used this dataset?]

8.4 Artifact Organization

Artifact Lineage

An artifact lineage describes all factors that produced an artifact:

graph TD
    subgraph Model Inputs
        TD[Training data]
        VD[Validation data]
        ED[Evaluation data]
        HP[Hyperparameters]
        CODE[Training code]
    end

    subgraph Model
        MOD[Trained model\nv1.0]
    end

    subgraph Model Outputs
        BP[Batch Prediction\nResults]
        METRICS[Performance metrics\n(accuracy, AUC, RMSE)]
    end

    TD --> MOD
    VD --> MOD
    ED --> MOD
    HP --> MOD
    CODE --> MOD
    MOD --> BP
    MOD --> METRICS

Where to Store Artifacts?

Best Practices

# Use Git to version pipelines and custom components
git tag -a "pipeline-v1.2.0" -m "Improved preprocessing"
git push origin pipeline-v1.2.0

# Artifact Registry — store Docker images securely
# (without making them publicly visible)
gcloud artifacts repositories create ml-containers \
    --repository-format=docker \
    --location=us-central1 \
    --description="ML training containers"

# Tag and push to Artifact Registry
docker tag trainer:latest \
    us-central1-docker.pkg.dev/my-project/ml-containers/trainer:latest
docker push \
    us-central1-docker.pkg.dev/my-project/ml-containers/trainer:latest

9. Vertex AI Tools Summary

mindmap
  root((Vertex AI))
    Data
      Feature Store
        Feature centralization
        Online + Batch Serving
        Anti-skew training/serving
      Data Catalog
        Native metadata
        Custom tags
      Dataplex
        Data Fabric
        Unified governance
      Analytics Hub
        Cross-organization sharing
        No duplication
    Training
      AutoML
        No code
        All data types
      BigQuery ML
        SQL only
        Data in BQ
      Custom Training
        Maximum flexibility
        Distributed Training
      Vertex Vizier
        Hyperparameter Tuning
        Bayesian Optimization
    Deployment
      Online Predictions
        Real-time REST API
        Autoscaling
      Batch Predictions
        Asynchronous
        CSV or BigQuery
      Prebuilt Containers
        TF · PyTorch · sklearn
      Custom Containers
        Any framework
    Monitoring
      Skew Detection
        Train vs Serving
      Drift Detection
        Temporal evolution
      Feature Attribution
        Explainable AI
    Pipelines
      Kubeflow Pipelines
        General SDK
      TFX
        Large-scale TensorFlow
      Vertex ML Metadata
        Full traceability
      Artifact Registry
        Secure Docker images

Module Summary Table

Search Terms

machine · enterprise · ml · platforms · deployment · data · science · vertex · model · custom · feature · monitoring · store · management · pipeline · architecture · batch · comparison · dataprep · prediction · workflow · artifact · cloud · dataplex