Deep Learning Model Deployment

Table of Contents

1. – Prepare models for production (22m 47s)

• 1.1 Course Introduction
• 1.2 Understanding template serialization formats
• 1.3 Save your X-ray model
• 1.4 Model quantization for speed
• 1.5 Create template package

2. – Deploy models as services (26m 5s)

• 2.1 Introduction to Model Serving
• 2.2 Building a REST API with FastAPI
• 2.3 Containerize your model service
• 2.4 Cloud deployment overview
• 2.5 Deploy to Google Cloud Run

3. – Monitor and maintain models in production (20m 7s)

• 3.1 Why do models degrade over time?
• 3.2 Monitor performance and detect drift
• 3.3 Model versioning and automated retraining
• 3.4 Balancing latency, scalability and costs

4. Appendix – Complete Code Files

1. – Prepare models for production

1.1 Course Introduction

Let’s say you have just finished training a deep learning model. Perhaps it is a model for computer vision, or perhaps for natural language processing. You have adjusted the hyperparameters, validated on your test set, and the accuracy is excellent. You feel pretty good about it. But here’s the problem: the template in your Jupyter Notebook isn’t really useful yet. This is what we call the last mile problem in machine learning, and it’s a bigger problem than most people realize when they start.

Welcome to Deep Learning Model Deployment. Here’s what we’ll cover in this course:

1. Prepare models for deployment – save them in portable formats like ONNX and TorchScript, optimize them with techniques like quantization and pruning, and package everything to run outside of your development environment.

2. Deploy our model as a real service – build REST APIs with FastAPI, containerize everything with Docker, and review different cloud deployment options so you can make informed decisions for your own projects.

3. Manage what happens after deployment – monitor performance degradation, detect when your model starts to drift, and set up automated workflows to keep everything running smoothly.

Throughout this course, we will work with a convolutional neural network that classifies chest X-rays to detect pneumonia. It’s a pre-trained model with around 88% accuracy, and we’ll take it from a saved checkpoint to a deployed service capable of processing real prediction queries.

1.2 Understanding Model Serialization Formats

You trained your model in PyTorch or TensorFlow. Now you need to save it for deployment. This is where things get interesting because the format you choose depends on two things: the framework you trained in, and where you deploy.

Framework native formats

When you train in PyTorch, you typically save your model using torch.save. When you train in TensorFlow, you use model.save, which creates a SavedModel format. These are your starting points. These are formats native to frameworks that work very well within their own ecosystems.

But these native formats have limits. A PyTorch model cannot run in a TensorFlow environment. A TensorFlow SavedModel needs TensorFlow installed at runtime. What if you want to deploy to a mobile device or use a specialized inference engine for better performance?

The three major formats

This is where different serialization formats come into play. Let’s talk about the main ones:

ONNX (Open Neural Network Exchange) – for cross-platform portability TorchScript – if you are in the PyTorch ecosystem TensorFlow SavedModel – if you are in the TensorFlow ecosystem

ONNX – Open Neural Network Exchange

ONNX is designed to be framework agnostic. You can train a model in PyTorch, export it to ONNX, and then run it with TensorFlow. It’s essentially a universal translator for neural networks. The key advantage here is flexibility. ONNX is supported by almost all major frameworks and inference engines. So if you’re not sure where your model will ultimately run, ONNX is a safe choice.

TorchScript

If you’ve trained in PyTorch, there’s TorchScript. This is PyTorch’s own serialization format, designed to bridge the gap between Python and production. Regular PyTorch models need Python at runtime, but TorchScript compiles your model into an intermediate representation that can run without Python at all. TorchScript models can also run in C++ applications, which is important for high-performance scenarios.

TensorFlow SavedModel and TFLite

If you are in the TensorFlow ecosystem, like our X-ray classifier, you already have the SavedModel format from training. This is TensorFlow’s native production format. SavedModel works great with TensorFlow Serving and other TensorFlow deployment tools.

But for mobile or edge deployment, you’ll want TensorFlow Lite (TFLite). It’s smaller, faster, and designed for resource-constrained environments.

Which format to choose?

1.3 Save your X-ray model

Let’s get to work and export our pneumonia detection model to different formats. The model is already saved in TensorFlow’s SavedModel format. You can see that it is a directory called xray_resnet152_model with the folders assets, variables, and the file saved_model.pb inside. This is TensorFlow’s native production format and our starting point. But we will also want to export to ONNX for cross-platform portability and TFLite for mobile deployment.

SavedModel model directory structure

xray_resnet152_model/
├── assets/
├── variables/
├── fingerprint.pb
├── keras_metadata.pb
└── saved_model.pb

Export script export_model.py

Create a new Python script called export_model.py:

import tensorflow as tf
import tf2onnx
import onnx
import numpy as np

# Charger le modèle pré-entraîné de classificateur X-ray
model = tf.keras.models.load_model("xray_resnet152_model")

print("Model loaded successfully")

# Créer un input de test - 1 batch, 224x224 pixels, 3 channels
test_input = np.random.rand(1, 224, 224, 3).astype(np.float32)

prediction = model.predict(test_input)
print("Inference successful. Prediction shape:", prediction.shape)
print("SavedModel prediction:", prediction)

# Convertir le modèle TF en ONNX
# Ce TensorSpec indique à ONNX la forme et le type de données à attendre
# Le None en première position est critique : batch size dynamique
spec = (tf.TensorSpec((None, 224, 224, 3), tf.float32, name="input"),)

model_proto, _ = tf2onnx.convert.from_keras(
    model,
    input_signature=spec,
    opset=13,                          # version 13 est largement supportée
    output_path="xray_classifier.onnx"
)

print("Model converted to ONNX format successfully")

# Vérifier que le modèle ONNX fonctionne
import onnxruntime

session = onnxruntime.InferenceSession("xray_classifier.onnx")

input_name = session.get_inputs()[0].name

test_input_onnx = np.random.randn(1, 224, 224, 3).astype(np.float32)
outputs = session.run(None, {input_name: test_input_onnx})

print(f"ONNX prediction shape: {outputs[0].shape}")
print(f"ONNX prediction: {outputs[0]}")

# Convertir en TensorFlow Lite pour déploiement mobile
converter = tf.lite.TFLiteConverter.from_saved_model("xray_resnet152_model")
tflite_model = converter.convert()

with open("xray_classifier.tflite", "wb") as f:
    f.write(tflite_model)
print("Model exported to TensorFlow Lite format")

# Vérifier que le modèle TFLite fonctionne
interpreter = tf.lite.Interpreter(model_path="xray_classifier.tflite")
interpreter.allocate_tensors()  # configure la mémoire pour les inputs/outputs

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

test_input_tflite = np.random.randn(1, 224, 224, 3).astype(np.float32)
interpreter.set_tensor(input_details[0]["index"], test_input_tflite)
interpreter.invoke()  # exécuter l'inférence

output_data = interpreter.get_tensor(output_details[0]["index"])

print(f"TFLite prediction shape: {output_data.shape}")
print(f"TFLite prediction: {output_data}")

Key Takeaways

• tf2onnx.convert.from_keras: opset version 13 is the most widely supported version of the ONNX operator set.
• None in the TensorSpec: means that the batch size is dynamic – one or more images can be processed at a time.
• TFLite: TFLite models work differently than regular TensorFlow models. We must use an Interpreter, call allocate_tensors(), then invoke().
• The three predictions may be slightly different because: the random inputs are different each time, and different inference engines handle floating point calculations slightly differently.

Execution: python export_model.py

1.4 Model Quantization for Speed

We have exported our model to ONNX and TFLite. Everything works. We’re ready for deployment, right? Not quite.

The problem with edge devices

Imagine deploying this X-ray classifier on a medical imaging device in a rural clinic, something like a Raspberry Pi connected to a portable X-ray scanner. Maybe you put it on a tablet that emergency responders carry in an ambulance. These are not high-end servers with powerful GPUs. These are edge devices with limited processing power and memory.

Here is what you will see:

• When you load our 230 MB model into memory and try to run an inference, it is slow. Really slow. We’re talking about 5, 10, maybe 15 seconds per prediction on a device without a GPU.
• On low-end embedded devices or tablets, you might only have 2-4 GB of total RAM. Loading a 230 MB model leaves less room for the operating system, other applications, and the X-ray image itself.
• There are also concerns about battery life and compute costs.

What is quantization?

Quantization means reducing the precision of the numbers in our model to make it smaller. Instead of 32-bit floating point numbers, 8-bit integers are used.

The key insight: During training, 32-bit precision is absolutely necessary. We calculate gradients, we update weights with small learning rates, we accumulate small changes over thousands of iterations. This precision is critical.

But for inference, we just do a forward pass. Precision does not provide additional value.

By switching to 8-bit integers, we obtain 3 major advantages:

1. The model is approximately 4 times smaller
2. Integer operations are fundamentally faster, especially on mobile processors and edge devices that have specialized integer hardware
3. Entire operations use less power, which is important for battery life

Quantization script quantize_model.py

import tensorflow as tf
import os

# Vérifier la taille de base avant quantification
original_size = os.path.getsize("xray_classifier.tflite") / (1024 * 1024)
print(f"Original SavedModel size: {original_size:.2f} MB")

# Quantification dynamic range
# Convertit les poids de float 32 bits en entiers 8 bits
# mais garde les activations en float pendant l'inférence
# C'est l'approche la plus simple et donne d'excellents résultats
converter = tf.lite.TFLiteConverter.from_saved_model("xray_resnet152_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quant_model = converter.convert()

with open("xray_classifier_dynamic_quant.tflite", "wb") as f:
    f.write(tflite_quant_model)

quantized_size = len(tflite_quant_model) / (1024 * 1024)
print(f"Dynamic Range Quantized model size: {quantized_size:.2f} MB")

Results observed

Original SavedModel size: 223.XX MB
Dynamic Range Quantized model size: ~57 MB

With just two lines of code (converter.optimizations = [tf.lite.Optimize.DEFAULT]), we reduce the size of the model from 223 MB to 57 MB, a reduction of approximately 75%. We now have a production-ready quantized model that is small and fast enough for real-world edge deployment.

Execution: python quantize_model.py

1.5 Create model package

We now have our quantized model. It’s small enough, optimized, and ready to deploy. But here’s the thing: you can’t just give someone a .tflite file and expect them to use it correctly.

What happens when someone else tries to deploy this template? He gets a new X-ray image. He needs to pass it through the model to get a prediction. But he needs to know: How do I preprocess this image? How big should it be? Do I normalize it? How ? If something is done wrong, the model will give absurd predictions, and it will have no idea why.

This is where correct packaging comes in. A good template package includes:

• The template file
• The preprocessing code
• Configuration documentation
• Dependency information

…everything needed for a reproducible deployment.

Package structure

# Créer la structure de répertoires
mkdir -p xray_model_package/model

# Copier le modèle quantifié dans le dossier model
cp xray_classifier_dynamic_quant.tflite xray_model_package/model/

xray_model_package/
├── model/
│   ├── xray_classifier_dynamic_quant.tflite
│   └── preprocessing.py
├── config.json
└── requirements.txt

Preprocessing code preprocessing.py

This code comes from the training process. It’s included so that anyone deploying the template knows exactly how to prepare the images.

import numpy as np
from PIL import Image

def preprocess_xray(image_path, target_size=(224, 224)):
    """
    Prétraiter l'image X-ray pour l'inférence du modèle.
    Correspond au prétraitement ImageDataGenerator de l'entraînement :
    - Redimensionner à 224x224
    - Convertir en RGB (3 channels)
    - Rescaler à [0, 1]
    
    Args:
        image_path: Chemin vers le fichier image X-ray
        target_size: Taille cible pour le redimensionnement (width, height)
    
    Returns:
        Tableau d'image prétraité prêt pour l'input du modèle
    """
    # Charger l'image
    img = Image.open(image_path)
    
    # Convertir en RGB - ImageDataGenerator fait ça automatiquement
    img = img.convert('RGB')
    
    # Redimensionner à 224x224
    img = img.resize(target_size)
    
    # Convertir en tableau numpy
    img_array = np.array(img, dtype=np.float32)
    
    # Rescaler à [0, 1] - correspond à rescale=1./255 de l'entraînement
    img_array = img_array / 255.0
    
    # Ajouter la dimension batch - le modèle attend la forme (1, 224, 224, 3)
    img_array = np.expand_dims(img_array, axis=0)
    
    return img_array

def postprocess_prediction(prediction):
    """
    Convertir la sortie du modèle en résultat lisible.
    Le modèle utilise l'activation sigmoid pour la classification binaire.
    
    Args:
        prediction: Sortie brute du modèle (valeur de probabilité unique)
    
    Returns:
        Dictionnaire avec la classe et le score de confiance
    """
    probability = float(prediction[0][0])
    
    # Classification binaire : > 0.5 = Pneumonia, <= 0.5 = Normal
    if probability > 0.5:
        class_name = "Pneumonia"
        confidence = probability
    else:
        class_name = "Normal"
        confidence = 1 - probability
    
    return {
        "class": class_name,
        "confidence": round(confidence, 4)
    }

Explanation of the two functions:

preprocess_xray takes a raw X-ray image file and does four things:

1. Load image with PIL
2. Converts to RGB with 3 channels
3. Resizes to 224 × 224 pixels
4. Rescales pixel values from [0-255] to [0-1] by dividing by 255
5. Add a batch dimension so that the shape becomes (1, 224, 224, 3)

postprocess_prediction takes the raw model output and converts it into a readable result, with the class name and confidence score.

Configuration file config.json

This file documents everything about the model:

{
    "model_name": "xray_pneumonia_classifier",
    "model_version": "1.0.0",
    "model_architecture": "ResNet152V2",
    "model_type": "tflite_quantized",
    "input_shape": [1, 224, 224, 3],
    "output_shape": [1, 1],
    "classes": ["Normal", "Pneumonia"],

    "preprocessing": {
        "resize": [224, 224],
        "color_mode": "rgb",
        "rescale": "divide_by_255",
        "notes": "Corresponds to ImageDataGenerator with rescale=1./255"
    },

    "training_info": {
        "framework": "tensorflow",
        "base_model": "ResNet152V2",
        "transfer_learning": true,
        "test_accuracy": 0.88
    },

    "optimization": {
        "quantization": "dynamic_range",
        "original_size_mb": 223,
        "quantized_size_mb": 57,
        "size_reduction_percent": 74.4
    }
}

This configuration file contains everything someone needs to know about the model: its architecture, how to preprocess the data, what performance to expect, and how it was optimized.

Dependency file requirements.txt

tensorflow==2.15.0
numpy==1.24.3
Pillow==10.0.0
fastapi==0.104.1
uvicorn==0.24.0
python-multipart==0.0.6

Note: Specifying exact versions is crucial for repeatable deployments.

Package usage

Now anyone who receives this package knows exactly what to do:

# 1. Installer les dépendances
pip install -r requirements.txt

# 2. Lire config.json pour comprendre le modèle
# 3. Utiliser preprocessing.py pour préparer les images
# 4. Charger le modèle depuis le dossier model/ et exécuter l'inférence

This package is not just a template file – it is a complete, documented, reproducible package.

2. – Deploy models as services

2.1 Introduction to Model Serving

We have exported our model in different formats, optimized with quantification, and packaged correctly with all the necessary documentation. But now comes the real deployment: turning this model into a service that applications can use.

What is “serving” a model?

Essentially, serving means making your model available as a service that applications can call to get predictions. Instead of the model living in a Python script on your laptop, it’s running somewhere – a server, a cloud instance, or even an edge device.

Two main approaches

Batch prediction

We process a large collection of inputs all at once. Maybe you have 1000 X-ray images that arrived overnight, and you want to run them all through your model in one go. This is useful for scenarios where you don’t need immediate results: generating reports, processing historical data, etc.

Real-time inference

This is when a user or application sends a single request and expects a response within milliseconds or seconds. For real-time inference, the standard approach is to expose your model via an API.

REST API for model serving

The most common type of API for model serving is a REST API. REST stands for Representational State Transfer, but what matters is that it uses standard HTTP requests – the same protocol your web browser uses.

A client sends an HTTP POST request with image data, and the API responds with a JSON object containing the prediction.

2.2 Building a REST API with FastAPI

Let’s build a REST API for our X-ray classifier. We will use FastAPI, which is a modern Python framework perfect for building APIs.

Imports and initialization

from fastapi import FastAPI, File, UploadFile, HTTPException
from fastapi.responses import JSONResponse
import tensorflow as tf
import numpy as np
from PIL import Image
import io
from xray_model_package.model.preprocessing import preprocess_xray, postprocess_prediction
import logging
from datetime import datetime
import json

app = FastAPI(
    title="X-Ray Pneumonia Classifier API",
    description="API for classifying chest X-rays as Normal or Pneumonia",
    version="1.0.0"
)

FastAPI uses this metadata to automatically generate interactive documentation.

Loading model at startup

model = None
interpreter = None

@app.on_event("startup")
async def load_model():
    global interpreter
    try:
        interpreter = tf.lite.Interpreter(
            model_path="xray_model_package/model/xray_classifier_dynamic_quant.tflite"
        )
        interpreter.allocate_tensors()
        logger.info("Model loaded successfully!")
    except Exception as e:
        logger.error(f"Error loading the model: {e}")
        raise

Important point: We do not want to load the model on each request. It would be incredibly slow. Instead, we load it once when the server starts. This decorator tells FastAPI to run this function exactly once on startup. We use async def because FastAPI is built on asynchronous programming – while a request is waiting (eg: file upload), the server can move on to another request.

API Endpoints

@app.get("/")
async def root():
    return {
        "message": "X-Ray Pneumonia Classifier API",
        "status": "running",
        "version": "1.0.0"
    }

@app.get("/health")
async def health_check():
    return {
        "status": "healthy",
        "model_loaded": interpreter is not None
    }

The /health endpoint is useful for monitoring. Other services can ping /health to check if our API is running. The model_loaded field checks if our interpreter is not None.

Prediction endpoint /predict

@app.post("/predict")
async def predict(file: UploadFile = File(...)):
    request_id = datetime.now().strftime("%Y%m%d%H%M%S%f")

    logger.info(f"Request {request_id}: received prediction request")

    if not file.content_type.startswith("image/"):
        logger.warning(f"Request {request_id}: Invalid file type {file.content_type}")
        raise HTTPException(
            status_code=400,
            detail="File must be an image"
        )
    
    try:
        start_time = datetime.now()

        # Lire et prétraiter l'image
        contents = await file.read()
        image = Image.open(io.BytesIO(contents))

        temp_path = "temp_image.jpg"
        image.save(temp_path)

        preprocessed = preprocess_xray(temp_path)

        # Exécuter l'inférence
        input_details = interpreter.get_input_details()
        output_details = interpreter.get_output_details()

        interpreter.set_tensor(input_details[0]["index"], preprocessed)
        interpreter.invoke()
        prediction = interpreter.get_tensor(output_details[0]["index"])

        # Post-traitement
        result = postprocess_prediction(prediction)

        # Calculer la latence
        end_time = datetime.now()
        latency_ms = (end_time - start_time).total_seconds() * 1000

        # Logger les détails de la prédiction
        log_data = {
            "request_id": request_id,
            "timestamp": end_time.isoformat(),
            "prediction": result["class"],
            "confidence": result["confidence"],
            "latency_ms": latency_ms,
            "image_size": len(contents)
        }
        logger.info(f"Prediction: {json.dumps(log_data)}")

        return JSONResponse(content={
            "success": True,
            "prediction": result,
            "latency_ms": latency_ms
        })
    
    except Exception as e:
        logger.error(f"Request {request_id}: Prediction failed - {str(e)}")
        raise HTTPException(
            status_code=500,
            detail=f"Prediction failed: {str(e)}"
        )

Key points:

• file: UploadFile = File(...): The three dots (...) mean that this parameter is required
• We immediately validate the file type with content_type
• The await keyword is necessary because file reading is asynchronous
• TFLite models use get_input_details(), set_tensor(), invoke(), get_tensor()

Starting the server

uvicorn app:app --reload

The --reload flag automatically restarts the server when you modify the code (useful for development).

# Installer les dépendances
pip install fastapi uvicorn python-multipart tensorflow numpy Pillow

FastAPI automatic documentation

FastAPI automatically generates interactive Swagger UI documentation accessible at /docs. From there, you can test the /predict endpoint directly from the browser by uploading an X-ray image.

2.3 Containerize your model service

We built our REST API and tested it locally. It works very well on our machine. The key question, however, remains: how to deploy this so that it works the same way everywhere?

Docker – Containerization

Docker allows us to package everything – our code, Python, all dependencies, and model files – into a single container that runs identically everywhere. It’s like a shipping container. No matter where you ship it, the contents inside work the same.

Dockerfile

# Utiliser l'image Python 3.11 slim comme base
# python:3.11-slim est une version légère sans les extras inutiles
FROM python:3.11-slim

# Définir le répertoire de travail dans le container
WORKDIR /app

# Copier les requirements en premier (pour le caching des layers)
COPY xray_model_package/requirements.txt .

# Installer les dépendances
# --no-cache-dir : évite de cacher les packages téléchargés → image plus petite
RUN pip install --no-cache-dir -r requirements.txt

# Copier le code de l'application
COPY app.py .
COPY xray_model_package/ ./xray_model_package/

# Exposer le port de l'API
# Documente que le container écoute sur le port 8000
# (ne publie pas réellement le port - ça se fait au moment du run)
EXPOSE 8000

# Commande qui s'exécute au démarrage du container
# --host 0.0.0.0 : accepter les connexions de l'extérieur du container
# (sans ça, l'API serait seulement accessible de l'intérieur)
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Why copy the requirements separately? Docker builds images in layers and caches each layer. If we copy the requirements first and they don’t change, Docker can reuse this cache layer. We only rebuild when the requirements really change. This makes rebuilds much faster.

.dockerignore

__pycache__/
*.pyc
*.pyo
*.pyd
.Python
*.so
*.egg
*.egg-info/
.pytest_cache/
.coverage
htmlcov/
dist/
build/
*.log
.DS_Store
.env
venv/
env/

This file tells Docker which files not to copy into the container. This makes the build faster and the image smaller.

Build and run the Docker container

# Construire l'image Docker
# -t : nom/tag de l'image
# . : utiliser le répertoire courant comme contexte de build
docker build -t xray-classifier-api .

# Exécuter le container
# -p 8000:8000 : mapper le port 8000 de la machine vers le port 8000 du container
docker run -p 8000:8000 xray-classifier-api

Same uvicorn startup messages appear. Open the browser at the same address as before, and the API works. You can test /health and /docs. Everything will work exactly as before, but now it runs inside a container.

This container will work the same on any machine that has Docker. Your colleague can pull this image and launch it, a production server can launch it, a cloud platform can launch it. To stop the container: Ctrl+C in the terminal.

2.4 Cloud deployment overview

We built our API and containerized it with Docker. It runs perfectly on our system, but for availability we need to deploy in the cloud where it is accessible 24/7.

The big three cloud providers

When to use what?

Fully managed serverless options (recommended to start): Google Cloud Run, Azure Container Apps, or AWS App Runner.

• Manage all network complexity
• Give URL automatically
• Pay only for actual usage

More network control (predictable traffic): AWS ECS with Fargate or managed container services.

• Requires more configuration

Kubernetes (EKS, AKS, GKE): Only if you have complex multi-service deployments. For a single template API like ours, this is excessive.

We use Google Cloud Run because it is the simplest and completely satisfies our current requirements.

2.5 Deploy to Google Cloud Run

Step 1: Create a Google Cloud account and project

1. Go to cloud.google.com and log in
2. Click on the project dropdown at the top → Create new project
3. Name it xray-classifier → Click Create

Step 2: Install the Google Cloud CLI

# Télécharger et installer depuis cloud.google.com/sdk
# Puis s'authentifier
gcloud auth login

Step 3: Find and configure your project ID

# Lister vos projets (le project ID inclut un suffixe numérique unique)
gcloud projects list
# Exemple : xray-classifier-481507

# Définir le projet comme défaut
gcloud config set project xray-classifier-481507

Step 4: Enable the necessary APIs

In the Google Cloud console:

1. Navigate to APIs & Services → Library
2. Find and enable Cloud Run Admin API
3. Find and enable Artifact Registry API (to store our Docker images)

Step 5: Create an Artifact Registry repository

In the console:

1. Search for Artifact Registry
2. Click Create Repository
3. Name: xray-models, Format: Docker, Location type: Region, Region: us-central1
4. Click Create

Step 6: Build the Image for Cloud Run

# Sur Apple Silicon Mac (architecture ARM) → construire pour AMD64 car Cloud Run utilise x86
docker build --platform linux/amd64 -t xray-classifier-api .

# Sur Windows ou Intel Mac
docker build -t xray-classifier-api .

Step 7: Authenticate Docker with Artifact Registry

gcloud auth configure-docker us-central1-docker.pkg.dev

Step 8: Tag and push image

# Tagger l'image (format spécifique Artifact Registry)
# region/project-id/repository-name/image-name
docker tag xray-classifier-api \
  us-central1-docker.pkg.dev/xray-classifier-481507/xray-models/xray-classifier-api

# Pousser vers Artifact Registry
docker push \
  us-central1-docker.pkg.dev/xray-classifier-481507/xray-models/xray-classifier-api

Step 9: Deploy to Cloud Run

gcloud run deploy xray-classifier-api \
  --image us-central1-docker.pkg.dev/xray-classifier-481507/xray-models/xray-classifier-api \
  --platform managed \
  --region us-central1 \
  --allow-unauthenticated \
  --port 8000

Explanation of flags:

• --platform managed: Google manages the entire infrastructure
• --region us-central1: where the service runs
• --allow-unauthenticated: anyone can access the API without authentication (required for public access)
• --port 8000: our container listens on 8000, not the default 8080

It takes 30-60 seconds. Cloud Run pulls your image, deploys it, and configures the infrastructure. When it’s finished, you get a URL. This is your live API!

Result

Our model is deployed. It can be accessed from anywhere with this URL. Cloud Run manages scaling automatically:

• 100 requests/second → it starts more instances
• 0 requests → it scales to 0 and costs nothing

You can see all metrics (queries, latencies, container instances, etc.) in the Cloud Run console.

3. – Monitor and maintain models in production

3.1 Why do models degrade over time?

Our X-ray classifier is deployed on Cloud Run, serving predictions to users. Everything works fine. But here’s something that surprises a lot of people deploying their first model: that 90% accuracy we achieved during training, it won’t last forever. Deployed models degrade over time.

The concept of drift

When we trained our model, we used a specific dataset: chest x-rays collected at a particular time, in certain hospitals, with specific equipment. Our model learned patterns from this data distribution – a snapshot of the world at that moment. The real world does not stand still.

There are two types of drift to understand:

Data drift Occurs when input data changes. Imagine that our X-ray classifier was trained on images from traditional X-ray machines. Two years later, hospitals are switching to new imaging technology that produces sharper images with different contrast. X-rays look different. Our model has never seen images like this during training. It could still make predictions, but those predictions might be less reliable.

Or consider a change in patient demographics. Maybe our training data came mostly from adults, but now the model is used extensively for pediatric cases.

Model drift This is when the relationship between input and output changes. For example, perhaps a new strain of pneumonia emerges that presents differently on X-rays than what the model has learned. The patterns learned by the model no longer correspond to reality.

Why it’s dangerous

What makes drifting dangerous is that it is silent. Unlike a bug that crashes your system, drifting does not generate errors. The API continues to run. It continues to return predictions with confidence scores, but these predictions may be increasingly unreliable.

This is not a theoretical problem. In real production systems, model degradation is normal and expected. Models that perform beautifully at deployment can become significantly less accurate within months or even weeks if the environment changes rapidly.

Monitoring is not optional. We need to detect when drift is occurring so we can re-train or update the model before accuracy drops too much.

3.2 Monitor performance and detect drift

We now know that models can degrade over time due to drift, but how do we actually detect when this happens? We need monitoring.

We need to log key metrics about the behavior of our model in production. Let’s update our FastAPI application to track three important things:

1. Prediction distribution (prediction distribution)
2. Confidence scores
3. Response latency

Logging configuration

import logging
from datetime import datetime
import json

# Configurer le logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)

logger = logging.getLogger(__name__)

logging.INFO: we capture informative messages and anything more severe (warnings, errors), but we ignore debug messages which would clutter the logs. We format the log entries with: timestamp, logger name, severity level, and the message itself.

Logging in the startup

@app.on_event("startup")
async def load_model():
    global interpreter
    try:
        interpreter = tf.lite.Interpreter(
            model_path="xray_model_package/model/xray_classifier_dynamic_quant.tflite"
        )
        interpreter.allocate_tensors()
        logger.info("Model loaded successfully!")
    except Exception as e:
        logger.error(f"Error loading the model: {e}")
        raise

When the model loads successfully, we log an info message. If the loading fails, we log an error with the details of the exception before rethrowing it. In production, these logs help diagnose issues like missing model files or memory issues.

Detailed logging in the prediction endpoint

@app.post("/predict")
async def predict(file: UploadFile = File(...)):
    # Générer un ID de requête unique basé sur le timestamp jusqu'aux microsecondes
    request_id = datetime.now().strftime("%Y%m%d%H%M%S%f")

    logger.info(f"Request {request_id}: received prediction request")

    if not file.content_type.startswith("image/"):
        logger.warning(f"Request {request_id}: Invalid file type {file.content_type}")
        raise HTTPException(status_code=400, detail="File must be an image")
    
    try:
        start_time = datetime.now()
        
        # ... (preprocessing et inférence)
        
        # Calculer la latence
        end_time = datetime.now()
        latency_ms = (end_time - start_time).total_seconds() * 1000

        # Logger toutes les métriques importantes en JSON structuré
        log_data = {
            "request_id": request_id,
            "timestamp": end_time.isoformat(),
            "prediction": result["class"],
            "confidence": result["confidence"],
            "latency_ms": latency_ms,
            "image_size": len(contents)
        }
        logger.info(f"Prediction: {json.dumps(log_data)}")
        
    except Exception as e:
        logger.error(f"Request {request_id}: Prediction failed - {str(e)}")

Why JSON for logs? When you analyze thousands of log entries later, having structured JSON data makes parsing and aggregation much easier.

The request_id ties all logs from the same request together, crucial when you process hundreds of requests and need to trace what happened with a specific request.

What to watch over time

Here are the three metrics to monitor with your logs:

1. Prediction distribution If your model normally predicts pneumonia 30% of the time (based on historical data), and suddenly it predicts pneumonia 80% of the time, that’s a major red flag. Either the patient population has changed dramatically (unlikely), or something is wrong with the model or input data.

2. Confidence scores Track average trust over time. If it was consistently around 0.85 and is now down to 0.65, that suggests the model is encountering data it is uncertain about. This is often a warning sign of drifting.

3. Latency If the response time starts to climb from 200 ms to 2 seconds, you might have infrastructure issues or a problem with model loading or inference.

In Cloud Run, these logs are automatically captured and sent to Google Cloud Logging. You can see them in the Cloud Run console under the Logs tab.

3.3 Model versioning and automated retraining

We have configured the monitoring to monitor the performance of the model, but what should we do when we detect drift, or if we have trained a new version of the model with better precision? How to update the model?

The problem without versioning

This is what often happens. You train a new model. It looks great on your test set, so you deploy it to production and replace the old one. Then, 2 hours later, you start getting complaints. The new model makes bizarre predictions about some borderline cases. You want to go back to the old version, but you already overwrote it, and now you’re stuck.

This is why model versioning is critical.

Versioning strategies

# Approche simple : inclure la version dans le nom de fichier
"xray_classifier_v1.tflite"
"xray_classifier_v2.tflite"

# Ou utiliser des timestamps
"xray_classifier_20240117_143022.tflite"

A/B testing with Cloud Run

A/B testing means running two versions of your model simultaneously in production. Some users (e.g. 10%) get predictions from the new version while others use the old version, allowing them to compare their performance before switching completely.

Cloud Run makes this very easy with traffic splitting:

# Étape 1 : Déployer la nouvelle version sans trafic
# --no-traffic : la nouvelle révision est déployée mais ne reçoit aucune requête
gcloud run deploy xray-classifier-api \
  --image us-central1-docker.pkg.dev/PROJECT/xray-models/xray-classifier-api:v2 \
  --no-traffic

# Étape 2 : Fractionner le trafic entre les versions
# 90% vers v1, 10% vers v2
gcloud run services update-traffic xray-classifier-api \
  --to-revisions xray-classifier-api-v1=90,xray-classifier-api-v2=10

# Si v2 performe mieux → augmenter progressivement
# 50/50, puis 80/20, puis 100%

# Si v2 performe moins bien → rollback immédiat
gcloud run services update-traffic xray-classifier-api \
  --to-revisions xray-classifier-api-v1=100

The thing to remember: Never delete old model versions until you are absolutely sure that the new one is stable. When something goes wrong in production, speed matters.

Automated retraining pipeline

Manually monitoring for drift and retraining models is tedious and error prone. We can automate this workflow:

1. Continuously monitor prediction logs, calculating metrics like average confidence score and distribution of predictions over sliding time windows.

2. Set thresholds that trigger retraining. For example, if the average confidence falls below 0.75, or if the pneumonia prediction ratio changes by more than 20%.

3. Automatically trigger a retraining job that retrieves new labeled data and trains a new version of the model.

4. Automated evaluation: If the new model does not meet the minimum thresholds, the pipeline stops and alerts the team.

5. If it passes, deploy with A/B testing. Start with 10% of traffic, monitor, and gradually increase if the metrics are good.

Drift detection script check_drift.py

import json
from datetime import datetime, timedelta

def check_drift_and_trigger_retraining():
    # Récupérer les logs des 24 dernières heures
    logs = fetch_prediction_logs(hours=24)

    # Calculer les métriques
    predictions = [json.loads(log)["prediction"] for log in logs]
    confidences = [json.loads(log)["confidence"] for log in logs]

    avg_confidence = sum(confidences) / len(confidences)
    pneumonia_ratio = predictions.count("Pneumonia") / len(predictions)

    # Valeurs de référence (baseline) basées sur les données historiques
    baseline_confidence = 0.85
    baseline_pneumonia_ratio = 0.30

    # Vérifier les seuils
    confidence_dropped = avg_confidence < 0.75
    distribution_shifted = abs(pneumonia_ratio - baseline_pneumonia_ratio) > 0.20

    if confidence_dropped or distribution_shifted:
        print(f"Drift detected! Avg confidence: {avg_confidence}, "
              f"Pneumonia ratio: {pneumonia_ratio}")
        trigger_retraining_pipeline()
    else:
        print("Metrics within normal range. No action needed")

This script would typically run on a schedule (like a cron job), not as part of the FastAPI API.

Explanation of the two metrics:

• avg_confidence < 0.75: signal that the model is uncertain about the data it sees
• Variation of pneumonia_ratio > 20% compared to baseline: suggests that the data distribution has changed

When one of the two thresholds is crossed, we call trigger_retraining_pipeline(). This feature would trigger your training job – perhaps by submitting to Vertex AI Training, triggering a GitHub Action, or calling an internal training service.

In production, you would use ML orchestration tools like Kubeflow Pipelines or Vertex AI Pipelines to manage the retraining, evaluation, and automated deployment workflow.

3.4 Balancing latency, scalability and cost

We have deployed our model, configured monitoring, and implemented versioning. But there is one more critical topic to address: the operational compromises you will encounter in production. Every deployment decision involves balancing three things: latency, scalability, and cost.

Latency

Latency asks the question: How ​​quickly does your model respond to queries? For our X-ray classifier, if a doctor waits 2 seconds for a result, it is acceptable. If it takes 2 minutes, the system becomes unusable.

Latency comes from:

• Network time
• Preprocessing
• Model inference
• Possible queues

Remember when we quantized the model from 223 MB to 57 MB? It wasn’t just about the size. Quantized models run faster because integer operations are faster than floating point math. On a device without a GPU, this could reduce inference time from 5 seconds to 2 seconds.

But there are tradeoffs: more aggressive optimization (further quantization or pruning) can reduce latency further, but could hurt accuracy.

Scalability

Scalability concerns the management of a variable load. With Cloud Run we get auto scaling. That’s the beauty of serverless: it scales on demand without manual configuration.

The key question is: what is your traffic pattern?

• Stable and predictable traffic → may work better with fixed infrastructure
• Variable and peaked traffic → benefits from serverless autoscaling

Cost

Cloud Run charges based on query time and memory usage. When you are inactive, you pay nothing. When you process, you pay for computing time. But when traffic increases, so do costs. At millions of requests per day, serverless can become expensive compared to dedicated servers with a fixed monthly cost.

Model size also affects cost. This is why optimization is important.

How to balance these tradeoffs

Cloud providers have cost analysis tools: use them.

4. Appendix: – Complete Code Files

export_model.py – Export model to ONNX and TFLite

import tensorflow as tf
import tf2onnx
import onnx
import numpy as np

# Charger notre classificateur X-ray pré-entraîné
model = tf.keras.models.load_model("xray_resnet152_model")

print("Model loaded successfully")

# Créer un input de test - 1 batch, 224x224 pixels, 3 channels
test_input = np.random.rand(1, 224, 224, 3).astype(np.float32)

prediction = model.predict(test_input)
print("Inference successful. Prediction shape:", prediction.shape)
print("SavedModel prediction:", prediction)

# --- Export ONNX ---
spec = (tf.TensorSpec((None, 224, 224, 3), tf.float32, name="input"),)

model_proto, _ = tf2onnx.convert.from_keras(
    model,
    input_signature=spec,
    opset=13,
    output_path="xray_classifier.onnx"
)

print("Model converted to ONNX format successfully")

import onnxruntime

session = onnxruntime.InferenceSession("xray_classifier.onnx")

input_name = session.get_inputs()[0].name

test_input_onnx = np.random.randn(1, 224, 224, 3).astype(np.float32)
outputs = session.run(None, {input_name: test_input_onnx})

print(f"ONNX prediction shape: {outputs[0].shape}")
print(f"ONNX prediction: {outputs[0]}")

# --- Export TFLite ---
converter = tf.lite.TFLiteConverter.from_saved_model("xray_resnet152_model")
tflite_model = converter.convert()

with open("xray_classifier.tflite", "wb") as f:
    f.write(tflite_model)
print("Model exported to TensorFlow Lite format")

interpreter = tf.lite.Interpreter(model_path="xray_classifier.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

test_input_tflite = np.random.randn(1, 224, 224, 3).astype(np.float32)
interpreter.set_tensor(input_details[0]["index"], test_input_tflite)
interpreter.invoke()

output_data = interpreter.get_tensor(output_details[0]["index"])

print(f"TFLite prediction shape: {output_data.shape}")
print(f"TFLite prediction: {output_data}")

quantize_model.py – Quantize model

import tensorflow as tf
import os

# Vérifier la taille originale
original_size = os.path.getsize("xray_classifier.tflite") / (1024 * 1024)
print(f"Original SavedModel size: {original_size:.2f} MB")

# Quantification dynamic range
converter = tf.lite.TFLiteConverter.from_saved_model("xray_resnet152_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_quant_model = converter.convert()

with open("xray_classifier_dynamic_quant.tflite", "wb") as f:
    f.write(tflite_quant_model)

quantized_size = len(tflite_quant_model) / (1024 * 1024)
print(f"Dynamic Range Quantized model size: {quantized_size:.2f} MB")

xray_model_package/model/preprocessing.py – Image preprocessing

import numpy as np
from PIL import Image

def preprocess_xray(image_path, target_size=(224, 224)):
    """
    Prétraiter l'image X-ray pour l'inférence du modèle.
    Correspond au prétraitement ImageDataGenerator de l'entraînement :
    - Redimensionner à 224x224
    - Convertir en RGB (3 channels)
    - Rescaler à [0, 1]
    """
    img = Image.open(image_path)
    img = img.convert('RGB')
    img = img.resize(target_size)
    img_array = np.array(img, dtype=np.float32)
    img_array = img_array / 255.0
    img_array = np.expand_dims(img_array, axis=0)
    return img_array

def postprocess_prediction(prediction):
    """
    Convertir la sortie du modèle en résultat lisible.
    Modèle avec activation sigmoid pour classification binaire.
    """
    probability = float(prediction[0][0])
    
    if probability > 0.5:
        class_name = "Pneumonia"
        confidence = probability
    else:
        class_name = "Normal"
        confidence = 1 - probability
    
    return {
        "class": class_name,
        "confidence": round(confidence, 4)
    }

app.py – Full FastAPI API with logging

from fastapi import FastAPI, File, UploadFile, HTTPException
from fastapi.responses import JSONResponse
import tensorflow as tf
import numpy as np
from PIL import Image
import io
from xray_model_package.model.preprocessing import preprocess_xray, postprocess_prediction
import logging
from datetime import datetime
import json

# Configuration du logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)

logger = logging.getLogger(__name__)

app = FastAPI(
    title="X-Ray Pneumonia Classifier API",
    description="API for classifying chest X-rays as Normal or Pneumonia",
    version="1.0.0"
)

model = None
interpreter = None

@app.on_event("startup")
async def load_model():
    global interpreter
    try:
        interpreter = tf.lite.Interpreter(
            model_path="xray_model_package/model/xray_classifier_dynamic_quant.tflite"
        )
        interpreter.allocate_tensors()
        logger.info("Model loaded successfully!")
    except Exception as e:
        logger.error(f"Error loading the model: {e}")
        raise

@app.get("/")
async def root():
    return {
        "message": "X-Ray Pneumonia Classifier API",
        "status": "running",
        "version": "1.0.0"
    }

@app.get("/health")
async def health_check():
    return {
        "status": "healthy",
        "model_loaded": interpreter is not None
    }

@app.post("/predict")
async def predict(file: UploadFile = File(...)):
    request_id = datetime.now().strftime("%Y%m%d%H%M%S%f")

    logger.info(f"Request {request_id}: received prediction request")

    if not file.content_type.startswith("image/"):
        logger.warning(f"Request {request_id}: Invalid file type {file.content_type}")
        raise HTTPException(
            status_code=400,
            detail="File must be an image"
        )
    
    try:
        start_time = datetime.now()

        contents = await file.read()
        image = Image.open(io.BytesIO(contents))

        temp_path = "temp_image.jpg"
        image.save(temp_path)

        preprocessed = preprocess_xray(temp_path)

        input_details = interpreter.get_input_details()
        output_details = interpreter.get_output_details()

        interpreter.set_tensor(input_details[0]["index"], preprocessed)
        interpreter.invoke()
        prediction = interpreter.get_tensor(output_details[0]["index"])

        result = postprocess_prediction(prediction)

        end_time = datetime.now()
        latency_ms = (end_time - start_time).total_seconds() * 1000

        log_data = {
            "request_id": request_id,
            "timestamp": end_time.isoformat(),
            "prediction": result["class"],
            "confidence": result["confidence"],
            "latency_ms": latency_ms,
            "image_size": len(contents)
        }
        logger.info(f"Prediction: {json.dumps(log_data)}")

        return JSONResponse(content={
            "success": True,
            "prediction": result,
            "latency_ms": latency_ms
        })
    
    except Exception as e:
        logger.error(f"Request {request_id}: Prediction failed - {str(e)}")
        raise HTTPException(
            status_code=500,
            detail=f"Prediction failed: {str(e)}"
        )

Dockerfile – Containerization

# Image de base Python 3.11 légère
FROM python:3.11-slim

# Répertoire de travail dans le container
WORKDIR /app

# Copier les requirements en premier (optimisation du cache Docker)
COPY xray_model_package/requirements.txt .

# Installer les dépendances (sans cache pour réduire la taille)
RUN pip install --no-cache-dir -r requirements.txt

# Copier le code de l'application
COPY app.py .
COPY xray_model_package/ ./xray_model_package/

# Documenter le port d'écoute
EXPOSE 8000

# Commande de démarrage (--host 0.0.0.0 pour accepter les connexions externes)
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

.dockerignore – Docker exclusions

__pycache__/
*.pyc
*.pyo
*.pyd
.Python
*.so
*.egg
*.egg-info/
.pytest_cache/
.coverage
htmlcov/
dist/
build/
*.log
.DS_Store
.env
venv/
env/

xray_model_package/config.json – Model package configuration

{
    "model_name": "xray_pneumonia_classifier",
    "model_version": "1.0.0",
    "model_architecture": "ResNet152V2",
    "model_type": "tflite_quantized",
    "input_shape": [1, 224, 224, 3],
    "output_shape": [1, 1],
    "classes": ["Normal", "Pneumonia"],

    "preprocessing": {
        "resize": [224, 224],
        "color_mode": "rgb",
        "rescale": "divide_by_255",
        "notes": "Corresponds to ImageDataGenerator with rescale=1./255"
    },

    "training_info": {
        "framework": "tensorflow",
        "base_model": "ResNet152V2",
        "transfer_learning": true,
        "test_accuracy": 0.88
    },

    "optimization": {
        "quantization": "dynamic_range",
        "original_size_mb": 223,
        "quantized_size_mb": 57,
        "size_reduction_percent": 74.4
    }
}

requirements.txt – Dependencies

tensorflow==2.15.0
numpy==1.24.3
Pillow==10.0.0
fastapi==0.104.1
uvicorn==0.24.0
python-multipart==0.0.6

monitoring/check_drift.py – Automated drift detection

import json
from datetime import datetime, timedelta

def check_drift_and_trigger_retraining():
    # Récupérer les logs de prédiction des 24 dernières heures
    logs = fetch_prediction_logs(hours=24)

    # Extraire les prédictions et scores de confiance
    predictions = [json.loads(log)["prediction"] for log in logs]
    confidences = [json.loads(log)["confidence"] for log in logs]

    # Calculer les métriques
    avg_confidence = sum(confidences) / len(confidences)
    pneumonia_ratio = predictions.count("Pneumonia") / len(predictions)

    # Valeurs de référence (baseline) historiques
    baseline_confidence = 0.85
    baseline_pneumonia_ratio = 0.30

    # Vérifier les seuils de drift
    confidence_dropped = avg_confidence < 0.75
    distribution_shifted = abs(pneumonia_ratio - baseline_pneumonia_ratio) > 0.20

    if confidence_dropped or distribution_shifted:
        print(f"Drift detected! Avg confidence: {avg_confidence}, "
              f"Pneumonia ratio: {pneumonia_ratio}")
        trigger_retraining_pipeline()
    else:
        print("Metrics within normal range. No action needed")

This script typically runs as a cron job, not as part of the FastAPI API.

5. Full deployment flow summary

Modèle entraîné (TensorFlow SavedModel)
    ↓
Export multi-format
    ├── ONNX (cross-platform)
    └── TFLite (mobile/edge)
    ↓
Quantification (223 MB → 57 MB, -75%)
    ↓
Package du modèle
    ├── model/xray_classifier_dynamic_quant.tflite
    ├── model/preprocessing.py
    ├── config.json
    └── requirements.txt
    ↓
REST API avec FastAPI (app.py)
    ├── /health → vérification de santé
    ├── /predict → prédiction avec logging structuré
    └── /docs → documentation Swagger auto-générée
    ↓
Conteneurisation Docker
    ├── Dockerfile
    └── .dockerignore
    ↓
Déploiement sur Google Cloud Run
    ├── Build → Artifact Registry
    ├── Deploy → URL publique automatique
    └── Auto-scaling (scale to 0 si inactif)
    ↓
Monitoring en production
    ├── Prediction distribution
    ├── Confidence scores
    ├── Response latency
    └── Drift detection → ré-entraînement automatisé

Search Terms

deep · model · deployment · neural · networks · machine · data · science · cloud · run · api · docker · drift · logging · models · package · automated · configuration · deploy · fastapi · formats · google · image · quantization