Deploying Open-source LLMs

Table of Contents

1. Module 1 — Select and Configure Deployment Strategies

   • Introduction
   • Aligning Deployment to Constraints
   • Serving Frameworks: vLLM, TGI, Ollama
   • Demo 1 — Local Deployment with Ollama
   • Demo 2 — Deployment on AWS with vLLM
   • Containerization and Orchestration

2. Module 2 — Configure the Technical Environment

   • Hardware Requirements
   • Demo 1 — Resource Usage by Model Size (Local PC)
   • Demo 2 — Resource Usage on AWS Cloud (GPU)
   • Model Quantization
   • Demo 3 — Applying Quantization to Reduce Resource Usage
   • Infrastructure Right-sizing
   • Demo 4 — Adjusting Configuration Parameters

3. Module 3 — Optimize and Monitor for Production

   • Optimization Techniques
   • Demo 1 — Latency and Throughput Benchmarking
   • Backup, Recovery, and Rollback
   • Demo 2 — Rollback to a Previous Model Version
   • Security Best Practices
   • Demo 3 — Adding API Key Authentication

Module 1 — Select and Configure Deployment Strategies

Introduction

Everyone talks about prompting, but nobody talks about the engineering that actually makes the model respond. This course dives into the world where LLMs are deployed, optimized, scaled, and made real.

The key engineering questions this course answers:

┌─────────────────────────────────────────────────────────────────┐
│  Critical engineering questions                                  │
├─────────────────────────────────────────────────────────────────┤
│  • How to serve massive models with low latency?                 │
│  • How to secure sensitive data in the cloud?                    │
│  • How to control costs when GPUs are expensive?                 │
└─────────────────────────────────────────────────────────────────┘

Course roadmap:

flowchart LR
    A[Trade-offs\nLatency / Privacy / Budget] --> B[Serving Frameworks\nvLLM · TGI · Ollama]
    B --> C[Containerization\n& Orchestration]
    C --> D[Infrastructure\n& Quantization]
    D --> E[Optimization\nMonitoring\nSecurity]

Aligning Deployment to Constraints

Before deploying an open-source LLM, the most important decision is where and how to deploy it. The ideal strategy depends on three key organizational constraints.

mindmap
  root((Deployment Constraints))
    Latency
      Response time
      Sub-second for real-time apps
      GPUs close to users
      CPU for batch processing
    Privacy
      GDPR / HIPAA
      On-premise or private cloud
      BYOK configurations
      Ollama for local hosting
    Budget
      GPUs are expensive
      Quantization to reduce
      Small models
      Hybrid strategy

1. Latency

Latency is the time between sending the prompt and receiving the response.

💡 Analogy: Latency is like the delivery time of a meal. If it takes forever, the customer loses their appetite.

2. Data Privacy

Organizations in healthcare, banking, and government often cannot send sensitive data to public clouds (GDPR, HIPAA).

💡 Analogy: Data privacy is like your secret recipe — you can’t share it with just any cloud provider.

3. Budget

GPUs are powerful but expensive. For example, running a 13 billion parameter model on a single A100 GPU can cost several dollars per hour.

Strategies to control costs:

• Select small models
• Apply quantization
• Run inference on CPU (with a speed trade-off)
• Adopt a hybrid strategy

💡 Token budget-aware reasoning: Recent research shows that LLMs often generate unnecessarily long chains of thought. By guiding the model with a token budget or dynamically adjusting this budget based on task complexity, you can reduce costs with minimal impact on accuracy.

Deployment Patterns

graph TD
    A{Which priority\nconstraint?} --> B[Privacy + Low Cost]
    A --> C[Scalability + Speed]
    A --> D[Both]

    B --> E[🖥️ Local Deployment\nOllama\nMaximum privacy, minimal cost\nIdeal for internal apps / prototypes]
    C --> F[☁️ Single-cloud GPU\nvLLM or TGI on AWS/GCP/Azure\nBalance scalability + speed\nFor client-facing apps]
    D --> G[🔀 Hybrid Deployment\nLocal control + cloud elasticity\nFor organizations needing\nboth]

Serving Frameworks: vLLM, TGI, Ollama

Serving frameworks are specialized runtimes that load LLMs, manage GPU memory, and expose APIs for inference. They eliminate the need to write low-level server code.

💡 Analogy: Serving frameworks are like different types of vehicles. One is a sports car, another an SUV, another a scooter. They all move, but you wouldn’t take a scooter on a road trip.

graph LR
    A[Choosing the Right\nServing Engine]
    A --> B[vLLM\nThroughput\n🏎️ Formula 1]
    A --> C[TGI\nFlexibility\n🚙 SUV]
    A --> D[Ollama\nSimplicity\n🛵 Scooter]

vLLM — The Speed Machine

Profile: High performance, production-grade inference.

When to use vLLM:

• Serving thousands of chat sessions
• Powering an AI assistant in a large enterprise application
• When concurrency and cost per token are critical

TGI — Text Generation Inference

Profile: Hugging Face production server, versatile.

When to use TGI:

• Experimenting with multiple different models
• Mix of architectures (e.g., summarizer + code generator)
• Native integration into the Hugging Face ecosystem

Ollama — Local and Edge Deployment

Profile: Designed for developers who want to download and run a model directly on a laptop or workstation.

Basic Ollama commands:

# Install and run a model
ollama run llama3

# Run a specific model
ollama run mistral

# Run a model with parameters
ollama run gpt-oss:20b

# List installed models
ollama list

# Stop a model
ollama stop llama3

Framework Comparison

quadrantChart
    title Serving Frameworks — Positioning
    x-axis Low Simplicity --> High Simplicity
    y-axis Low Performance --> High Performance
    quadrant-1 Large-scale production
    quadrant-2 Balance
    quadrant-3 Restricted use
    quadrant-4 Local use
    vLLM: [0.15, 0.95]
    TGI: [0.40, 0.75]
    Ollama: [0.90, 0.40]

Demo 1 — Local Deployment with Ollama

Deployment Steps

1. Download Ollama
   └─ https://ollama.com
   └─ Available installers: Windows / macOS / Linux

2. Explore the Ollama application (GUI)
   ├─ Model library (local and cloud)
   ├─ Thinking mode: High / Medium / Low
   ├─ File upload for summary/reference
   └─ Clean interface for prompts and responses

3. Browse available models
   ├─ Sort by "Popular" or "Newest"
   ├─ See: number of parameters, context length, quantization type
   └─ Example: GPT-OSS 20B — MXFP4 quantization (4-bit mixed precision)

4. Install a model via CLI (optional)
   └─ Copy the command from the model page

5. Run the model in the Ollama app
   ├─ Model appears as "Downloaded"
   ├─ Select the model and type the prompt
   └─ Ollama uses local CPU/GPU for inference

Reference CLI commands:

# Install and run GPT-OSS 20B
ollama run gpt-oss:20b

# Test Phi-3 Mini (3.8B - lightweight)
ollama run phi3

# Test Mistral 7B
ollama run mistral

# Test LLaMA 3
ollama run llama3

Tip: Open Task Manager / Activity Monitor to observe CPU/GPU usage while the model is running.

Demo 2 — Deployment on AWS with vLLM

Deployment Architecture

┌─────────────────────────────────────────────────────────┐
│                    AWS EC2 (GPU)                        │
│                                                         │
│  ┌─────────────┐    ┌────────────────────────────────┐  │
│  │  NVIDIA GPU │    │  Python Virtual Environment    │  │
│  │  (A10G)     │◄───│  vllm-env                      │  │
│  └─────────────┘    │                                │  │
│                     │  pip install vllm              │  │
│                     │           transformers         │  │
│                     │           accelerate           │  │
│                     └────────────────────────────────┘  │
│                              │                          │
│                     ┌────────▼───────────────────────┐  │
│                     │  vLLM API Server               │  │
│                     │  Port 8000                     │  │
│                     │  Mistral-7B-Instruct            │  │
│                     └────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘
         │ Port 8000 (Security Group open)
         ▼
   External client / curl / Postman

Complete Step-by-Step Guide

Step 1 — Launch an EC2 instance with GPU

EC2 Console → Launch Instance
├─ AMI: Deep Learning OSS NVIDIA Driver AMI PyTorch 2.8 (Ubuntu 24.04)
├─ Instance type: g5.xlarge (or g5.2xlarge, A10G)
├─ Key pair: create or use an existing one
├─ Security Group: open SSH (22) + port 8000
└─ Storage: 80 GB minimum

Step 2 — Connect to the instance

# Via SSH from your local machine
ssh ubuntu@<EC2-PUBLIC-IP>

# Or via AWS browser-based client

Step 3 — Verify GPU availability

nvidia-smi

Expected output: NVIDIA GPU details, driver version, memory status.

Step 4 — Install basic tools

sudo apt-get update
sudo apt-get install -y python3-pip python3-venv git

Step 5 — Create a Python virtual environment

cd /home/ubuntu
python3 -m venv vllm-env
source vllm-env/bin/activate
# The (vllm-env) prefix appears in the terminal

Step 6 — Install vLLM and its dependencies

pip install --upgrade pip
pip install vllm transformers accelerate

Step 7 — Start the vLLM server

python3 -m vllm.entrypoints.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --host 0.0.0.0 \
  --port 8000

Note: vLLM will download model weights on first run. The server blocks this terminal — leave it running.

Step 8 — Open port 8000 in AWS

EC2 → Security Groups → Inbound Rules
├─ Type: Custom TCP
├─ Port: 8000
└─ Source: Your IP (recommended) or 0.0.0.0/0 (demo only)

Step 9 — Test the endpoint

# Open a 2nd SSH session
source vllm-env/bin/activate

curl http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Explain what vLLM does in one sentence.",
    "max_tokens": 64
  }'

Expected response: generated text in JSON format.

Step 10 — Monitor GPU usage

nvidia-smi
# See: GPU memory used by Python → model loaded
# See: vLLM process running → processing requests

Step 11 — Shutdown and cleanup (save costs!)

# Stop the vLLM server
Ctrl + C

# Deactivate the environment
deactivate

# Stop or terminate the EC2 instance from the AWS console

Containerization and Orchestration

LLM Deployment Layers

graph TD
    A["🖥️ Application Layer\nChatbots · Copilots · Enterprise Assistants\nSend requests → Get responses in ms"]
    B["⚙️ Serving Engine Layer\nvLLM · TGI · Ollama · TorchServe · Ray Serve\nBatching · Caching · Latency · API"]
    C["🧠 Deep Learning Framework Layer\nPyTorch · TensorFlow · JAX\nLoad model · Computation graph · CUDA math"]
    D["🔌 GPU Runtime / Driver Layer\nCUDA · cuDNN · TensorRT\nBridge software ↔ hardware"]
    E["💾 GPU Hardware Layer\nNVIDIA A100 · H100 · B200\nBillions of parallel operations"]

    A --> B --> C --> D --> E

Containerization

Containerization means packaging the application — model weights, serving code, libraries, and environment parameters — into a single, self-contained image.

┌─────────────────────────────────────────────────────────┐
│                    Docker Image                         │
│                                                         │
│   ┌─────────────┐  ┌──────────────┐  ┌──────────────┐  │
│   │Model weights│  │Serving code  │  │  Libraries   │  │
│   │(e.g. LLaMA) │  │(e.g. vLLM)   │  │(CUDA, PyTorch│  │
│   └─────────────┘  └──────────────┘  └──────────────┘  │
│                                                         │
│   ┌──────────────────────────────────────────────────┐  │
│   │     Environment settings & configurations        │  │
│   └──────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

Docker advantages:

Example Dockerfile for vLLM:

FROM nvidia/cuda:12.1.0-devel-ubuntu22.04

# System dependencies
RUN apt-get update && apt-get install -y \
    python3-pip python3-venv git && \
    rm -rf /var/lib/apt/lists/*

# Create and activate Python environment
RUN python3 -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"

# Install vLLM and dependencies
RUN pip install --upgrade pip && \
    pip install vllm transformers accelerate

# Exposed port
EXPOSE 8000

# Startup command
CMD ["python3", "-m", "vllm.entrypoints.api_server", \
     "--model", "mistralai/Mistral-7B-Instruct-v0.2", \
     "--host", "0.0.0.0", \
     "--port", "8000"]

Orchestration with Kubernetes

Orchestration manages multiple containers, their scaling, rolling updates, and automatic recovery.

graph TD
    A[Kubernetes Cluster]
    A --> B[Node 1 GPU]
    A --> C[Node 2 GPU]
    A --> D[Node 3 GPU]

    B --> B1[Pod: vLLM\nMistral-7B]
    C --> C1[Pod: vLLM\nMistral-7B]
    D --> D1[Pod: vLLM\nMistral-7B]

    E[Load Balancer] --> B1
    E --> C1
    E --> D1

    F[Clients / API] --> E

Kubernetes responsibilities:

• Container scheduling
• Rolling updates (updates without interruption)
• Pod auto-recovery (automatic restart of failing pods)
• Traffic balancing

Example Kubernetes manifest for vLLM:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-deployment
  labels:
    app: vllm
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - name: vllm
        image: your-registry/vllm-mistral:v1.0
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
        env:
        - name: MODEL_NAME
          value: "mistralai/Mistral-7B-Instruct-v0.2"
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
spec:
  selector:
    app: vllm
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8000
  type: LoadBalancer

Expected results:

┌─────────────────────────────────────────────────────────┐
│  Containerization + Orchestration = Production-grade    │
│                                                         │
│  ✅ Effortless scalability                              │
│  ✅ Fast recovery                                       │
│  ✅ Predictable uptime                                  │
│  ✅ Consistent deployments                              │
└─────────────────────────────────────────────────────────┘

Module 2 — Configure the Technical Environment

Hardware Requirements

Hardware defines the performance ceiling, cost curve, and operational risk.

Deploying a massive LLM on an underpowered GPU is like trying to tow a trailer with a bicycle — technically possible for a few meters, but you’ll regret it quickly.

Model Sizes and VRAM Requirements

graph LR
    A["Phi-3 Mini\n3.8B params"] -->|~6 GB VRAM| G1["RTX 3060\nLaptop GPU"]
    B["LLaMA 2 / Mistral\n7B params"] -->|12-16 GB VRAM| G2["RTX 3090\nNVIDIA T4"]
    C["13B models\n(e.g. LLaMA 2 13B)"] -->|24-32 GB VRAM| G3["A100 40GB\nNVIDIA L4"]
    D["LLaMA 2 70B\nMixtral 8x7B"] -->|≥ 80 GB VRAM| G4["Multi-GPU\nTensor Parallelism"]
    E["DeepSeek V3\nGLaM (1.2T)"] -->|Multi-GPU clusters| G5["A100/H100/B200\nClusters"]

Reference Table: Model → Required GPU

The Role of KV Cache

The KV Cache (Key-Value Cache) is the working memory during inference. It stores intermediate attention states to avoid recalculating them.

┌─────────────────────────────────────────────────────────────────┐
│  VRAM consumption = Model weights + KV Cache                    │
│                                                                 │
│  KV Cache grows linearly with:                                  │
│  ├─ Batch size (more concurrent requests)                       │
│  ├─ Context length (long conversations)                         │
│  └─ Length of generated outputs                                 │
│                                                                 │
│  ⚠️ Most common cause of OOM (Out of Memory) errors             │
└─────────────────────────────────────────────────────────────────┘

How vLLM solves the KV Cache problem:

graph TD
    A[Problem: GPU memory\nfragmentation with KV Cache] --> B[Solution: PagedAttention\nby vLLM]
    B --> C[KV cache storage\nin non-contiguous pages]
    B --> D[Reduced memory\nfragmentation]
    B --> E[Dynamic batching\nhundreds of concurrent requests]
    C --> F[More concurrent\nusers supported]
    D --> F
    E --> F

Aligning Hardware to Deployment Needs

flowchart TD
    A{Main constraint?} --> B[Low latency\ninteractive team]
    A --> C[Cost control\nprototyping]
    A --> D[Privacy\non-premise]

    B --> E["High VRAM GPU\nA100 / H100 / B200\nMulti-GPU if > 30B"]
    C --> F["Small quantized models\n4-bit on T4 or RTX\nCPU acceptable for < 7B"]
    D --> G["On-premise setup\nLocal GPU or private cluster\nOllama or vLLM internally"]

Demo 1 — Resource Usage by Model Size (Local PC)

Environment: CPU-only desktop (Intel UHD 630 integrated), 64 GB RAM, Windows

Observations Before Launching a Model

CPU  : ~20–30%
RAM  : ~10–15 GB used (out of 64 GB)
GPU  : ~70–80% (screen recording overhead)

Test 1: Phi-3 Mini (3.8B)

# In Ollama
ollama run phi3
# Prompt: "Give a brief summary of deploying LLM"

Results:
CPU     : ⬆ spike to 100%
RAM     : ⬆ > 95%
GPU     : ➡ unchanged (70–80%)
Latency : Acceptable for light workloads

Insight: Even small models can quickly saturate CPU cores, but remain usable for light loads.

Test 2: Mistral 7B

ollama run mistral
# Prompt: response in 2 sentences only

Results:
CPU     : ⬆ 100%
RAM     : ~30% (stabilized — short prompt = less pressure)
GPU     : ➡ unchanged
Latency : Notably slower

Insight: Model size matters — but prompt length and output size matter just as much.

Test 3: LLaMA 3 (quantized)

ollama run llama3

Results:
CPU/RAM : Brief spike, then stabilization
GPU     : ➡ unchanged
RAM     : ~30%

Key insight: Despite a larger model, shorter responses lead to lower sustained resource usage.

Summary Table — CPU-only

Conclusions:

• CPU-only inference scales poorly with model size
• Short prompts = less resource pressure
• Model right-sizing is critical for cost efficiency
• CPU environments are viable — but limited

Demo 2 — Resource Usage on AWS Cloud (GPU)

Environment: AWS EC2 g5.2xlarge — NVIDIA A10G GPU (24 GB VRAM), Windows Server 2025

GPU Monitoring Setup

# Check available GPU
nvidia-smi

# Expected result:
# GPU 0: NVIDIA A10G - 22,400 MiB total - 0% utilization

Required configuration: Enable GPU performance counters in NVIDIA Control Panel → Manage GPU Performance Counters.

Test 1: Gemma 3 12B

ollama run gemma3:12b
# Prompt: test question

Results:
Disk         : ⬆ spike (model download)
GPU Utilization: ⬆ ~90% during inference
GPU after    : ⬇ back to 0% after completion
GPU Memory   : Remains allocated while Ollama runs

After stopping Ollama: GPU memory is completely freed.

Test 2: Gemma 3 27B

ollama run gemma3:27b

Results:
GPU Utilization: ⬆ 90–95%
Similar behavior to 12B but more intensive

CPU vs GPU Comparison

graph LR
    A["CPU-only\nPhi-3 Mini 3.8B"] -->|Latency| L1["🐌 Slow\nCPU 100%\nRAM 95%"]
    B["GPU A10G\nGemma 3 12B"] -->|Latency| L2["⚡ Fast\nGPU 90%\nClean release after"]

GPU vs CPU conclusions:

Model Quantization

The Problem

A 13B parameter model in FP16 precision (each parameter = 16 bits) can easily consume 28 GB of VRAM. Most GPUs have limited VRAM.

The Solution: Quantization

Quantization reduces the numerical precision used to represent model weights and activations.

FP32 (32 bits) → INT8 (8 bits) → INT4 (4 bits)
    ↓                  ↓               ↓
  High precision   Compromise      Very compact
  Very accurate    Good balance   Lightweight, fast

Key Benefits

graph TD
    Q[Quantization] --> A[Reduced\nmemory footprint]
    Q --> B[Faster\ninference]
    Q --> C[Reduced\ncosts]

    A --> A1["7B FP16: 14 GB VRAM\n→ INT4: ~4 GB VRAM\nDeployable on mid-tier GPU or CPU"]
    B --> B1["Smaller representations\n= Faster matrix multiplications\n= More tokens/sec"]
    C --> C1["Works on cheaper hardware\nAvoids expensive high-end GPUs"]

Reference Table: Precision → VRAM

Quantization Trade-offs

┌──────────────────────────────────────────────────────────────┐
│  Advantages               │  Disadvantages                   │
├──────────────────────────────────────────────────────────────┤
│  ✅ Less VRAM              │  ⚠️ Loss of fine information      │
│  ✅ Faster inference       │  ⚠️ Slight accuracy drop          │
│  ✅ Reduced costs          │  ⚠️ Impact on output quality      │
│  ✅ Cheaper hardware       │  ⚠️ Not free in terms of perf     │
└──────────────────────────────────────────────────────────────┘

💡 Analogy: Quantization is like vacuum-compressing your clothes for luggage. Without compression: the suitcase doesn’t close. With it: everything fits neatly and travels easily.

Quantization Techniques

graph TD
    QT[Quantization\nTechniques]
    QT --> PTQ[Post-Training\nQuantization - PTQ]
    QT --> QAT[Quantization-Aware\nTraining - QAT]
    QT --> DQ[Dynamic\nQuantization]
    QT --> SQ[Static\nQuantization]

    PTQ --> PTQ1["Applied after full training\nNo retraining required\nbitsandbytes → easy 4-bit conversion\nRun large models on consumer GPUs"]
    QAT --> QAT1["Trains under quantized conditions\nSimulates low precision during training\nBetter accuracy than PTQ\nMore compute and training time"]
    DQ --> DQ1["Reduces precision during inference\nConverts weights at runtime (FP32 → INT8)\nVery flexible and easy to apply\nNo retraining or calibration required"]
    SQ --> SQ1["Pre-calculated structured ranges\nFaster but requires calibration data\nTypical LLM use cases covered by PTQ"]

Best practices:

"Quantize first" — the biggest gain per line of code
├─ Enables models to run on limited, economical hardware
├─ Reduces memory usage with minimal accuracy loss
└─ Recommended libraries: bitsandbytes, LLM.int8(), GPTQ

Demo 3 — Applying Quantization to Reduce Resource Usage

Environment: AWS EC2 with NVIDIA A10G, LM Studio, nvidia-smi monitoring every 2 seconds

Configuration of Tested Models

Both models are architecturally identical — only the precision differs.

Q4 Results

# In LM Studio — load LLaMA 3.1 8B Q4
# Identical prompt for both tests

Tokens/sec          : 82.3
Total tokens        : 953
Time to first token : 0.02s
GPU utilization     : ~90%
VRAM used           : ~5 GB

Q8 Results

Tokens/sec          : 54.48
Total tokens        : 762
Time to first token : 0.03s
GPU utilization     : ~90%
VRAM used           : ~8 GB

Visual Comparison Q4 vs Q8

                Q4              Q8
Speed       : ████████ 82.3   █████ 54.5  tokens/sec
Memory      : ████░░░░ 5 GB   ████████ 8 GB
Time to 1st : 0.02s           0.03s

Conclusions:

• Lower precision = faster inference
• Reduced memory footprint = better throughput
• Quantization enables economical scaling
• Quality remains high for most use cases

Key insight: Quantization doesn’t just make models fit in memory — it makes them viable.

Infrastructure Right-sizing

Right-sizing means choosing the right balance between CPU and GPU, single-GPU and multi-GPU setups, and configuring serving parameters to deliver acceptable performance without overspending.

CPU or GPU?

graph LR
    A{What use case?}

    A --> CPU[CPU]
    A --> GPU[GPU]

    CPU --> C1["Widely available\nLow cost\nIdeal for small quantized models\n(Phi-3 Mini, LLaMA 2 7B 4-bit)"]
    CPU --> C2["Trade-off: High latency\nSeveral times slower inference\nNot ideal for real-time apps"]

    GPU --> G1["Massive parallel processing\nIdeal matrix multiplications\nLow latency + high throughput"]
    GPU --> G2["Higher cost\nEssential for chatbots, content generation\nenterprise integrations"]

Single GPU vs Multi-GPU

Single GPU:
├─ Works well for models ≤ 13B parameters
├─ A100 40GB handles a 13B in FP16
└─ Or much larger models if quantized

Multi-GPU:
├─ Necessary for models > 30B parameters generally
├─ Tensor parallelism: splits the model between GPUs
├─ Pipeline parallelism: layers on different GPUs
└─ Increased complexity: deployment, monitoring, scaling

AWS EC2 Instance Table for LLM

Serving Configuration Parameters

Right-sizing ≠ hardware only → also configuration!

Key parameters to tune in vLLM / LM Studio:
├─ Batch size    : ⬆ throughput but ⬆ individual latency
├─ Context length: ⬆ long context support but ⬆ VRAM
├─ GPU offload   : number of layers offloaded to GPU
└─ Tensor/pipeline parallelism: split across multiple GPUs

Practical rules:
├─ Interactive chat  → small batches, low latency
└─ Background jobs   → large batches, high throughput

Concrete Use Cases

🏢 Startup — Internal FAQ chatbot (low traffic):
   ├─ Model: 7B (4-bit)
   ├─ GPU: Single GPU (RTX 4090 / B200)
   ├─ CPU acceptable if budget tight
   └─ Avoid expensive GPU clusters

🏭 Enterprise — High traffic service:
   ├─ Model: 13B–70B based on needs
   ├─ Multi-GPU setup
   ├─ vLLM with PagedAttention
   └─ Kubernetes for orchestration

Demo 4 — Adjusting Configuration Parameters

AWS EC2 Instance Comparison

Critical point: High CPU capacity and network do not replace GPUs for LLM inference, especially for low-latency workloads.

Runtime Optimization with LM Studio

Basic configuration (starting point):

GPU offload     : 16 / 32 layers
Context length  : unchanged (default)
Batch size      : default

→ Result: 9.15 tokens/sec
          0.24s time to first token

Optimized configuration (after tuning):

Context length  : 131,072 tokens
GPU offload     : 32 / 32 layers (100%)
Batch size      : 2048

→ Result: ~83 tokens/sec
          0.03s time to first token

Performance Comparison

Configuration   Tokens/sec   Time to 1st token   Ratio
Basic           9.15         0.24s                1x
Optimized       ~83          0.03s                ~9x

Conclusions:

• Hardware choice is only the first step
• Runtime tuning unlocks hidden performance
• Same instance → radically different results
• Intelligent configuration saves real money

Module 3 — Optimize and Monitor for Production

Optimization Techniques

Even with optimal right-sizing, deploying LLMs at scale remains expensive. A single NVIDIA A100 GPU costs approximately $3–4 per hour in cloud environments, and a single 70B model can consume 140 GB of VRAM. Poor optimization can double the bill.

The Cost-Control Trifecta

graph TD
    CC[Cost-Control Trifecta]
    CC --> COMP[Compression\nShrinks models]
    CC --> CACHE[Caching\nReuses computation]
    CC --> BATCH[Batching\nBoosts throughput]

    COMP --> COMP1["Quantization\nPruning\nDistillation"]
    CACHE --> CACHE1["KV Cache\nvLLM & TGI\nLong conversations"]
    BATCH --> BATCH1["Parallel GPU\nMultiple requests\n→ single forward pass"]

1. Model Compression

Quantization (already covered in Module 2)

• Optimization standard: reduce precision of weights
• Tools: bitsandbytes, GPTQ, LLM.int8()

Pruning

Pruning identifies redundant parameters and eliminates them, creating a smaller, faster model.

Full model (Dense)         Pruned model (Sparse)
┌──────────────────┐        ┌──────────────────┐
│ ●●●●●●●●●●●●●●●● │        │ ●░●░●●░●░●●░●●░● │
│ ●●●●●●●●●●●●●●●● │   →    │ ░●●░●░●●●░●●░●●░ │
│ ●●●●●●●●●●●●●●●● │        │ ●●░●●●░●●●░●●●●░ │
└──────────────────┘        └──────────────────┘
  Billions of connections     30–60% eliminated

Pruning tools:

• SparseGPT
• DeepSpeed Compression

Note: Pruning is applied with a sparsity schedule and often requires retraining or fine-tuning. It is not a zero-cost operation.

Distillation

Train a smaller “student” model to mimic a larger “teacher” model.

graph LR
    T["🎓 Teacher Model\n(Large, accurate)\nEx: LLaMA 70B"] -->|Knowledge\nTransfer| S["📚 Student Model\n(Small, fast)\nEx: Mistral 7B"]
    S --> P["🚀 Production\nFast + Cost-effective\nInherited reasoning"]

Concrete examples:

• DeepSeek-R1-distill — distilled from a large reasoning model
• Mistral Instruct 7B — inherits capabilities from a larger model

Practical choices:

Quantization + Pruning = Quick win
├─ Reduces inference costs at scale
└─ Use bitsandbytes / LLM.int8()

Distillation = More effort
└─ Requires training cycles

2. KV Caching

Without cache: each request reprocesses the entire history — like re-reading an entire book at each page.

Without KV Cache:
Request 1: [token1 token2 token3] → full computation
Request 2: [token1 token2 token3 token4] → entire recomputation!
                                          → Huge latency overhead

With KV Cache:
Request 1: [token1 token2 token3] → compute + store KV
Request 2: [        ...reused     token4] → only new token computed
                                          → Massive acceleration

Implementation in vLLM:

# vLLM activates KV Cache automatically via PagedAttention
# Configuring KV Cache size
python3 -m vllm.entrypoints.api_server \
  --model mistralai/Mistral-7B-Instruct-v0.2 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 4096  # Max context length (cache)

Results:

• Major speed increases at reduced cost
• Ideal for long chats
• vLLM > TorchServe for stateless pipelines

3. Batching

Batching groups multiple requests and processes them in a single GPU forward pass.

Without batching:
Request 1 → [GPU Forward Pass] → Response 1
Request 2 → [GPU Forward Pass] → Response 2
Request 3 → [GPU Forward Pass] → Response 3
GPU used at ~33%

With batching:
Requests 1+2+3 → [Single GPU Forward Pass] → Responses 1+2+3
GPU used at ~95%

Batching trade-off:

graph LR
    BS[Batch size] --> T[Throughput\n⬆ higher]
    BS --> L[Individual latency\n⬆ higher]

    T --> U1["✅ Ideal for:\nBatch summaries\nContent pipelines"]
    L --> U2["⚠️ Not ideal for:\nInteractive chat\nReal-time apps"]

Optimization Overview

graph TD
    LLM[LLM in Production]
    LLM --> C1["Compression\n(Quantization + Pruning + Distillation)\nReduces model size"]
    LLM --> C2["Caching\n(KV Cache in vLLM/TGI)\nReuses previous computations"]
    LLM --> C3["Batching\n(Parallel GPU processing)\nIncreases throughput"]

    C1 --> R["💰 Cost reduction\n⚡ Better performance\n🏭 Scalable deployment"]
    C2 --> R
    C3 --> R

Demo 1 — Latency and Throughput Benchmarking

Objective: Establish a performance baseline and stress-test a local LLM to identify its breaking point under concurrency.

Environment:

• LM Studio (local server on port 1234)
• Model: Microsoft Phi-3 Mini
• Postman Desktop Agent

Environment Configuration

LM Studio:
├─ "Local Server" tab → Server ON
├─ Port: 1234 on 127.0.0.1
└─ Phi-3 model loaded (green bar confirmed)

Part 1 — Establish Baseline

Create the Postman collection:

Collections → (+) New collection → "LLM Benchmark"
→ Add a request:
   Method: POST
   URL: http://localhost:1234/v1/chat/completions
   Body: raw → JSON

JSON Payload:

{
  "model": "phi-3",
  "messages": [
    {
      "role": "user",
      "content": "Explain benchmarking LLM in 50 words."
    }
  ],
  "temperature": 0.7
}

Baseline result:

Status   : 200 OK ✅
Latency  : ~480 ms
Traffic  : 0 concurrent users
→ This is our latency baseline

Part 2 — Stress Testing (the “Runner”)

Configuration in Postman:

Runner → "Performance" tab
├─ Virtual Users  : 20 (simulates 20 simultaneous students)
├─ Duration       : 1 minute
└─ Load Profile   : Fixed
→ Click Run

Results Analysis

Latency evolution:
├─ Start      : ~500ms (baseline)
├─ Progression: Rapid rise over time
└─ Peak       : 4,000ms – 7,000ms

Throughput    : ~2 req/sec (stable)
Local GPU     : Fully utilized

Approximate timeline:
t=0s   : 500ms latency (baseline)
t=10s  : 1,000ms (queue building up)
t=20s  : 2,500ms (saturation)
t=30s  : 4,000–7,000ms (peak — breaking point)
t=60s  : End of test

Key insight:

┌─────────────────────────────────────────────────────────────────┐
│  The model didn't get "slower" because of the weights.          │
│  It got slow because of CONCURRENCY.                            │
│                                                                 │
│  Hardware processes as fast as possible (Throughput) ✅         │
│  But user experience degrades (Latency) ❌                      │
│  because requests are stuck in the queue                        │
└─────────────────────────────────────────────────────────────────┘

Backup, Recovery, and Rollback

In production, one thing is certain: something will go wrong. GPU failure, container crash, or that beautiful new model update that starts giving inconsistent responses.

Reliability isn’t about luck — it’s about design.

The 4 Pillars of Reliability

graph TD
    R[Production Reliability]
    R --> V[Versioning]
    R --> B[Backup]
    R --> RC[Recovery]
    R --> RB[Rollback]

    V --> V1["Each deployment = distinct version\nModel registry: MLflow / HuggingFace Hub / Vertex AI\nTag each deployment\nWithout versioning = blind fixes"]
    B --> B1["Weights, tokenizer, configs = the DNA\nRedundant storage (S3 + versioning)\n3 last versions accessible\nContainer images + libraries + orchestration"]
    RC --> RC1["Redeploy containers or restore models\nKubernetes restarts failing pods\nOrchestration tools reschedule workloads\nIntegrate recovery into CI/CD pipeline"]
    RB --> RB1["Instantly revert to stable version\nBlue/green deployments\nCanary rollouts\nCouple with health checks + alerts"]

Backup Strategy

What to back up:

1. The Model (the DNA):
   ├─ Model weights
   ├─ Tokenizer
   └─ Configurations

2. The Environment (the habitat):
   ├─ Container images
   ├─ Libraries (exact versions)
   └─ Orchestration parameters

→ Without both: recovery is incomplete
→ A model that runs fine on PyTorch 2.0 may break on 2.1!

Example: S3 Storage with versioning

# Configure S3 bucket with versioning
aws s3api put-bucket-versioning \
  --bucket my-llm-artifacts \
  --versioning-configuration Status=Enabled

# Upload model artifacts
aws s3 cp ./model-weights/ s3://my-llm-artifacts/llama3-v1.2/ \
  --recursive

# List versions
aws s3api list-object-versions \
  --bucket my-llm-artifacts

Rollback with Kubernetes

# Deployment with version configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-deployment
  annotations:
    deployment.kubernetes.io/revision: "3"
spec:
  replicas: 3
  selector:
    matchLabels:
      app: llm
      version: v1.2-stable   # ← Explicit version tag
  template:
    spec:
      containers:
      - name: vllm
        image: registry/vllm-llama3:v1.2-stable  # ← Versioned image

# View deployment history
kubectl rollout history deployment/llm-deployment

# Rollback to previous version
kubectl rollout undo deployment/llm-deployment

# Rollback to a specific version
kubectl rollout undo deployment/llm-deployment --to-revision=2

# Check rollback status
kubectl rollout status deployment/llm-deployment

Fundamental Principle

┌─────────────────────────────────────────────────────────────────┐
│  🚨 RESTORE SERVICE FIRST → DEBUG LATER 🚨                      │
│                                                                 │
│  1. Detect the problem (health checks / alerts)                 │
│  2. Rollback IMMEDIATELY to last stable version                 │
│  3. Service restored → happy users                              │
│  4. Only then: analyze and fix the problem                      │
└─────────────────────────────────────────────────────────────────┘

Demo 2 — Rollback to a Previous Model Version

Environment: LM Studio (simulates vLLM/TGI in production)

Part 1 — Stable State (Version 1)

# In LM Studio: load Phi-3 Mini
# → Simulates our current efficient production deployment

# Health check:
# Test prompt → >130 tokens/sec ✅
# RAM usage: low and stable ✅
# Status: healthy system

Part 2 — Simulate a Bad Deployment (Version 2)

# Eject Phi-3 Mini
# Load Gemma-2 27B (model too large for our hardware)
# → Simulates: pushing an update without resource benchmarking

# Same request as Part 1:
# Latency    : ⬆ Massive delay before generation
# RAM usage  : ⬆ Spike → saturated resources
# Speed      : ⬇ Crash to ~2 tokens/sec ❌
# UX         : Massive lag → failed deployment

Part 3 — Rollback Strategy

Action: DO NOT wait for generation to complete
        DO NOT try to optimize parameters now

Steps:
1. Click "Eject" on the bad model IMMEDIATELY
2. Reload Phi-3 Mini (Version 1)
3. Rerun the same prompt

→ Result: ~140 tokens/sec ✅
→ Service restored!

Kubernetes Context (real production)

# This manual "Eject/Load" in LM Studio corresponds to:

# Kubernetes - liveness probe detects degradation
livenessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 3

# → Traffic is redirected to the healthy container (V1)
# → Automatically, without manual intervention

Security Best Practices

You’ve optimized for speed and built safety nets for crashes, but none of that matters if a hacker ruins you.

💡 Leaving your LLM endpoint unsecured is like giving your credit card to ChatGPT saying “be reasonable”. You’re not going to like the results.

Security Overview

graph TD
    SEC[Model Endpoint\nSecurity]

    SEC --> AUTH[Authentication\nWho are you?]
    SEC --> AUTHZ[Authorization\nWhat can you do?]
    SEC --> TLS[Transport Security\nHow does data travel?]
    SEC --> RATE[Abuse Protection\nPrevent flooding]
    SEC --> MON[Monitoring & Auditing\nWhat is happening?]
    SEC --> LLM_RISK[LLM-Specific Risks\nLLM-specific threats]

    AUTH --> A1["API Keys\nRotate regularly\nRevoke if compromised\nScope per endpoint"]
    AUTH --> A2["OAuth2 / JWT\nAzure AD / Okta / AWS Cognito\nRole-based access control"]
    AUTHZ --> AZ1["Read-only for external clients\nAdmin access for engineers\nPrevent privilege abuse"]
    TLS --> TLS1["HTTPS required\nTLS certificates\nMutual TLS for sensitive workloads"]
    RATE --> R1["Token-based rate limiting\nRequest quotas\nAPI Gateways: Nginx / Kong / AWS APIGW\nvLLM behind these layers"]
    MON --> M1["Prometheus + Grafana\nCloudWatch\nCentralized logs\nGDPR/HIPAA/SOC 2 compliance"]
    LLM_RISK --> LR1["Prompt injection\nData leakage\nSanitize inputs\nLimit context exposure"]

Authentication

API Keys (simple, small teams):

# FastAPI example with API Key authentication
from fastapi import FastAPI, HTTPException, Security
from fastapi.security.api_key import APIKeyHeader
import secrets

app = FastAPI()
API_KEY_HEADER = APIKeyHeader(name="X-API-Key")

VALID_API_KEYS = {
    "client-app-1": "sk-abc123def456",
    "internal-tool": "sk-xyz789uvw012",
}

async def verify_api_key(api_key: str = Security(API_KEY_HEADER)):
    if api_key not in VALID_API_KEYS.values():
        raise HTTPException(status_code=403, detail="Invalid API Key")
    return api_key

@app.post("/generate")
async def generate(request: dict, api_key: str = Security(verify_api_key)):
    # Forward to vLLM
    return {"response": "..."}

OAuth2 / JWT (enterprise):

# JWT verification example with python-jose
from fastapi import HTTPException, Security
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
from jose import JWTError, jwt

security = HTTPBearer()
SECRET_KEY = "your-secret-key"  # In production: use a vault
ALGORITHM = "HS256"

async def verify_token(
    credentials: HTTPAuthorizationCredentials = Security(security)
):
    try:
        payload = jwt.decode(
            credentials.credentials,
            SECRET_KEY,
            algorithms=[ALGORITHM]
        )
        username: str = payload.get("sub")
        if username is None:
            raise HTTPException(status_code=401)
        return payload
    except JWTError:
        raise HTTPException(status_code=401, detail="Invalid token")

Rate Limiting

# Example with slowapi (rate limiter for FastAPI)
from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/generate")
@limiter.limit("10/minute")  # Max 10 requests per minute per IP
async def generate(request: Request, payload: dict):
    # ...

With Nginx as reverse proxy:

# Nginx configuration for vLLM rate limiting
http {
    limit_req_zone $binary_remote_addr zone=llm_api:10m rate=10r/s;

    server {
        listen 443 ssl;
        ssl_certificate /etc/ssl/certs/cert.pem;
        ssl_certificate_key /etc/ssl/private/key.pem;

        location /v1/ {
            limit_req zone=llm_api burst=20 nodelay;
            proxy_pass http://localhost:8000;
            proxy_set_header X-API-Key $http_x_api_key;
        }
    }
}

Monitoring and Auditing

# Example Prometheus configuration for vLLM monitoring
# prometheus.yml
scrape_configs:
  - job_name: 'vllm'
    static_configs:
      - targets: ['localhost:8000']
    metrics_path: '/metrics'

# Key metrics to monitor:
# - vllm:num_requests_total
# - vllm:request_latency_seconds
# - vllm:gpu_cache_usage_perc
# - vllm:num_running_requests

Authentication Error Codes

Demo 3 — Adding API Key Authentication

Objective: Demonstrate how cloud infrastructure uses “Authentication Handshakes” to protect compute resources from unauthorized access.

Environment: Google AI Studio + Postman

Step 1 — Get Credentials

Google AI Studio → "Get API key"
├─ Click "Create API key"
├─ Select or create a cloud project
└─ ⚠️ SECURITY: Copy the key and keep it secret
   In banking production: exposing this key in public code
   can lead to massive financial costs or data leaks

Step 2 — Test the Unsecured State

Postman:
├─ Method: POST
├─ URL: https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent
└─ Body (raw JSON):

{
  "contents": [{
    "parts": [
      {"text": "Verify secure connection test."}
    ]
  }]
}

→ Send WITHOUT authentication key
→ Expected result: 400 (API_KEY_INVALID) or 403 Forbidden ❌

"Slamming the door" = first layer of infrastructure security.
Only identified traffic reaches your models.

Step 3 — Perform the “Secure Handshake”

Method 1 — Key in URL (avoid in production!):

URL: https://...googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key=YOUR_KEY
→ Status 200 OK ✅
⚠️ Risk: the key can be logged in servers / history

Method 2 — Custom header (recommended in production):

Postman → Headers tab:
├─ Key   : x-goog-api-key
└─ Value : YOUR_API_KEY

→ Status 200 OK ✅
→ More secure: the key is not in the URL or logs

// Complete payload for the Secure Handshake
{
  "contents": [{
    "parts": [
      {"text": "Verify secure handshake for production demo. Output a status report."}
    ]
  }]
}

Step 4 — Verify Success

Status: 200 OK ✅
The model responded → it recognized our secure context

Resulting pipeline:
Client → [API Key in Header] → Google Infrastructure → Gemini Model → Response

Key Technical Points

┌─────────────────────────────────────────────────────────────────┐
│  Authentication concepts                                        │
├─────────────────────────────────────────────────────────────────┤
│  Authentication (AuthN)  : Proving who you are (API Key)        │
│  Authorization (AuthZ)   : What you are allowed to do           │
│                                                                 │
│  401 vs 403:                                                    │
│  ├─ 401 = Missing or invalid credentials                        │
│  └─ 403 = Authenticated but not authorized for this resource    │
│                                                                 │
│  Bearer Tokens / Headers:                                       │
│  Industry standard for transporting credentials                 │
│  in HTTP header → "password" not logged in URL                  │
└─────────────────────────────────────────────────────────────────┘

General Summary

Complete Architecture of a Production LLM Deployment

graph TD
    subgraph Clients
        C1[Web App]
        C2[Chatbot]
        C3[API Client]
    end

    subgraph Security Layer
        GW[API Gateway\nNginx / Kong / AWS APIGW\nRate Limiting + Auth]
    end

    subgraph Orchestration Layer
        K8S[Kubernetes\nLoad Balancing\nAuto-scaling\nHealth Checks]
    end

    subgraph Serving Layer
        VLLM1[vLLM Pod 1\nMistral-7B\nPagedAttention]
        VLLM2[vLLM Pod 2\nMistral-7B\nPagedAttention]
    end

    subgraph Hardware Layer
        GPU1[NVIDIA A100\nCUDA + cuDNN]
        GPU2[NVIDIA A100\nCUDA + cuDNN]
    end

    subgraph Monitoring
        PROM[Prometheus]
        GRAF[Grafana]
        CW[CloudWatch]
    end

    subgraph Backup
        S3[AWS S3\nModel Artifacts\nVersioning]
        REG[Model Registry\nMLflow / HF Hub]
    end

    C1 & C2 & C3 --> GW
    GW --> K8S
    K8S --> VLLM1 & VLLM2
    VLLM1 --> GPU1
    VLLM2 --> GPU2
    VLLM1 & VLLM2 --> PROM --> GRAF
    VLLM1 & VLLM2 -.-> S3
    K8S -.-> REG

Final Decision Matrix

graph TD
    START{New deployment\ndecision}

    START --> Q1{Sensitive data\nGDPR/HIPAA?}
    Q1 -->|Yes| LOC[On-premise or\nBYOK Private Cloud\nOllama or internal vLLM]
    Q1 -->|No| Q2{GPU budget\navailable?}

    Q2 -->|Tight| QUANT[Quantization + CPU\nor small GPU\nLocal Ollama\n7B 4-bit max]
    Q2 -->|Medium| SINGLE[Single Cloud GPU\nvLLM or TGI\nA100 / A10G]
    Q2 -->|High| MULTI[Multi-GPU\nvLLM + Kubernetes\nA100/H100/B200]

    SINGLE --> Q3{High traffic?}
    Q3 -->|No| SINGLE
    Q3 -->|Yes| MULTI

Production Deployment Checklist

BEFORE DEPLOYMENT:
□ Choose the right serving framework (vLLM / TGI / Ollama)
□ Evaluate constraints: latency, privacy, budget
□ Apply appropriate quantization
□ Right-size the GPU instance
□ Containerize with Docker
□ Configure Kubernetes (if > 1 instance)

SECURITY:
□ Configure authentication (API Key or OAuth2/JWT)
□ Set up rate limiting
□ Enable HTTPS/TLS
□ Configure authorization policies
□ Implement monitoring (Prometheus + Grafana)

RELIABILITY:
□ Version each deployment
□ Configure backups (S3 + versioning)
□ Test rollback before production
□ Configure Kubernetes liveness/readiness probes
□ Integrate recovery into CI/CD pipeline

OPTIMIZATION:
□ Benchmark latency and throughput (Postman/locust)
□ Tune batch size based on use case
□ Enable KV Cache in vLLM/TGI
□ Evaluate pruning/distillation if costs too high
□ Monitor and adjust continuously

References and Tools

Search Terms

deploying · open-source · llms · llm · application · development · artificial · intelligence · generative · ai · deployment · model · gpu · test · comparison · quantization · aws · configuration · production · rollback · authentication · kubernetes · local · ollama