LLMOps: Evaluation, Observability and Quality

Topics: LLM Evaluation · Observability · Automated Testing · Drift Detection

Table of Contents

1. LLM Evaluation Metrics and Frameworks
   • 1.1 Categories of evaluation metrics
   • 1.2 Automated metrics
   • 1.3 LLM-as-a-judge
   • 1.4 Common evaluation dimensions
   • 1.5 Human evaluation
   • 1.6 Evaluation dataset design
   • 1.7 Multidimensional evaluation
   • 1.8 Safety and instruction adherence
   • 1.9 Task-specific quality metrics
   • 1.10 Multimodal evaluation strategies
   • 1.11 Demo — Multimodal evaluation datasets
2. Observability, Logging and Continuous Evaluation
   • 2.1 GenAI request logging
   • 2.2 Logging schemas and compliance
   • 2.3 Regression testing for GenAI
   • 2.4 Continuous evaluation pipelines
   • 2.5 Evaluation dashboards
   • 2.6 Demo — Observability platform
3. Automated Testing Strategies
   • 3.1 The GenAI testing pyramid
   • 3.2 Unit testing of LLM components
   • 3.3 Integration testing of workflows
   • 3.4 Property-based testing
   • 3.5 Demo — Automated GenAI tests
4. Drift Detection and Model Monitoring
   • 4.1 Types of drift in GenAI systems
   • 4.2 Drift detection systems
   • 4.3 Monitoring dashboards
   • 4.4 Canary testing and gradual deployments
   • 4.5 Experiment tracking and rollback
   • 4.6 Demo — Experiment tracking with MLflow

1. LLM Evaluation Metrics and Frameworks

1.1 Categories of Evaluation Metrics

LLM evaluation metrics help understand the real quality of a model’s outputs. They provide a consistent way to measure the quality, correctness, and usefulness of responses for:

• Comparing different models
• Tracking improvements over time
• Deciding whether a model is ready for production

mindmap
  root((LLM Evaluation))
    Automated metrics
      BLEU
      ROUGE
      BERTScore
    Model-based evaluation
      LLM-as-a-judge
      Numeric scores
      Categorical labels
    Human evaluation
      Experts
      Crowd review
      Clear rubrics

1.2 Automated Metrics

BLEU      ──► n-gram overlap (precision)
ROUGE     ──► n-gram recall (coverage)
BERTScore ──► semantic similarity (embeddings)

1.3 LLM-as-a-Judge

The LLM-as-a-judge approach uses an LLM to automatically evaluate the outputs of another model according to defined criteria (accuracy, relevance, coherence, instruction adherence).

flowchart TD
    A[User prompt + Model response] --> B[Provide full context to the judge model]
    B --> C[Define the evaluation rubric\naccuracy · relevance · clarity · safety]
    C --> D[The judge produces structured outputs\nscores · labels · comparisons]
    D --> E[Apply uniformly across a large dataset]
    E --> F[Aggregate results\nand calculate global metrics]
    F --> G[Actionable insights\nfor model iteration]

Advantages:

• Scalable and repeatable evaluation
• No exclusive dependency on human evaluators
• Results are easy to analyze programmatically

1.4 Common Evaluation Dimensions

graph LR
    A[LLM Output] --> B{Evaluation Dimensions}
    B --> C[Accuracy and factual correctness]
    B --> D[Relevance]
    B --> E[Coherence and clarity]
    B --> F[Instruction adherence]
    B --> G[Safety and compliance]

Concrete example — Multidimensional evaluation:

Question: What was the cause of the 2008 financial crisis?

1.5 Human Evaluation

Human evaluation captures nuances that automated metrics often miss (clarity, usefulness, intent).

flowchart LR
    A[Human evaluation] --> B[Experts]
    A --> C[Crowd review]
    B --> D[Deep domain knowledge\nEssential for high-stakes cases]
    C --> E[High volume of outputs evaluated quickly\nRequires clear instructions and rubrics]
    D --> F[Optimal combination:\nreliable evaluation + cost control]
    E --> F

Best practices:

• Define precise rubrics to align evaluators
• Combine experts and crowd review to balance depth and scalability
• Reduce bias through calibration examples

1.6 Evaluation Dataset Design

Evaluation datasets provide a consistent and repeatable way to measure model quality.

graph TD
    A[Ideal evaluation dataset] --> B[Real production prompts]
    A --> C[Full context\nsystem prompts · constraints]
    A --> D[Imperfect data\nincomplete · ambiguous · noisy]
    A --> E[High-impact failure cases]
    A --> F[Diversity of styles and domains]
    A --> G[Aligned with production criteria]

Considerations for ground truth and references:

1.7 Multidimensional Evaluation

Main Challenges

graph LR
    A[Multidimensional evaluation challenges] --> B[Single metric insufficient]
    A --> C[Trade-offs between dimensions]
    A --> D[Subjective dimensions resisting automation]
    A --> E[Weighting depends on the task]
    A --> F[Consistency difficult across evaluators]

    C --> G[E.g.: verbosity ↑ completeness but ↓ relevance]

Main KPIs

┌─────────────────────────────────────────────────────────────┐
│                   Multidimensional KPIs                      │
├──────────────┬──────────────────────────────────────────────┤
│  Accuracy    │ Factually correct outputs vs. ground         │
│              │ truth. Evaluated via comparisons, experts     │
│              │ or LLM-as-a-judge.                            │
├──────────────┼──────────────────────────────────────────────┤
│  Relevance   │ Does the response address the user's         │
│              │ intent? Evaluated via semantic similarity      │
│              │ or human evaluation.                          │
├──────────────┼──────────────────────────────────────────────┤
│  Coherence   │ Logical structure, readability, internal     │
│              │ consistency. Essential for long or            │
│              │ multi-step responses.                         │
└──────────────┴──────────────────────────────────────────────┘

Insight: A model scoring 5/5/2 (accuracy, relevance, coherence) tells a very different story from a model scoring 2/5/5.

1.8 Safety and Instruction Adherence

flowchart TD
    A[Safety & Adherence Evaluation] --> B[Verify compliance with security policies]
    A --> C[Evaluate adherence to system prompts\nand user instructions]
    A --> D[Measure refusal quality\nfor unauthorized requests]
    A --> E[Detect dangerous behaviors\nhallucinations · toxicity · data exposure]
    A --> F[Combine automated classifiers,\nrules and targeted human reviews]
    A --> G[Track separately to avoid masking\ncritical failures behind task successes]

Risks to monitor:

• Hallucinations and false claims
• Toxic or biased language
• Unintentional exposure of sensitive data
• Policy or regulatory requirement violations

1.9 Task-Specific Quality Metrics

1.10 Multimodal Evaluation Strategies

Vision Question Answering (Vision-QA)

graph TD
    A[Vision-QA Evaluation] --> B[Comparison with ground truth]
    B --> C[Exact match\nfor closed questions\nyes/no · colors · categories]
    B --> D[Soft accuracy\nfor semantic equivalents]
    B --> E[Accuracy by question type\nnumeric · descriptive · yes/no]

Document Understanding Benchmarks

Document Evaluation Dimensions

┌──────────────────────────────────────────────────────────────┐
│              Dimensions — Document Understanding              │
├─────────────────────────────┬────────────────────────────────┤
│ Text extraction accuracy    │ Text conversion from scanned    │
│                             │ docs and digital PDFs           │
├─────────────────────────────┼────────────────────────────────┤
│ Layout understanding        │ Tables, form fields,            │
│                             │ headers, spatial hierarchy      │
├─────────────────────────────┼────────────────────────────────┤
│ Semantic accuracy           │ Values assigned to correct      │
│                             │ fields, meaning preserved       │
├─────────────────────────────┼────────────────────────────────┤
│ End-to-end performance      │ Business results (QA,           │
│                             │ structured extraction)          │
└─────────────────────────────┴────────────────────────────────┘

OCR Quality Metrics

Illustrative example — OCR invoice analysis:

┌──────────────────────────────────────────────────────────────┐
│  OCR Analysis — Invoice                                      │
├──────────────────────┬──────────────────────────────────────┤
│ Character accuracy   │ 97.4%  ← seems excellent...           │
│ Word accuracy        │ 90.4%  ← ~1 word in 10 is incorrect   │
├──────────────────────┴──────────────────────────────────────┤
│ PROBLEM: Errors are concentrated in critical fields —        │
│ zip codes, phone numbers, amounts.                           │
│                                                              │
│ "90210"  → "9021O"  (digit → letter, silent error)          │
│ "materials" → "materlals" (one character = wrong token)      │
│                                                              │
│ CONCLUSION: Not ready for production financial document      │
│ processing.                                                  │
└──────────────────────────────────────────────────────────────┘

1.11 Demo — Multimodal Evaluation Datasets

Objective: Create a systematic multimodal evaluation pipeline covering vision, documents, and cross-modal reasoning.

Notebook: 01/demos.ipynb

Step 1 — Setup and imports

from openai import OpenAI
from PIL import Image
import base64

Step 2 — Secure API key loading

with open("../api_key.txt", "r") as f:
    API_KEY = f.read().strip()

client = OpenAI(api_key=API_KEY)

Best practice: Never hard-code API keys in code. Load them from a file or environment variables.

Step 3 — Multimodal evaluation dataset

evaluation_dataset = [
    {
        "id": "vision_1",
        "type": "vision",
        "image_url": "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg",
        "question": "What animal is shown in the image?",
        "ground_truth": "cat"
    },
    {
        "id": "doc_1",
        "type": "document",
        "text": "Invoice Total: $250",
        "question": "What is the total amount?",
        "ground_truth": "$250"
    },
    {
        "id": "cross_modal_1",
        "type": "cross_modal",
        "image_url": "https://upload.wikimedia.org/wikipedia/commons/8/81/Stop_sign.png",
        "text": "This sign means vehicles must come to a complete halt.",
        "question": "What action should a driver take?",
        "ground_truth": "start"
    }
]

Step 4 — Image encoding for vision inputs

def encode_image(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

Step 5 — Multimodal evaluation loop

results = []

for sample in evaluation_dataset:

    # Vision only
    if sample["type"] == "vision":
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": sample["question"]},
                        {
                            "type": "image_url",
                            "image_url": {"url": sample["image_url"]}
                        }
                    ]
                }
            ]
        )
        answer = response.choices[0].message.content

    # Document only
    elif sample["type"] == "document":
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": sample["text"] + "\n\n" + sample["question"]
                }
            ]
        )
        answer = response.choices[0].message.content

    # Cross-modal (image + text)
    elif sample["type"] == "cross_modal":
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": sample["text"]},
                        {"type": "text", "text": sample["question"]},
                        {
                            "type": "image_url",
                            "image_url": {"url": sample["image_url"]}
                        }
                    ]
                }
            ]
        )
        answer = response.choices[0].message.content

    # Simple accuracy check
    correct = sample["ground_truth"].lower() in answer.lower()

    results.append({
        "id": sample["id"],
        "type": sample["type"],
        "correct": correct,
        "model_answer": answer
    })

Step 6 — Overall accuracy calculation

total = len(results)
correct = sum(r["correct"] for r in results)
accuracy = correct / total

print("Evaluation Results")
print("Total Samples:", total)
print("Correct:", correct)
print("Accuracy:", round(accuracy, 2))

Step 7 — Accuracy by modality

from collections import defaultdict

by_type = defaultdict(list)

for r in results:
    by_type[r["type"]].append(r["correct"])

print("Accuracy by Modality")
for modality, scores in by_type.items():
    print(modality, ":", round(sum(scores) / len(scores), 2))

Multimodal evaluation pipeline architecture:

flowchart LR
    DS[Evaluation dataset\nvision · document · cross-modal] --> LOOP[Evaluation loop]
    LOOP --> V[Vision\nimage_url + question]
    LOOP --> D[Document\ntext + question]
    LOOP --> CM[Cross-modal\nimage + text + question]
    V --> M[gpt-4o-mini]
    D --> M
    CM --> M
    M --> ACC[Check vs. ground truth]
    ACC --> REPORT[Report by modality\nglobal accuracy]

2. Observability, Logging and Continuous Evaluation

2.1 GenAI Request Logging

To make every interaction observable, you need to capture the inputs and outputs of every request:

graph TD
    REQ[User request] --> PC[Prompt construction]
    PC --> RET[Retrieval / RAG]
    RET --> INF[Model inference]
    INF --> POST[Post-processing]
    POST --> RES[Response]

    PC --> L1[Log: prompt construction time]
    RET --> L2[Log: retrieval latency]
    INF --> L3[Log: inference latency]
    POST --> L4[Log: end-to-end duration]
    
    L1 & L2 & L3 & L4 --> STORE[Structured log store]

Core elements to log

Latency components to log

┌────────────────────────────────────────────────────────────┐
│              Latency Breakdown                              │
├─────────────────────────────┬──────────────────────────────┤
│ Prompt construction time    │ Template rendering, context   │
│                             │ injection, system prompts    │
├─────────────────────────────┼──────────────────────────────┤
│ Retrieval latency           │ Embedding lookup, similarity  │
│                             │ search, ranking/filtering     │
├─────────────────────────────┼──────────────────────────────┤
│ Model inference latency     │ Token generation,             │
│                             │ streaming, post-processing    │
├─────────────────────────────┼──────────────────────────────┤
│ End-to-end request duration │ Total duration as felt        │
│                             │ by the user                   │
└─────────────────────────────┴──────────────────────────────┘

Token consumption metrics

Cost per request

• Treat cost as a first-class signal, not an afterthought
• Log separately: prompt cost, completion cost, embedding cost
• Include retrieval cost, vector search, reranking
• Attribute costs by request context (feature, user, model version)
• Correlate cost with latency and quality to reason about trade-offs

2.2 Logging Schemas and Compliance

Minimal structured schemas

A minimal and consistent schema captures only what is necessary for observability, debugging, and evaluation.

Core fields to include:

┌─────────────────────────────────────────────────────────────┐
│                  Minimal log schema                          │
├──────────────────────┬──────────────────────────────────────┤
│ Request ID           │ Unique identifier to trace the        │
│                      │ request across services               │
├──────────────────────┼──────────────────────────────────────┤
│ Start/End timestamps │ Latency measurement and time-based    │
│                      │ dashboards                            │
├──────────────────────┼──────────────────────────────────────┤
│ Model name & version │ Attribution of quality changes to a   │
│                      │ specific release                      │
├──────────────────────┼──────────────────────────────────────┤
│ Prompt hash/ref      │ Traceability without storing          │
│                      │ sensitive prompt content              │
├──────────────────────┼──────────────────────────────────────┤
│ Output hash/summary  │ Regression detection without          │
│                      │ storing full payloads                 │
└──────────────────────┴──────────────────────────────────────┘

Sensitive data handling (PII)

flowchart TD
    DATA[Incoming data] --> DETECT[Detect and classify PII]
    DETECT --> REDACT[Mask or tokenize sensitive fields\nnames · emails · identifiers]
    REDACT --> STORE{Storage needed?}
    STORE -->|Yes| SECURE[Strict access control\nLimited retention]
    STORE -->|No| DISCARD[Do not store]
    SECURE --> AUDIT[Audit logs for regulatory compliance]

Retention rules:

• High-sensitivity data → shortest possible duration
• Aggregated metrics → can be retained longer
• Automated deletion policies → reduced operational risk

Sampling strategies for scalability

graph LR
    A[Sampling strategies] --> B[Random sampling\nFixed percentage of requests\nSimple but may miss rare cases]
    A --> C[Stratified sampling\nGuaranteed coverage by segment\nusers · prompts · model versions]
    A --> D[Adaptive sampling\nDynamic rate based on behavior\nIncreased during anomalies or degradations]

2.3 Regression Testing for GenAI

Snapshot testing

Saves model outputs for a fixed set of prompts at a point in time and uses them as a reference. When changes occur, new outputs are compared to snapshots to detect silent regressions.

Golden dataset evaluation

Run a fixed set of trusted prompts through the system after each change. Comparing new outputs to known reference results quickly detects regressions before deployment.

Property-based testing

flowchart LR
    A[Property-based Testing] --> B[Define invariants\nvalid JSON · safe content · consistent structure]
    B --> C[Automatically generate many\ninput variations]
    C --> D[Validate that properties hold\nno exact comparison]
    D --> E[Detect robustness, security\nand compliance regressions]

Typical properties to test:

• The response is valid JSON
• No unsafe content in the output
• The response structure is maintained
• Output length stays within acceptable limits

Key advantage: Property-based testing embraces the non-deterministic variability of LLMs by validating high-level guarantees rather than exact text.

2.4 Continuous Evaluation Pipelines

Continuous evaluation pipelines automatically run tests when code changes, model updates, or new data is introduced.

Evaluation triggers

graph TD
    A[Continuous evaluation triggers] --> B[Code changes\nprompts · orchestration]
    A --> C[Model version changes\nor parameters]
    A --> D[Data changes\nembeddings · index]
    A --> E[Dependency updates\nSDK · libraries]

Common trigger mechanisms

Automated batch evaluation

Batch pipeline operation:

1. Scheduled run (nightly / weekly)
      ↓
2. Standardized prompts across the full system
      ↓
3. Compare outputs with golden datasets
      ↓
4. Calculate metrics (accuracy, relevance, safety, latency, cost)
      ↓
5. Store results for trend analysis
      ↓
6. Deployment gates — automatically block if thresholds are not met

Deployment gates: Transform continuous evaluation into a real control mechanism, not just a passive signal. Deployments are automatically blocked if quality, latency, safety, or cost metrics don’t meet defined thresholds.

2.5 Evaluation Dashboards

Tracking trends over time

graph LR
    A[Evaluation dashboard] --> B[Quality metrics\nrelevance · accuracy · groundedness]
    A --> C[Operational metrics\nlatency · tokens · costs]
    A --> D[Segmentation\nversions · prompts · retrieval strategies]
    A --> E[Before/after deployment comparison]
    A --> F[Slow degradation detection]

Comparing model and prompt versions

Segmenting results by user

• Break down by meaningful attributes (role, region, language)
• Side-by-side comparisons to identify gaps between segments
• Track independently to avoid masking regressions for a group
• Prioritize improvements toward the users who benefit most

2.6 Demo — Observability Platform

Objective: Demonstrate a real-time observability dashboard with logging, visualization, and alerting.

Notebook: 02/demos.ipynb

Step 1 — Setup

import time
import pandas as pd
import matplotlib.pyplot as plt
import ipywidgets as widgets
from openai import OpenAI

Step 2 — In-memory observability store

logs = []

Step 3 — LLM call with logging

def logged_llm_call(prompt: str):
    start_time = time.time()
    success = True

    try:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}]
        )
        output = response.choices[0].message.content

    except Exception as e:
        output = str(e)
        success = False

    latency = time.time() - start_time

    logs.append({
        "timestamp": time.time(),
        "prompt": prompt,
        "latency": latency,
        "success": success
    })

    return output

Step 4 — Traffic simulation

prompts = [
    "Explain observability in one sentence",
    "What is LLM logging?",
    "Define alerting in monitoring systems"
]

for p in prompts:
    print(logged_llm_call(p))

Step 5 — Interactive dashboard

# Convert logs to DataFrame
df = pd.DataFrame(logs)

# Interactive dashboard widget
output = widgets.Output()

def update_dashboard():
    output.clear_output()
    with output:
        print("LLM Observability Dashboard")
        print("Total Requests:", len(df))
        print("Failures:", (~df["success"]).sum())
        print("Avg Latency (sec):", round(df["latency"].mean(), 2))

        df["latency"].plot(kind="bar", title="Request Latency")
        plt.show()

update_dashboard()
output

Step 6 — Alert logic (anomaly detection)

LATENCY_THRESHOLD = 2.0  # seconds

def check_alerts():
    if df["latency"].max() > LATENCY_THRESHOLD:
        print("ALERT: High latency detected!")
    if (~df["success"]).any():
        print("ALERT: Request failure detected!")

check_alerts()

Key insight: In production, 8-second latency on a stable workload would trigger an alert immediately. The threshold must be calibrated to the SLA, not hard-coded.

Observability pipeline architecture:

flowchart TD
    REQ[LLM Requests] --> LOG[logged_llm_call\nTimestamp · latency · success]
    LOG --> STORE[In-memory log store\nlogs = []]
    STORE --> DF[Pandas DataFrame]
    DF --> DASH[Dashboard\nTotal · Failures · Avg latency · Chart]
    DF --> ALERT{Thresholds exceeded?}
    ALERT -->|Latency > 2s| WARN[ALERT: High latency]
    ALERT -->|Failure detected| WARN2[ALERT: Request failure]

3. Automated Testing Strategies

3.1 The GenAI Testing Pyramid

graph TB
    A["🔺 Evaluation Tests\n(quality · safety · alignment)\nSlow · Expensive · Critical"]
    B["🔷 Integration Tests\n(RAG workflows · agents · tool calling)\nModerately slow · High confidence"]
    C["🔵 Unit Tests\n(prompts · parsers · retrievers)\nFast · Deterministic · Mocked"]

    C --> B --> A

    style A fill:#ff6b6b,color:#fff
    style B fill:#ffa500,color:#fff
    style C fill:#4ecdc4,color:#fff

Speed vs. realism trade-offs

Fast tests ──────────────────────────────── Slow tests
     │                                              │
Unit tests                              Evaluation tests
(LLM mocked)                            (real calls)
Fast feedback                           High confidence
Low cost                                High cost
Deterministic                           Non-deterministic

Quality and safety coverage

graph LR
    A[Quality and safety tests] --> B[Measure output quality]
    A --> C[Detect unsafe behaviors]
    A --> D[System-level evaluation]
    A --> E[Track accuracy metrics]
    A --> F[Identify security risks]
    A --> G[Enable continuous monitoring]

3.2 Unit Testing of LLM Components

Why mock LLM responses?

graph LR
    A[Mocking LLM] --> B[Fast, deterministic tests\nwithout external calls]
    A --> C[Eliminates variability\nof non-deterministic outputs]
    A --> D[Isolates application logic\nfrom LLM variability]
    A --> E[Simulates edge cases\nmalformed outputs · errors]
    A --> F[Reduces costs and\nservice dependencies]

Testing prompt templates and output parsers

Testing retrieval functions

Retrieval tests verify that components responsible for fetching context return correct and consistent results, independent of the LLM.

Elements to validate:

• Similarity filters and thresholds
• Edge case handling (no results, partial results)
• Similarity scoring accuracy

3.3 Integration Testing of Workflows

Why integration testing is critical

Integration testing ensures that prompts, models, tools, and data work together safely and reliably in real workflows, preventing hidden regressions and validating intelligence in context, not just functions.

GenAI vs. traditional systems

┌────────────────────────────────────────────────────────────┐
│     Traditional Systems  vs.  GenAI Systems                │
├─────────────────────────────┬──────────────────────────────┤
│ Deterministic outputs       │ Probabilistic outputs         │
│ Fixed APIs                  │ Prompt + model behavior       │
│ Schema-based validation     │ Semantic validation           │
│ Response mocks              │ Real data + calls             │
│ Pass/Fail                   │ Score · threshold · trend     │
└─────────────────────────────┴──────────────────────────────┘

Integration testing workflow (step by step)

flowchart TD
    A[Define end-to-end scenarios] --> B[Fix test determinism\nwhere possible]
    B --> C[Run the complete pipeline\nwithout mocks]
    C --> D[Evaluate with multidimensional metrics]
    D --> E[Combine automated evaluation\nand human-in-the-loop]

Components under integration testing

End-to-end RAG test

flowchart LR
    Q[Query] --> EMB[Embedding]
    EMB --> VS[Vector Store\nSimilarity Search]
    VS --> CTX[Context injection\ncontext window]
    CTX --> LLM[LLM\nGeneration]
    LLM --> ANS[Response]
    ANS --> EVAL[Evaluation\nvs. golden response]
    
    EVAL --> V1{Retrieval\ncorrect?}
    EVAL --> V2{Context\nproperly injected?}
    EVAL --> V3{Output quality\nacceptable?}

Common failure modes detected by integration tests:

┌──────────────────────────────────────────────────────┐
│          Failure Modes — Integration Tests            │
├──────────────────────┬───────────────────────────────┤
│ Contextual           │ Invented content in the        │
│ hallucinations       │ context of a full workflow     │
├──────────────────────┼───────────────────────────────┤
│ Synthesis errors     │ Incorrect combination of       │
│                      │ information from multiple      │
│                      │ sources                        │
├──────────────────────┼───────────────────────────────┤
│ Tool misuse          │ Incorrect tool calls or         │
│                      │ wrong sequences                │
├──────────────────────┼───────────────────────────────┤
│ Response imbalance   │ Responses too long or too      │
│                      │ short for the context          │
├──────────────────────┼───────────────────────────────┤
│ Silent failures      │ Failures not generating an     │
│                      │ explicit error                 │
└──────────────────────┴───────────────────────────────┘

Best practices:

• Treat prompts as versioned code
• Maintain a suite of golden scenarios
• Track quality metrics with costs
• Log everything (inputs, outputs, metadata)
• Re-run tests periodically, even without code changes

3.4 Property-Based Testing

Property-based testing validates system behavior by verifying that defined invariants hold for a wide range of automatically generated inputs.

graph TD
    A[Property-based Testing] --> B[Identify properties\nthat must always hold]
    B --> C{Types of invariants}
    C --> D[Safety\nno harmful content]
    C --> E[Structure\nvalid JSON · format respected]
    C --> F[Behavior\ncorrect refusal · acceptable length]
    B --> G[Automatically generate\ndiverse inputs]
    G --> H[Test robustness\nand edge cases]

Typical invariants:

• The output is valid JSON (if requested)
• No unsafe or biased content
• The response structure is maintained
• Refusals are correct for forbidden requests
• Output length stays within limits

Advantages for GenAI systems:

• Ideal for non-deterministic systems
• Explores edge cases hard to anticipate manually
• Detects robustness regressions during model updates
• Effective when prompts evolve or data distributions change

3.5 Demo — Automated GenAI Tests

Context: Testing an LLM-powered summarization function in a document processing pipeline.

Notebook: 03/demos.ipynb

Test strategy:

Step 1 — GenAI function under test

from openai import OpenAI

with open("../api_key.txt", "r") as f:
    API_KEY = f.read().strip()

client = OpenAI(api_key=API_KEY)

def summarize_text(text: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarize the text in one sentence."},
            {"role": "user", "content": text}
        ]
    )
    return response.choices[0].message.content

Step 2 — Unit test

def test_summary_is_short():
    text = "Agentic AI systems use tools to reason, plan, and act autonomously."
    summary = summarize_text(text)

    assert len(summary.split()) <= 20, "Summary is too long"

Step 3 — Integration test (end-to-end)

def test_end_to_end_pipeline():
    text = "Agentic AI systems are used in automation and decision-making workflows."
    summary = summarize_text(text)

    assert isinstance(summary, str), "Output is not a string"
    assert len(summary) > 0, "Summary is empty"

Step 4 — Test runner (CI simulation)

tests = [
    test_summary_is_short,
    test_end_to_end_pipeline
]

results = []

for test in tests:
    try:
        test()
        results.append((test.__name__, "PASS"))
    except Exception as e:
        results.append((test.__name__, "FAIL", str(e)))

Step 5 — Test report and CI/CD gate

print("Test Report")
failed = False

for result in results:
    print(result)
    if result[1] == "FAIL":
        failed = True

if failed:
    raise Exception("Tests failed. Deployment blocked.")
else:
    print("All tests passed. Ready to deploy.")

CI/CD flow with deployment gate:

flowchart LR
    CODE[Code or prompt change] --> CI[CI Pipeline triggered]
    CI --> UNIT[Unit tests\ntest_summary_is_short]
    CI --> INTEG[Integration tests\ntest_end_to_end_pipeline]
    UNIT --> GATE{All tests\npassing?}
    INTEG --> GATE
    GATE -->|Yes| DEPLOY[Deployment authorized]
    GATE -->|No| BLOCK[Deployment blocked\nException raised]

4. Drift Detection and Model Monitoring

4.1 Types of Drift in GenAI Systems

graph TD
    DRIFT[Drift types] --> ID[Input Distribution Drift]
    DRIFT --> OQ[Output Quality Drift]
    DRIFT --> RSD[Retrieval & System Drift]
    
    ID --> ID1[Evolving user behavior]
    ID --> ID2[Prompt template updates]
    ID --> ID3[Upstream data source changes]
    ID --> ID4[Seasonal or domain variations]
    
    OQ --> OQ1[Accuracy degradation]
    OQ --> OQ2[Relevance loss]
    OQ --> OQ3[Safety violations]
    
    RSD --> RSD1[Embedding drift]
    RSD --> RSD2[Index drift]
    RSD --> RSD3[Tool drift]

4.2 Drift Detection Systems

A drift detection system continuously monitors inputs, outputs, and the behavior of the GenAI system to identify significant changes.

flowchart TD
    A[Establish statistical baselines\nfrom input and output history] --> B[Monitor input\ndistribution changes]
    B --> C[Track prompt length,\ntokens, language, embeddings]
    C --> D[Measure output length,\nconfidence and content categories]
    D --> E[Compare distributions with\nstatistical distance metrics]
    E --> F[Apply thresholds to\ndetect significant drift]
    F --> G[Trigger automatic alerts]

Anomaly detection and reporting

┌─────────────────────────────────────────────────────────────┐
│              Anomaly Detection Pipeline                      │
├──────────────────┬──────────────────────────────────────────┤
│ Baseline         │ Establish reference distributions         │
├──────────────────┼──────────────────────────────────────────┤
│ Statistical      │ Statistical distance metrics              │
│ comparison       │ (KL divergence, PSI, etc.)                │
├──────────────────┼──────────────────────────────────────────┤
│ Thresholds       │ Define trigger values                     │
├──────────────────┼──────────────────────────────────────────┤
│ Detection        │ Sustained drift vs. isolated events       │
├──────────────────┼──────────────────────────────────────────┤
│ Trend analysis   │ Patterns across multiple periods          │
│                  │                                           │
├──────────────────┼──────────────────────────────────────────┤
│ Auto alerts      │ Routing to appropriate channels           │
└──────────────────┴──────────────────────────────────────────┘

Alert configuration:

• Define thresholds on statistically significant deviations
• Trigger on sustained drift, not isolated noisy events
• Route to correct channels or incident management systems
• Classify alerts: informational signals vs. critical failures
• Include metadata: model version, prompt type, time window

4.3 Monitoring Dashboards

Monitoring dashboards provide real-time visibility into the health, performance, and costs of production GenAI systems.

Key health metrics

graph LR
    DASH[Monitoring Dashboard] --> A[Error rates]
    DASH --> B[Latency percentiles\np50 · p95 · p99]
    DASH --> C[Cost per request]
    DASH --> D[Output quality signals]
    DASH --> E[Safety & policy violations]
    DASH --> F[Retrieval quality metrics]

Monitoring metrics over time

Comparing performance between versions

• Compare error rate, latency, cost, and quality between releases
• Assess the impact of prompt and system parameter updates
• Contrast health metrics between experimental and stable versions
• Identify performance drops or safety regressions after deployment
• Use past baselines to contextualize current performance

4.4 Canary Testing and Gradual Deployments

Canary testing

flowchart LR
    PROD[Production traffic] --> SPLIT{Routing}
    SPLIT -->|95% traffic| STABLE[Stable version\nBaseline]
    SPLIT -->|5% traffic| CANARY[New version\nCanary]
    STABLE --> M1[Baseline metrics]
    CANARY --> M2[Canary metrics]
    M1 & M2 --> COMPARE[Metrics comparison]
    COMPARE --> DECISION{Performance OK?}
    DECISION -->|Yes| ROLLOUT[Increase traffic progressively]
    DECISION -->|No| ROLLBACK[Automatic rollback]

Canary strategy: Expose a new model or prompt version to a restricted subset of traffic. Compare performance and quality metrics with the baseline before a full deployment.

Other testing strategies

Gradual deployments

Steps of a gradual deployment:

1. Define baseline and candidate versions
      ↓
2. Route a small percentage (e.g.: 5%) to the new version
      ↓
3. Increase traffic in controlled increments
      ↓
4. Monitor metrics at each step
      ↓
5. Automate rollback if metrics degrade

4.5 Experiment Tracking and Rollback

Experiment tracking tools

graph LR
    A[Experiment tracking tools] --> B[MLflow\nOpen source · LLM integration]
    A --> C[Weights & Biases\nAdvanced visualization]
    A --> D[Neptune.ai\nTeam collaboration]
    A --> E[ClearML\nFull MLOps]
    A --> F[Comet\nReal-time tracking]
    A --> G[DVC\nData Version Control]

Common experimental configurations

Rollback decisions

graph LR
    A[Rollback triggers] --> B[Degradation of accuracy,\nrelevance or user satisfaction]
    A --> C[Increase in error rates,\nlatency or costs]
    A --> D[Safety, compliance\nor policy violations]
    A --> E[Performance below baseline\nor best known run]
    A --> F[Canary or gradual\ndeployment failure]

4.6 Demo — Experiment Tracking with MLflow

Context: Jane, an ML engineer at Amazing Company, needs to decide which LLM configuration to deploy to production. She compares temperature=0.0 (deterministic) vs temperature=0.7 (expressive) with the same prompt.

Notebook: 04/demos.ipynb

Tracking strategy:

Step 1 — Setup

import mlflow
from openai import OpenAI
import time
import logging

with open("../api_key.txt", "r") as f:
    API_KEY = f.read().strip()

client = OpenAI(api_key=API_KEY)

Step 2 — MLflow experiment configuration

logging.getLogger("mlflow").setLevel(logging.ERROR)
mlflow.set_experiment("llm_experiment_tracking_demo")

Step 3 — LLM evaluation function

def run_llm_experiment(model_name, temperature, prompt):
    start_time = time.time()

    response = client.chat.completions.create(
        model=model_name,
        messages=[{"role": "user", "content": prompt}],
        temperature=temperature
    )

    latency = time.time() - start_time
    output = response.choices[0].message.content

    # Simple proxy metrics
    output_length = len(output)
    success = int(len(output.strip()) > 0)

    return output, latency, output_length, success

Step 4 — Log two experiments (different configurations)

# Run 1 — Low temperature (deterministic)
with mlflow.start_run(run_name="low_temperature_run"):
    mlflow.log_param("model", "gpt-4o-mini")
    mlflow.log_param("temperature", 0.0)

    output, latency, length, success = run_llm_experiment(
        model_name="gpt-4o-mini",
        temperature=0.0,
        prompt="Explain agentic AI in one sentence."
    )

    mlflow.log_metric("latency", latency)
    mlflow.log_metric("output_length", length)
    mlflow.log_metric("success", success)

    print("Run 1 Output:\n", output)

# Run 2 — High temperature (expressive)
with mlflow.start_run(run_name="higher_temperature_run"):
    mlflow.log_param("model", "gpt-4o-mini")
    mlflow.log_param("temperature", 0.7)

    output, latency, length, success = run_llm_experiment(
        model_name="gpt-4o-mini",
        temperature=0.7,
        prompt="Explain agentic AI in one sentence."
    )

    mlflow.log_metric("latency", latency)
    mlflow.log_metric("output_length", length)
    mlflow.log_metric("success", success)

    print("Run 2 Output:\n", output)

Step 5 — Launch the MLflow UI

# Access the MLflow UI at http://localhost:5000
!mlflow ui

Step 6 — Identify and restore the best run

runs = mlflow.search_runs(
    experiment_names=["llm_experiment_tracking_demo"]
)

best_run = runs.sort_values(
    by=["metrics.success", "metrics.latency"],
    ascending=[False, True]
).iloc[0]

print("Best Run ID:", best_run.run_id)
print("Best Run Params:", best_run.filter(like="params."))

Experiment tracking flow:

flowchart TD
    JANE[Jane — ML Engineer\nAmazing Company] --> QUESTION{Which config\nto deploy?}
    QUESTION --> R1[Run 1\ntemperature=0.0\ndeterministic]
    QUESTION --> R2[Run 2\ntemperature=0.7\nexpressive]
    R1 --> LOG1[MLflow log\nparams · latency · length · success]
    R2 --> LOG2[MLflow log\nparams · latency · length · success]
    LOG1 --> MLFLOW[MLflow Experiment Store]
    LOG2 --> MLFLOW
    MLFLOW --> UI[MLflow UI\nhttp://localhost:5000\nVisual comparison]
    MLFLOW --> QUERY[mlflow.search_runs\nSort by success DESC, latency ASC]
    QUERY --> BEST[Best run automatically selected\nnot by intuition]
    BEST --> PROD[Production deployment\nwith reproducible evidence]

Conclusion: Experiment tracking enables iteration, comparison, and rollback with confidence. Deployment decisions are guided by data, not intuition.

Summary and Key Points

mindmap
  root((LLMOps))
    Evaluation
      Automated metrics\nBLEU · ROUGE · BERTScore
      LLM-as-a-judge
      Multidimensional evaluation\nAccuracy · Relevance · Coherence
      Multimodal\nVision · Documents · Cross-modal
    Observability
      Structured logging\nPrompts · Outputs · Latency · Tokens · Costs
      Minimal schemas
      PII compliance
      Strategic sampling
    Automated testing
      Testing pyramid
      Mocked unit tests
      Integration tests
      Property-based testing
      CI/CD gates
    Monitoring
      Drift detection
      Real-time dashboards
      Canary testing
      MLflow experiment tracking
      Automatic rollback

Production Readiness Checklist

Search Terms

llmops · evaluation · observability · quality · model · governance · artificial · intelligence · generative · ai · testing · metrics · genai · test · integration · llm · automated · detection · experiment · logging · multimodal · strategies · tracking · components