Fine-tuning and Customizing LLMs

Complete course on specializing and customizing large language models (LLMs).

Table of Contents

1. Module 1 — Specializing Models for Better Results
   • 1.1 LLMs: Powerful but Imperfect
   • 1.2 From General Model to Specialized Model
   • 1.3 Training Strategies: RLHF, PEFT vs Full Fine-tuning
   • 1.4 Evals API: Installation and Configuration
   • 1.5 Evals API: Working with Evals
   • 1.6 Building and Uploading Training Datasets
   • 1.7 Fine-tuning OpenAI Models
   • 1.8 Testing and Evaluating Fine-tuned Models
2. Module 2 — Standardizing LLM Outputs for Reliability and Consistency
   • 2.1 Understanding the OpenAI Harmony Renderer
   • 2.2 Structuring and Standardizing Outputs
   • 2.3 Demo: Customer Support with Training Data
3. Module 3 — Evaluating Fine-tuning Performance for Transparency
   • 3.1 Comparing and Evaluating LLM Outputs and Performance
   • 3.2 Error Analysis and Inconsistency Detection (Sentry)
   • 3.3 Quality Audits, Dataset Refinement, and Hyperparameters (LangSmith)
4. Overall Project Architecture
5. Quick Command Reference

Module 1 — Specializing Models for Better Results

1.1 LLMs: Powerful but Imperfect

Language models (LLMs) can automate repetitive tasks, solve complex questions, and even generate complete projects. Despite this, they have important limitations:

Why Use Fine-tuning?

mindmap
  root((Fine-tuning))
    Improve accuracy
      More reliable responses
      Fewer hallucinations
    Specialize a domain
      Medicine
      E-commerce
      Legal
    Adapt the style
      Brand tone
      Response format
      Business vocabulary

1.2 From General Model to Specialized Model

Education Analogy

flowchart LR
    A["🎓 Bachelor's Degree\n(General Education)\n= Base Model\npre-trained on\ninternet data"] -->|"Fine-tuning"| B["🎓 Master's Degree\n(Specialization)\n= Fine-tuned Model\nspecialized in\na specific domain"]
    
    style A fill:#4a90d9,color:#fff
    style B fill:#27ae60,color:#fff

Medicine Analogy

flowchart LR
    A["🏥 Medical School\n(General Practitioner)\n= Base Model"] --> B["🦷 Dentist"]
    A --> C["🔬 Surgeon"]
    A --> D["👶 Pediatrician"]
    
    style A fill:#4a90d9,color:#fff
    style B fill:#e74c3c,color:#fff
    style C fill:#e74c3c,color:#fff
    style D fill:#e74c3c,color:#fff

Limitations of Generalist LLMs

• ChatGPT does not know the current time ("I don't have access to a live clock")
• Each model has a knowledge cutoff date (e.g. GPT-3.5 Turbo = September 2021)
• The context window varies by model (smaller = less processing capacity)

Concrete example: For the fictional e-commerce assistant AI-Genius Shopper, a generalist model would not know how to recommend seasonal products or adapt its tone to the brand’s style without fine-tuning.

1.3 Training Strategies: RLHF, PEFT vs Full Fine-tuning

flowchart TD
    A["🧠 Base Model\n(Pre-trained model)"] --> B{Training\nstrategy}
    
    B --> C["RLHF\nReinforcement Learning\nfrom Human Feedback"]
    B --> D["PEFT\nParameter Efficient\nFine-tuning"]
    B --> E["RFT\nReinforcement\nFine-tuning"]
    B --> F["RAG\nRetrieval Augmented\nGeneration"]
    
    C --> G["Provide good response\nexamples. The model\nlearns to recognize\na good response."]
    D --> H["Adjust a small number\nof parameters to\nspecialize the model."]
    E --> I["Use graders\n(evaluation algorithms)\nto score outputs\nand adjust weights."]
    F --> J["Provide context\nto the pre-trained model\nfor more precise\nand current outputs."]

    style A fill:#4a90d9,color:#fff
    style B fill:#f39c12,color:#fff
    style C fill:#9b59b6,color:#fff
    style D fill:#9b59b6,color:#fff
    style E fill:#9b59b6,color:#fff
    style F fill:#9b59b6,color:#fff

Strategy Comparison

1.4 Evals API: Installation and Configuration

Prerequisites

python-dotenv==1.1.0
rich>=13.5.0
colorama==0.4.6
openai
pandas

Environment Setup

macOS / Linux

# Create the virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip3 install -r requirements.txt

# Configure the API key
echo 'export OPENAI_API_KEY="your-secret-key"' >> ~/.bashrc
source ~/.bashrc

Windows

# Create the virtual environment
python -m venv .venv
.venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

.env file

OPENAI_API_KEY="YOUR_OPENAI_API_KEY"

1.5 Evals API: Working with Evals

Complete Evals Workflow

sequenceDiagram
    participant Dev as Developer
    participant API as OpenAI API
    participant Eval as Evals Engine
    participant File as JSONL File

    Dev->>API: 1. Create a task (Chat Completions)
    API-->>Dev: Response generated

    Dev->>Eval: 2. Create an Eval object
    Note over Dev,Eval: Define schema and test criteria
    Eval-->>Dev: eval_id

    Dev->>File: 3. Prepare the data file (JSONL)
    Dev->>API: Upload the file
    API-->>Dev: file_id

    Dev->>Eval: 4. Create an Eval Run
    Note over Dev,Eval: Associate eval_id + file_id + model
    Eval-->>Dev: run_id

    Dev->>Eval: 5. Analyze results
    Eval-->>Dev: pass/fail per example

Sample Evaluation Dataset (outputs.jsonl)

{ "item": { "request": "I want to track my order #12345", "correct_label": "Tracking Orders" } }
{ "item": { "request": "What size should I pick for this jacket if my chest is 38 inches?", "correct_label": "Size Guide" } }
{ "item": { "request": "I received the wrong color. How can I get this fixed?", "correct_label": "Customer Service" } }
{ "item": { "request": "Do you have gift options for birthdays under $50?", "correct_label": "Other" } }

Complete Code — Evals API (Module 1.4)

from dotenv import load_dotenv
from openai import OpenAI
from colorama import Fore
from rich.console import Console
from rich.pretty import Pretty
from rich import print

load_dotenv()

client = OpenAI()
console = Console()
console.rule("[bold green]Running Eval for User Request Categorization[/bold green]")

taskInstructions = """
You are an expert in categorizing user requests to better dispatch them to the correct team. 
Given the request, categorize using the same words from this set: 
'Tracking Orders', 'Size Guide', 'Customer Service', or 'Other'.
"""

userRequest = "I want to track my order"

# 1. Create a task
response = client.responses.create(
    model="gpt-4.1",
    input=[
        {"role": "developer", "content": taskInstructions},
        {"role": "user", "content": userRequest},
    ],
)
print(response.output_text)

# 2. Create an Eval object with test criteria
evalObj = client.evals.create(
    name="User Request Categorization",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "request": {"type": "string"},
                "correct_label": {"type": "string"},
            },
            "required": ["request", "correct_label"],
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "Match output to human label",
            "input": "{{ sample.output_text }}",
            "operation": "eq",
            "reference": "{{ item.correct_label }}",
        }
    ],
)

evalId = evalObj.id

# 3. Upload the data file
uploadedFile = client.files.create(
    file=open("outputs.jsonl", "rb"),
    purpose="evals"
)
fileId = uploadedFile.id

# 4. Create an Eval Run
evalRun = client.evals.runs.create(
    evalId,
    name="Categorization text run",
    data_source={
        "type": "responses",
        "model": "gpt-4.1",
        "input_messages": {
            "type": "template",
            "template": [
                {"role": "developer", "content": taskInstructions},
                {"role": "user", "content": "{{ item.request }}"},
            ],
        },
        "source": {"type": "file_id", "id": fileId},
    },
)

# 5. Retrieve and analyze results
results = client.evals.runs.retrieve(eval_id=evalId, run_id=evalRun.id)
while results.status != "completed":
    results = client.evals.runs.retrieve(eval_id=evalId, run_id=evalRun.id)
    console.print(results)

1.6 Building and Uploading Training Datasets

AI-Genius Shopper Project Architecture

01/demos/exercise-files/1.6/end/
├── main.py              # Main entry point
├── utils.py             # Data preparation + onboarding questions
├── function_calling.py  # OpenAI function calls
├── requirements.txt
└── data/
    ├── dataset.csv              # Raw data
    ├── dataset.jsonl            # Training data (80%)
    └── dataset_validation.jsonl # Validation data (20%)

CSV Dataset Format

Training data examples:

"I'm a 25-year-old woman looking for stylish winter boots"
→ response_type: seasonal_outfit_recommendation
→ "Great choice for winter! For a 25-year-old woman, I'd recommend 
   Insulated Ankle Boots for everyday wear, Knee-High Leather Boots 
   for a more polished look..."

Data Preparation Flow

flowchart TD
    A["📄 dataset.csv\n(raw data)"] --> B["loadAndSplitDataset()\nCleaning + splitting"]
    B --> C["Training DataFrame\n80% of data"]
    B --> D["Validation DataFrame\n20% of data"]
    C --> E["prepareConversationRecord()\nConversion to JSONL format"]
    D --> F["prepareConversationRecord()\nConversion to JSONL format"]
    E --> G["exportToJsonl()\ndataset.jsonl"]
    F --> H["exportToJsonl()\ndataset_validation.jsonl"]

    style A fill:#f39c12,color:#fff
    style G fill:#27ae60,color:#fff
    style H fill:#27ae60,color:#fff

Code — utils.py: Data Preparation

import pandas as pd
from colorama import Fore
import json

systemInstruction = (
    "You are an intelligent shopping assistant. "
    "You help customers find products, track orders, get personalized recommendations, "
    "and provide excellent customer service. You're knowledgeable about our products, "
    "policies, and always aim to create a personalized shopping experience. "
    "Be helpful, friendly, and professional."
)

def loadAndSplitDataset(csvPath: str = "data/dataset.csv"):
    """Load the dataset, clean it, and split into training/validation."""
    shoppingDf = pd.read_csv(csvPath)
    shoppingDf = shoppingDf.dropna(subset=["customer_query", "assistant_response"])
    
    # 80% / 20% split
    trainSize = int(len(shoppingDf) * 0.8)
    trainingDf   = shoppingDf.iloc[:trainSize]
    validationDf = shoppingDf.iloc[trainSize:]
    return trainingDf, validationDf

def prepareConversationRecord(row):
    """Convert CSV data to OpenAI JSONL format."""
    return {
        "messages": [
            {"role": "system",    "content": systemInstruction},
            {"role": "user",      "content": str(row["customer_query"])},
            {"role": "assistant", "content": str(row["assistant_response"])},
        ]
    }

def exportToJsonl(records, filename: str):
    """Write records to a JSONL file."""
    with open(filename, "w", encoding="utf-8") as f:
        for record in records:
            f.write(json.dumps(record, ensure_ascii=False) + "\n")

def main():
    trainingDf, validationDf = loadAndSplitDataset()
    trainingData   = trainingDf.apply(prepareConversationRecord, axis=1).tolist()
    validationData = validationDf.apply(prepareConversationRecord, axis=1).tolist()
    
    exportToJsonl(trainingData,   "data/dataset.jsonl")
    exportToJsonl(validationData, "data/dataset_validation.jsonl")
    print(f"✅ {len(trainingData)} training examples, {len(validationData)} validation examples")

Expected JSONL Format for Fine-tuning

{
  "messages": [
    {"role": "system",    "content": "You are an intelligent shopping assistant..."},
    {"role": "user",      "content": "I'm a 25-year-old woman looking for stylish winter boots"},
    {"role": "assistant", "content": "Great choice for winter! For a 25-year-old woman..."}
  ]
}

Important: It is recommended to provide at least 10 good examples in JSONL format to start an effective fine-tuning.

Code — function_calling.py: Function Call Orchestration

from openai import OpenAI
import json

client = OpenAI()

# Function definition for OpenAI
functions = [
    {
        "name": "generate_finetune_instruction",
        "description": "Generate instructions for the fine-tuned AI-Genius Shopper model.",
        "parameters": {
            "type": "object",
            "properties": {
                "age": {"type": "integer", "description": "User age"},
                "gender": {
                    "type": "string",
                    "enum": ["Male", "Female", "Non-binary", "Prefer not to say"]
                },
                "shopping_need": {
                    "type": "string",
                    "description": "What the user wants to shop for"
                }
            },
            "required": ["age", "gender", "shopping_need"]
        }
    }
]

def generate_finetune_instructions(userProfile):
    """Generate fine-tuning instructions based on the user profile."""
    return {
        "instructions": f"Generate personalized shopping recommendations for a "
                        f"{userProfile['age']} year old {userProfile['gender']} "
                        f"looking for {userProfile['shopping_need']}."
    }

1.7 Fine-tuning OpenAI Models

Fine-tuning Flow

flowchart LR
    A["📊 Dataset CSV\n(raw data)"] -->|"utils.py\nprepare"| B["📄 dataset.jsonl\n(JSONL format)"]
    B -->|"API or Dashboard"| C["☁️ Upload\nthe file\n→ file_id"]
    C -->|"Create job"| D["⚙️ Fine-tuning Job\n→ ftjob-xxxxx"]
    D -->|"Training\nin progress..."| E["✅ Fine-tuned model\nft:gpt-4.1-nano-...:personal::xxxxx"]
    E -->|"Use"| F["🤖 Personalized\nresponses"]

    style A fill:#f39c12,color:#fff
    style B fill:#3498db,color:#fff
    style C fill:#9b59b6,color:#fff
    style D fill:#e74c3c,color:#fff
    style E fill:#27ae60,color:#fff
    style F fill:#27ae60,color:#fff

Uploading the Training File via curl

curl https://api.openai.com/v1/files \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F purpose="fine-tune" \
  -F file=@data/dataset.jsonl

Response:

{
  "object": "file",
  "id": "file-BGvvykXqwUSTi3d1P617yX",
  "purpose": "fine-tune",
  "filename": "dataset.jsonl",
  "bytes": 12265,
  "created_at": 1764344550,
  "status": "processed"
}

Creating a Fine-tuning Job via curl

curl https://api.openai.com/v1/fine_tuning/jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "training_file": "file-BGvvykXqwUSTi3d1P617yX",
    "model": "gpt-4.1-nano-2025-04-14"
  }'

Response:

{
  "object": "fine_tuning.job",
  "id": "ftjob-ws81xVV3Ao33mbiMjA8MLFeR",
  "model": "gpt-4.1-nano-2025-04-14",
  "status": "validating_files",
  "hyperparameters": {
    "n_epochs": "auto",
    "batch_size": "auto",
    "learning_rate_multiplier": "auto"
  },
  "method": {
    "type": "supervised"
  }
}

Note: You can also use the OpenAI Dashboard to create a fine-tuning job via the graphical interface. An email notification will be sent when the job completes with status succeeded.

1.8 Testing and Evaluating Fine-tuned Models

Checking the Status of a Fine-tuning Job

curl https://api.openai.com/v1/fine_tuning/jobs/[FT_JOB_ID] \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Using the Fine-tuned Model

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="ft:gpt-4.1-nano-2025-04-14:personal::Cj3alpVy",  # Replace with your ID
    messages=[
        {"role": "system", "content": systemInstruction},
        {"role": "user",   "content": "Generate personalized shopping recommendations "
                                      "for a 29 year old Female looking for Seasonal Outfits."}
    ]
)
print(response.choices[0].message.content)

User Onboarding Questions

def onboarding_questions():
    """Collect user profile via interactive questions."""
    # Age
    age = int(input("👤 Enter your age: "))
    
    # Gender
    genderOptions = {"1": "Male", "2": "Female", "3": "Non-binary", "4": "Prefer not to say"}
    gender = genderOptions[input("Choose gender [1-4]: ")]
    
    # Shopping need
    needOptions = {
        "1": "Clothing", "2": "Shoes", "3": "Accessories",
        "4": "Seasonal Outfits", "5": "Gifts", "6": "Electronics",
        "7": "Home & Living", "8": "Other"
    }
    shoppingNeed = needOptions[input("What are you looking for? [1-8]: ")]
    
    return {"age": age, "gender": gender, "shopping_need": shoppingNeed}

Module 2 — Standardizing LLM Outputs for Reliability and Consistency

2.1 Understanding the OpenAI Harmony Renderer

The OpenAI Harmony Renderer is a formatting and rendering system that OpenAI uses to transform structured model outputs into readable, well-styled text. It is used with gpt-oss models (Open Source Style).

Harmony Renderer Architecture

flowchart TD
    A["System Message\n(Harmony metadata)"] --> D["Conversation Object"]
    B["Developer Message\n(system instructions)"] --> D
    C["User Message\n(user input)"] --> D
    D -->|"render_conversation_for_completion()"| E["Token IDs\n(tokenized format)"]
    E -->|"encoding.decode()"| F["Formatted text\nfor the model"]
    F --> G["🤖 gpt-oss model\nCompletion"]

    style A fill:#3498db,color:#fff
    style B fill:#9b59b6,color:#fff
    style C fill:#27ae60,color:#fff
    style D fill:#f39c12,color:#fff
    style G fill:#e74c3c,color:#fff

Roles in the Harmony Format

Installation

pip install openai-harmony

Code — harmony_format.py: Training Data Formatting

import json
from typing import Dict, Any

from openai_harmony import (
    Conversation,
    HarmonyEncodingName,
    Message,
    Role,
    SystemContent,
    DeveloperContent,
    load_harmony_encoding,
    ReasoningEffort,
)

# Load the Harmony encoder compatible with gpt-oss models
encoding = load_harmony_encoding(HarmonyEncodingName.HARMONY_GPT_OSS)

def make_training_example(
    system_message: str,
    user_payload: Dict[str, Any],
    assistant_reply: str,
) -> Dict[str, Any]:
    """
    Creates a training example rendered by Harmony.
    Use ONLY for:
    - Custom trainers
    - Pre-tokenized datasets
    - Text-to-text fine-tuning
    """
    # System metadata (Harmony only)
    sysContent = (
        SystemContent.new()
        .with_reasoning_effort(ReasoningEffort.MEDIUM)
    )

    # Developer instructions (your actual system prompt)
    devContent = (
        DeveloperContent.new()
        .with_instructions(system_message.strip())
    )

    convo = Conversation.from_messages([
        Message.from_role_and_content(Role.SYSTEM, sysContent),
        Message.from_role_and_content(Role.DEVELOPER, devContent),
        Message.from_role_and_content(
            Role.USER,
            json.dumps(user_payload, ensure_ascii=False),
        ),
    ])

    # Render the prompt up to the assistant turn
    promptTokenIds = encoding.render_conversation_for_completion(
        convo,
        Role.ASSISTANT,
    )
    promptText = encoding.decode(promptTokenIds)

    return {
        "prompt": promptText,
        "completion": assistant_reply.strip(),
    }

def harmony_preview_prompt(system_message: str, user_text: str) -> str:
    """
    Display a preview of the Harmony prompt for inspection/debugging.
    Does NOT call any model. Shows only Harmony serialization.
    """
    devContent = (
        DeveloperContent.new()
        .with_instructions(system_message.strip())
    )

    convo = Conversation.from_messages([
        Message.from_role_and_content(Role.SYSTEM, SystemContent.new()),
        Message.from_role_and_content(Role.DEVELOPER, devContent),
        Message.from_role_and_content(Role.USER, user_text),
    ])

    tokenIds = encoding.render_conversation_for_completion(convo, Role.ASSISTANT)
    return encoding.decode(tokenIds)

2.2 Structuring and Standardizing Outputs

Structured System Message for AI-Genius Shopper

🛍️ You are AI-Genius Shopper, a personalized shopping assistant.

━━━━━━━━━━━━━━━━━━━━━━
🎯 YOUR GOALS
━━━━━━━━━━━━━━━━━━━━━━
• Quickly understand the user's profile and intent
• Recommend 3–5 actionable product categories
• Suggest practical outfit ideas when relevant
• Keep responses friendly, helpful, and concise

━━━━━━━━━━━━━━━━━━━━━━
📦 RESPONSE FORMAT (STRICT)
━━━━━━━━━━━━━━━━━━━━━━
🧾 Summary
One short sentence summarizing the recommendation focus.

🛒 Product Suggestions
• Category – short reason why it fits (3–5 total)

👗 Outfit Ideas (if relevant)
• Simple, wearable outfit idea

❓ Clarifying Question (optional)
One brief question if more detail is needed.

Preparation Flow with Harmony (Module 2)

sequenceDiagram
    participant CSV as dataset.csv
    participant Utils as utils.py
    participant Harmony as harmony_format.py
    participant JSONL as dataset.jsonl

    CSV->>Utils: loadAndSplitDataset()
    Utils->>Utils: Cleaning + 80/20 split
    Utils->>Harmony: make_training_example(system_msg, user_payload, reply)
    Harmony->>Harmony: Build Conversation (SYSTEM + DEVELOPER + USER)
    Harmony->>Harmony: render_conversation_for_completion()
    Harmony->>Harmony: encoding.decode()
    Harmony-->>Utils: {"prompt": "...", "completion": "..."}
    Utils->>JSONL: exportToJsonl()

Code — utils.py (Module 2): Harmony Integration

from harmony_format import make_training_example

def prepareConversationRecord(row):
    """Convert CSV data to Harmony format for fine-tuning."""
    userPayload = {
        "age": 24,
        "gender": "Female",
        "shopping_need": "Clothing"
    }

    return make_training_example(
        system_message=systemInstruction,
        user_payload=userPayload,
        assistant_reply=str(row["assistant_response"])
    )

2.3 Demo: Customer Support with Training Data

Project Architecture (Module 2 - Final)

02/demos/exercise-files/final/
├── main.py              # Main orchestrator
├── utils.py             # Data preparation + onboarding
├── function_calling.py  # OpenAI function calls
├── harmony_format.py    # Harmony formatting
├── system.txt           # Detailed system message
└── data/
    ├── dataset.csv
    ├── dataset.jsonl            # Training (with Harmony)
    └── dataset_validation.jsonl

Main Application Flow (Module 2)

flowchart TD
    A["🚀 start()"] --> B{Menu}
    B -->|"[1] Shop"| C["run()"]
    B -->|"[2] Exit"| Z["👋 Goodbye"]
    
    C --> D["onboarding_questions()\nAge + Gender + Need"]
    D --> E["generate(user_payload)\nFunction Calling GPT-4o-mini"]
    E --> F["generate_finetune_instructions()\nBuild the instructions"]
    F --> G["use_finetuned_model(instructions)\nft:gpt-4.1-nano:personal::xxxxx"]
    G --> H["harmony_preview_prompt()\nDisplay Harmony prompt"]
    H --> I["🛍️ Display recommendations"]
    I --> J{Continue?}
    J -->|"yes"| G
    J -->|"no/menu"| B
    J -->|"exit"| Z

    style A fill:#3498db,color:#fff
    style Z fill:#e74c3c,color:#fff
    style G fill:#27ae60,color:#fff

Module 3 — Evaluating Fine-tuning Performance for Transparency

3.1 Comparing and Evaluating LLM Outputs and Performance

Evaluation is critical for transparency and continuous improvement. It enables you to:

• Build reliable and transparent AI systems
• Gain insights into best practices
• Verify that the model behaves as expected

Project Architecture (Module 3 - Final)

03/demos/exercise-files/final/
├── main.py              # Main orchestrator
├── evaluation.py        # Complete evaluation pipeline
├── utils.py             # Data preparation
├── function_calling.py  # Function calls
├── harmony_format.py    # Harmony formatting
└── data/
    ├── dataset.csv
    ├── dataset.jsonl
    └── dataset_validation.jsonl

Classification Categories for Evaluation

budget_fitness_recommendation
professional_outfit_planning
minimalist_style_recommendation
event_outfit_recommendation
seasonal_outfit_recommendation
wardrobe_refresh
lifestyle_casual_recommendation
travel_clothing_recommendation
trend_based_recommendation
remote_work_lifestyle
career_transition_style

Complete Code — evaluation.py (Module 3)

from dotenv import load_dotenv
from openai import OpenAI
from colorama import Fore
from rich.console import Console
from utils import main as build_dataset
import sentry_sdk
import os

load_dotenv()

client = OpenAI()
console = Console()

# Configure Sentry for error monitoring
sentry_sdk.init(
    dsn=os.getenv("SENTRY_DSN"),
    send_default_pii=True,
)

# Build the training dataset
build_dataset()

# --------------------------------------------------
# 1. TASK INSTRUCTIONS
# --------------------------------------------------
classificationInstructions = """
You are an expert at classifying shopping-related user queries.
Given a customer query, classify it using ONE of the following labels:
- budget_fitness_recommendation
- professional_outfit_planning
- minimalist_style_recommendation
- event_outfit_recommendation
- seasonal_outfit_recommendation
- wardrobe_refresh
- lifestyle_casual_recommendation
- travel_clothing_recommendation
- trend_based_recommendation
- remote_work_lifestyle
- career_transition_style

Return ONLY the label.
"""

sampleRequest = "I'm a 25-year-old woman looking for stylish winter boots"

# --------------------------------------------------
# 2. QUICK CHECK
# --------------------------------------------------
sampleResponse = client.responses.create(
    model="gpt-4.1",
    input=[
        {"role": "developer", "content": classificationInstructions},
        {"role": "user",      "content": sampleRequest},
    ],
)
print(f"Sample Response: {Fore.GREEN}{sampleResponse.output_text}{Fore.RESET}")

# --------------------------------------------------
# 3. CREATE THE EVAL OBJECT
# --------------------------------------------------
evalObj = client.evals.create(
    name="Shopping Recommendation Classification Eval",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {
                "customer_query": {"type": "string"},
                "response_type":  {"type": "string"},
            },
            "required": ["customer_query", "response_type"],
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "Match response_type label",
            "input": "{{ sample.output_text }}",
            "operation": "eq",
            "reference": "{{ item.response_type }}",
        }
    ],
)
evalId = evalObj.id

# --------------------------------------------------
# 4. UPLOAD THE DATASET (JSONL)
# --------------------------------------------------
uploadedFile = client.files.create(
    file=open("data/dataset.jsonl", "rb"),
    purpose="evals",
)
fileId = uploadedFile.id

# --------------------------------------------------
# 5. CREATE THE EVAL RUN
# --------------------------------------------------
evalRun = client.evals.runs.create(
    eval_id=evalId,
    name="Shopping Dataset Classification Run",
    data_source={
        "type": "responses",
        "model": "gpt-4.1",
        "input_messages": {
            "type": "template",
            "template": [
                {"role": "developer", "content": classificationInstructions},
                {"role": "user",      "content": "{{ item.customer_query }}"},
            ],
        },
        "source": {"type": "file_id", "id": fileId},
    },
)
runId = evalRun.id

# --------------------------------------------------
# 6. WAIT AND ANALYZE RESULTS
# --------------------------------------------------
console.print("⏳ Waiting for eval run to complete...")
results = client.evals.runs.retrieve(eval_id=evalId, run_id=runId)
while results.status != "completed":
    results = client.evals.runs.retrieve(eval_id=evalId, run_id=runId)
    console.print(results.status)

console.rule("[bold green]Eval Completed[/bold green]")
console.print(f"Results: {Fore.GREEN}{results}{Fore.RESET}")

Complete Evaluation Pipeline (Module 3)

flowchart LR
    A["📊 Dataset\n(dataset.jsonl)"] --> B["Build\nEval Object"]
    B --> C["Upload\nJSONL file"]
    C --> D["Create Eval Run\nwith model gpt-4.1"]
    D --> E{Status\n== completed?}
    E -->|No| F["Waiting...\nPolling"]
    F --> E
    E -->|Yes| G["✅ Analyze\nresults"]
    G --> H{"Pass / Fail\nper example"}
    H -->|"Pass"| I["✅ Correct label"]
    H -->|"Fail"| J["❌ Incorrect label\n→ Refine"]

    style A fill:#f39c12,color:#fff
    style G fill:#27ae60,color:#fff
    style I fill:#27ae60,color:#fff
    style J fill:#e74c3c,color:#fff

3.2 Error Analysis and Inconsistency Detection (Sentry)

Sentry.io is an error monitoring solution that quickly identifies problems in your applications, including errors from LLM API calls.

Why Use Sentry?

mindmap
  root((Sentry))
    Real-time monitoring
      Automatic exception capture
      Detailed stack traces
    Rapid identification
      Precise error location
      Full context
    LLM integration
      API call errors
      Timeouts and rate limits
      Unexpected responses
    Dashboard
      Error history
      Trends and frequencies

Installing and Configuring Sentry

pip install sentry-sdk

Integration in the Project

import sentry_sdk
import os
from dotenv import load_dotenv

load_dotenv()

# Add at the beginning of each module you want to monitor
sentry_sdk.init(
    dsn=os.getenv("SENTRY_DSN"),
    send_default_pii=True,
)

.env file:

OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
SENTRY_DSN="YOUR_SENTRY_DSN"

Example Error Detected by Sentry

Sentry will show in the dashboard:

• The exact line where the error occurred
• The full error message (e.g. AttributeError: 'Client' object has no attribute 'creat' — did you mean 'create'?)
• The number of tokens used per call
• The full execution context

Monitoring Flow with Sentry

Python Application
       │
       ▼
  sentry_sdk.init()
       │
       ▼
  OpenAI API call
       │
    ┌──┴──┐
    │     │
 Success  Error
    │        │
    ▼        ▼
 Continue  Sentry captures
           the exception
               │
               ▼
          Sentry Dashboard
          (error visible
           with context)

3.3 Quality Audits, Dataset Refinement, and Hyperparameters (LangSmith)

LangSmith (provided by LangChain) allows you to observe model outputs in real time to verify LLMs behave as expected and to refine datasets and hyperparameters.

Installing LangSmith

pip install langsmith
pip install openai-agents  # or your agents SDK

.env file:

OPENAI_API_KEY="YOUR_OPENAI_API_KEY"
LANGCHAIN_API_KEY="YOUR_LANGSMITH_API_KEY"
LANGCHAIN_TRACING_V2=true
LANGCHAIN_PROJECT="fine-tuning-project"

Configuration and Usage

from langsmith import traceable

# Decorate the functions you want to trace
@traceable
def generate_completion(userInput: str) -> str:
    response = client.responses.create(
        model="gpt-4.1",
        input=[{"role": "user", "content": userInput}]
    )
    return response.output_text

@traceable
def use_finetuned_model(modelInstructions: str) -> str:
    response = client.chat.completions.create(
        model="ft:gpt-4.1-nano-2025-04-14:personal::Cj3alpVy",
        messages=[
            {"role": "system", "content": systemInstruction},
            {"role": "user",   "content": modelInstructions}
        ]
    )
    return response.choices[0].message.content

What LangSmith Lets You Observe

Complete Evaluation and Monitoring Ecosystem

flowchart TD
    A["🤖 LLM Application\n(AI-Genius Shopper)"] --> B["Evals API\n(OpenAI)"]
    A --> C["Sentry.io"]
    A --> D["LangSmith\n(LangChain)"]

    B --> E["Test outputs\nagainst criteria\npass/fail"]
    C --> F["Capture errors\nin real time\nstack traces"]
    D --> G["Observe traces\nrefine datasets\nand hyperparameters"]

    E --> H["✅ Reliable\nand transparent system"]
    F --> H
    G --> H

    style A fill:#4a90d9,color:#fff
    style H fill:#27ae60,color:#fff
    style B fill:#9b59b6,color:#fff
    style C fill:#e74c3c,color:#fff
    style D fill:#f39c12,color:#fff

Overall Project Architecture

File Structure

fine-tuning-customizing-llms/
│
├── 01/ ─ Module 1: Specialization
│   └── demos/exercise-files/
│       ├── 1.4/ ─ Evals API
│       │   ├── start/          ← Starting point
│       │   └── end/            ← Complete solution
│       │       ├── main.py     ← Evals pipeline
│       │       └── outputs.jsonl
│       └── 1.6/ ─ Fine-tuning
│           ├── start/
│           └── end/
│               ├── main.py        ← Complete application
│               ├── utils.py       ← Data preparation
│               ├── function_calling.py
│               └── data/
│                   ├── dataset.csv
│                   ├── dataset.jsonl
│                   └── dataset_validation.jsonl
│
├── 02/ ─ Module 2: Harmony Renderer
│   └── demos/exercise-files/
│       ├── start/
│       └── final/
│           ├── main.py
│           ├── utils.py
│           ├── harmony_format.py  ← New!
│           ├── function_calling.py
│           └── data/
│
└── 03/ ─ Module 3: Evaluation and Transparency
    └── demos/exercise-files/
        ├── start/
        └── final/
            ├── main.py
            ├── evaluation.py      ← New!
            ├── utils.py
            ├── harmony_format.py
            └── data/

Full Journey Overview

flowchart TD
    subgraph M1 ["Module 1 — Specialization"]
        A["CSV Data\n(20 examples)"] --> B["utils.py\nJSONL Preparation"]
        B --> C["Evals API\nTest + Validation"]
        C --> D["Fine-tuning Job\ngpt-4.1-nano"]
        D --> E["Fine-tuned model\nft:gpt-4.1-nano:..."]
    end

    subgraph M2 ["Module 2 — Standardization"]
        E --> F["harmony_format.py\nHarmony Formatting"]
        F --> G["Standardized outputs\nStrict format"]
    end

    subgraph M3 ["Module 3 — Evaluation"]
        G --> H["evaluation.py\nEvals API"]
        G --> I["Sentry.io\nError monitoring"]
        G --> J["LangSmith\nObservability"]
        H --> K["✅ Reliable\nand transparent system"]
        I --> K
        J --> K
    end

    style M1 fill:#d5e8d4,stroke:#82b366
    style M2 fill:#dae8fc,stroke:#6c8ebf
    style M3 fill:#ffe6cc,stroke:#d6b656
    style K fill:#27ae60,color:#fff

Quick Command Reference

Environment Setup

# macOS / Linux
python3 -m venv .venv
source .venv/bin/activate
pip3 install -r requirements.txt

# Windows
python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

# Deactivate environment
deactivate

Running the Application

# macOS / Linux
python3 main.py

# Windows
python main.py

OpenAI API — Essential curl Commands

# Upload a fine-tuning file
curl https://api.openai.com/v1/files \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F purpose="fine-tune" \
  -F file=@data/dataset.jsonl

# Create a fine-tuning job
curl https://api.openai.com/v1/fine_tuning/jobs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{"training_file": "FILE_ID", "model": "gpt-4.1-nano-2025-04-14"}'

# Check job status
curl https://api.openai.com/v1/fine_tuning/jobs/FTJOB_ID \
  -H "Authorization: Bearer $OPENAI_API_KEY"

Python Dependencies by Module

Useful resources:

Search Terms

fine-tuning · customizing · llms · llm · application · development · artificial · intelligence · generative · ai · architecture · flow · harmony · sentry · api · data · evals · evaluation · curl · dataset · evaluating · format · langsmith · models