Domain-specific LLM Agents

Level: Intermediate to advanced

Table of Contents

• Course Overview
• Technology stack used
• Developing and Managing Domain specific LLM Agents
• Lesson 1 — Introduction to LLM Agents
• What is an LLM?
• LLM generalists vs LLM domain-specific
• Key differences: generalist vs domain-specific
• Advantages of LLM domain-specific
• Respective limits
• Demo — Generalist vs domain-specific LLM comparison
• Lesson 2 — Curation and preparation of domain-specific datasets
• Efficient data collection
• Raw data preparation
• Impact of high-quality data
• Demo — Data preprocessing pipeline
• Lesson 3 — Fine-tuning techniques for domain-specific applications
• Fine-tuning overview
• 4-step fine-tuning workflow
• Fine-tuning methods
• Full Supervised Fine-tuning (SFT)
• Low-Rank Adaptation (LoRA)
• Parameter-Efficient Fine-Tuning (PEFT)
• Demo — Construction of a domain-specific LLM for fitness
• Lesson 4 — Compliance, confidentiality and continuous improvement
• Regulatory frameworks
• Data protection policies
• AI Model Governance
• Continuous learning and feedback loops
• Demo — Privacy-Preserving Feedback Loop
• Synthesis and key concepts
• Glossary

1. Course Overview

This course explores LLM domain-specific agents (Large Language Models specialized by domain). It covers four main areas:

1. What domain-specific LLMs are — how they differ from general models and why they are essential for specialized applications.
2. Curation and preparation of domain-specific datasets — a key factor in the success of specialized LLMs is data quality. Strategies for curating, cleaning, and preparing datasets tailored to specific industries and use cases are examined in detail.
3. Fine-tuning techniques for domain-specific applications — once the dataset is well prepared, you can fine-tune a language model to improve its performance in a specific domain. Different fine-tuning methods and optimization best practices are covered.
4. Compliance, privacy and continuous improvement — deploying a model in a specialized environment comes with challenges around privacy, governance and continuous improvement. Compliance requirements, bias mitigation and continuous learning strategies are examined.

2. Technology stack used

3. Developing and Managing Domain specific LLM Agents

Lesson 1 — Introduction to LLM Agents

What is an LLM?

Large Language Models (LLMs) are deep learning models designed to understand and generate text. They are trained on vast amounts of data to develop a generalized understanding of language.

At their heart, these models:

• Process natural language and generate human-like responses.
• Leverage deep learning architectures.
• Are pre-trained on massive datasets to predict and generate text.

Generalist LLM vs domain-specific LLM

There are two main categories of LLM:

┌─────────────────────────────────────────────────────────────────┐
│          LLM GÉNÉRALISTES          │     LLM DOMAIN-SPECIFIC     │
├────────────────────────────────────┼─────────────────────────────┤
│ Entraînés sur des datasets divers  │ Fine-tunés sur des données  │
│                                    │ spécifiques au domaine      │
│ Ex: GPT-4, Llama 3                 │ Ex: TSLAM-4B (télécoms)     │
│                                    │                             │
│ Très flexibles                     │ Précis et contextuellement  │
│                                    │ adaptés                     │
│ Peuvent manquer de précision       │ Réduisent les               │
│ sur des sujets très spécialisés    │ hallucinations              │
└────────────────────────────────────┴─────────────────────────────┘

General models (e.g. GPT-4, Llama 3):

• Trained on various datasets.
• Very flexible: suitable for various use cases such as conversational AI, content generation, general questions.
• May lack precision on very specialized subjects.

Domain-specific models:

• Fine-tuned to industry-specific data.
• Provide more precise and relevant answers in specific areas: legal, health, fitness, data center operations, finance, etc.
• Reduce hallucinations.
• Improve reliability in critical applications.

Key Differences: Generalist vs Domain-Specific

graph LR
    A[LLM Généraliste] --> B[Flexibilité]
    A --> C[Large compréhension]
    A --> D[Polyvalence]
    
    E[LLM Domain-Specific] --> F[Précision accrue]
    E --> G[Conscience contextuelle]
    E --> H[Conformité réglementaire]
    
    style A fill:#4a90d9,color:#fff
    style E fill:#e07b39,color:#fff

Advantages of domain-specific LLMs

Generalist LLMs struggle with niche accuracy and may generate wild or misleading information. Domain-specific LLMs solve this lack of depth and precision by being fine-tuned to industry-specific data, which improves:

• Accuracy: best answers in the field
• Relevance: contextually appropriate responses
• Compliance: compliance with sectoral regulations
• Decision-making efficiency: better decision support in specialized industries such as data center operations, finance, legal, medicine

Respective limits

Limits of generalist LLMs:

• Difficulty with accuracy in niches
• Risk of generating hallucinatory or misleading information

Limits of domain-specific LLMs:

• Require high quality curated datasets
• Require continuous updates to maintain performance
• Without continuous fine-tuning, risk becoming obsolete as domain knowledge evolves

Demo — General versus domain-specific LLM comparison

In this demo, two Python scripts are built and compared: one using a general LLM, the other a domain-specific LLM. By asking the same questions to both scripts, we observe how domain adaptation improves accuracy, reduces hallucinations, and improves practicability in specialized domains.

Script 1: General_Purpose_LLM.py — General model

# Étape 1 : Import de la bibliothèque transformers
# Fournit des modèles de langage pré-entraînés
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Étape 2 : Chargement du modèle Llama-2-7b-chat-hf depuis Hugging Face
# torch_dtype=torch.float16 optimise la mémoire et permet au modèle de tenir sur le GPU local
model_name = "meta-llama/Llama-2-7b-chat-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Étape 3 : Chargement du tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Étape 4 : Fonction generate_response
# Prend un prompt textuel, le tokenize, l'exécute via le modèle, retourne une réponse textuelle
def generate_response(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=200)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

# Étape 5 : Boucle interactive
# L'utilisateur entre des prompts; le modèle génère des réponses
# Saisir "quit" ou "exit" termine la boucle
while True:
    user_input = input("Entrez votre prompt (ou 'quit' pour quitter) : ")
    if user_input.lower() in ["quit", "exit"]:
        break
    print(generate_response(user_input))

Key points of the general script:

• Uses model Llama-2-7b-chat-hf from Hugging Face
• torch_dtype=torch.float16: optimizes memory and allows loading on local GPU
• The generate_response function: tokenize the prompt → execute the model → return a textual response
• Interactive loop allowing user to enter prompts

Script 2: LLM Domain-Specific (TSLAM-4B — Telecommunications)

# Étape 1 : Téléchargement des bibliothèques nécessaires
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# Étape 2 : Vérification de la disponibilité d'un GPU compatible CUDA
# Permet le déchargement du traitement du CPU vers le GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Étape 3 : Chargement du modèle domain-specific TSLAM-4B
# Modèle fine-tuné pour les réseaux informatiques / télécommunications
model_name = "TSLAM-4B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Étape 4 : Fonction de génération de réponse domain-specific
def generate_domain_response(prompt):
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    outputs = model.generate(**inputs, max_new_tokens=300)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

# Boucle interactive similaire au script généraliste
while True:
    user_input = input("Entrez votre prompt (ou 'quit' pour quitter) : ")
    if user_input.lower() in ["quit", "exit"]:
        break
    print(generate_domain_response(user_input))

Key points of the domain-specific script:

• Uses TSLAM-4B: fine-tuned model for telecommunications and computer networks
• Checking CUDA availability to offload processing from CPU to GPU
• Same interactive loop structure as the general script

Observed result: By testing the two scripts with the same questions, TSLAM-4B (fine-tuned for telecommunications) provides a much more detailed and precise answer than the general model Llama-2-7b-chat-hf.

Lesson 2 — Curation and preparation of domain-specific datasets

Domain-specific models are built with highly specialized and calibrated datasets specific to particular domains. To achieve this, the domain-specific model requires the curation and preparation of domain-specific data.

Efficient data collection

Efficient data collection is the foundation for training high-performance domain-specific language models.

flowchart TD
    A[Sources de données] --> B[Datasets ouverts et\nrepositories publics]
    A --> C["Données privées\n(intranets internes)"]
    B --> D[Données structurées\net non-structurées]
    C --> D
    D --> E[Assurer la diversité\ndes données]
    E --> F[Meilleure généralisation\naux scénarios réels]
    F --> G[Adaptabilité et robustesse\nde l'application]

Collection strategies:

1. Open datasets and public repositories: provide access to vast amounts of structured and unstructured data relevant to the targeted domain.
2. Data diversity: crucial to help the model generalize across various real-world scenarios, improving its adaptability and robustness in the intended application.
3. Internal private data: Organizations can also use their own data within their internal intranets.

Preparing raw data

Once collected, the raw data must undergo rigorous preparation to meet the training requirements of LLMs:

flowchart LR
    A[Données brutes] --> B[Nettoyage\nCleaning]
    B --> C[Formatage\nFormatting]
    C --> D[Annotation]
    D --> E[Données prêtes\npour l'entraînement]

    B1["• Suppression des incohérences\n• Suppression des doublons\n• Suppression des informations\n  non pertinentes\n• Maintien de l'intégrité des données"] --> B
    C1["• Structuration correcte des données\n• Alignement avec le format d'entrée\n  attendu pour l'entraînement du modèle"] --> C
    D1["• Étiquetage de la terminologie\n  domain-specific\n• Identification des concepts clés\n• Amélioration de la compréhension\n  et de la génération précise"] --> D

1. Cleaning:

• Removed inconsistencies.
• Remove duplicates.
• Elimination of irrelevant information.
• Maintaining data integrity.

2. Formatting:

• The data is structured correctly.
• Alignment with expected input format for model training.

3. Annotation:

• Added contextual value by tagging domain-specific terminology and key concepts.
• Improved the model’s ability to understand and generate accurate responses.

Impact of high quality data

High-quality, well-prepared data significantly impacts the accuracy and reliability of an LLM:

• Better accuracy: Well-curated datasets improve the model’s ability to generate accurate, contextually appropriate output, while minimizing the risk of hallucinations.
• Bias Reduction: Careful data selection and pre-processing helps mitigate bias, ensuring fair and ethical AI decision-making.
• Increased reliability: organizations can build more efficient and trustworthy domain-specific LLMs.

Demo — Data Preprocessing Pipeline

In this section, the Python script data_prep.py is designed to pre-process a dataset from the MIT SuperCloud project. This dataset provides a comprehensive view of GPU utilization metrics for various jobs in a high-performance computing (HPC) environment. The objective is to transform this raw data into a structured format suitable for machine learning applications, in particular for the fine-tuning of domain-specific LLMs in data center operations.

Script: data_prep.py

import pandas as pd
import numpy as np

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 1 : Chargement du dataset
# Dataset contenant les statistiques d'utilisation GPU pour datacenters
# Source : MIT SuperCloud project
# ─────────────────────────────────────────────────────────────────
df = pd.read_csv("dcgm.csv")
print("Structure initiale du dataset :")
print(df.info())
print(df.head())

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 2 : Nettoyage des données
# ─────────────────────────────────────────────────────────────────

# Gestion des valeurs manquantes : remplacement des NaN par 0
# pour éviter les problèmes de données manquantes
df = df.fillna(0)

# Suppression des lignes dupliquées pour éviter la redondance dans l'analyse
df = df.drop_duplicates()

# Suppression de la colonne ID (non utile pour l'entraînement)
if 'ID' in df.columns:
    df = df.drop(columns=['ID'])

print(f"\nDataset après nettoyage : {df.shape[0]} lignes, {df.shape[1]} colonnes")

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 3 : Formatage du dataset
# ─────────────────────────────────────────────────────────────────

# Conversion des colonnes numériques en float32 pour optimiser les performances
# et réduire l'utilisation mémoire
numeric_cols = df.select_dtypes(include=[np.number]).columns
df[numeric_cols] = df[numeric_cols].astype(np.float32)

# Assurer que le temps d'exécution (execution time) est stocké comme float
if 'execution_time' in df.columns:
    df['execution_time'] = df['execution_time'].astype(float)

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 4 : Annotation du dataset
# Catégorisation des niveaux d'utilisation GPU :
#   0-10%    → idle (inactif)
#   10-40%   → low (faible)
#   40-70%   → medium (moyen)
#   70-100%  → high (élevé)
# Cette classification améliore la compréhension et l'entraînement des modèles
# ─────────────────────────────────────────────────────────────────
if 'gpu_utilization' in df.columns:
    df['gpu_utilization_level'] = pd.cut(
        df['gpu_utilization'],
        bins=[0, 10, 40, 70, 100],
        labels=['idle', 'low', 'medium', 'high'],
        include_lowest=True
    )

print("\nDistribution des niveaux d'utilisation GPU :")
if 'gpu_utilization_level' in df.columns:
    print(df['gpu_utilization_level'].value_counts())

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 5 : Sauvegarde des données prétraitées
# Le dataset nettoyé et annoté est prêt pour les tâches
# de machine learning, notamment l'entraînement de modèles domain-specific
# pour optimiser les opérations datacenter
# ─────────────────────────────────────────────────────────────────
output_file = "prepared_dcgm.csv"
df.to_csv(output_file, index=False)
print(f"\nDataset prétraité sauvegardé : {output_file}")
print(f"Colonnes finales : {list(df.columns)}")

Execution result:

The script successfully loads a dataset of 96,000 entries detailing GPU utilization metrics. He:

1. Cleans the data by removing unnecessary columns.
2. Optimizes memory by converting numeric values ​​to float32.
3. Categorizes GPU usage into levels: idle, low, medium, high.
4. Saves the final dataset as prepared_dcgm.csv, ready for subsequent analyzes or machine learning applications.

Through this pre-processing pipeline, raw GPU usage logs are transformed into a structured and enriched dataset, ready for fine-tuning domain-specific models to optimize data center operations.

Lesson 3 — Fine-tuning techniques for domain-specific applications

Fine tuning overview

fine-tuning allows you to take a general LLM and adapt it to specific domains by training it on relevant data. This improves accuracy and relevance while maintaining the fundamental strengths of the model.

flowchart TD
    A[LLM Généraliste\npré-entraîné] --> B[Sélection d'un dataset\ndomain-specific de haute qualité]
    B --> C[Choix du modèle pré-entraîné\nMistral / Llama / autre]
    C --> D[Fine-tuning avec\nLoRA / PEFT / SFT]
    D --> E[Évaluation sur données\ndomaine non vues]
    E --> F{Performances\nsatisfaisantes ?}
    F -->|Non| G[Ajustements :\nhyperparamètres, data augmentation]
    G --> D
    F -->|Oui| H[Modèle domain-specific\nfinal déployé]

    style A fill:#4a90d9,color:#fff
    style H fill:#27ae60,color:#fff

4-step fine-tuning workflow

Step 1 — Dataset selection: Select a high-quality domain-specific dataset to ensure the model learns relevant knowledge. For example, a legal AI model needs data from case law and judicial opinions to provide accurate analytics.

Step 2 — Choosing the pre-trained model: Choose a pre-trained LLM (like Mistral or Llama) based on factors such as efficiency, architecture, and compatibility with the dataset. The right model serves as a solid basis for fine-tuning.

Step 3 — Fine-tuning: Train the model on domain-specific data using techniques like Low-Rank Adaptation (LoRA) and Parameter-Efficient Fine-Tuning (PEFT), which optimize performance while minimizing computational overhead. This ensures that the model scales effectively without sacrificing efficiency.

Step 4 — Evaluation: Evaluate the fine-tuned model by testing it on previously unpublished domain-specific data, measuring accuracy, relevance and consistency. Tweaks like hyperparameter tuning and data augmentation help to further refine performance.

Fine-tuning methods

There are several fine-tuning methods:

graph TD
    A[Méthodes de Fine-tuning] --> B[Full Supervised\nFine-tuning - SFT]
    A --> C[Low-Rank Adaptation\nLoRA]
    A --> D[Parameter-Efficient\nFine-Tuning - PEFT]

    B --> B1["✅ Haute précision\n❌ Très intensif en ressources\n❌ Met à jour TOUS les poids"]
    C --> C1["✅ Mémoire réduite\n✅ Efficace\n✅ Modifie uniquement\n   des couches sélectionnées"]
    D --> D1["✅ Très efficace\n✅ Moins de mémoire/calcul\n✅ Utilise des adaptateurs\n   légers"]

    style B fill:#e07b39,color:#fff
    style C fill:#27ae60,color:#fff
    style D fill:#27ae60,color:#fff

Full Supervised Fine-tuning (SFT)

Supervised Fine-tuning involves training the entire model on labeled domain-specific data, updating all its weights for maximum adaptation.

• Advantage: high precision, maximum adaptation.
• Disadvantage: requires substantial computational resources; resource intensive.
• Updates all model weights.

Low-Rank Adaptation (LoRA)

LoRA optimizes only selected layers of the model, significantly reducing memory usage while maintaining fine-tuning efficiency.

• Allows efficient adaptation without modifying the entire model.
• Ideal for resource-constrained environments.
• Significantly reduces memory usage.
• Maintaining fine-tuning quality.

Parameter-Efficient Fine-Tuning (PEFT)

PEFT takes a different approach by using lightweight adapters instead of changing all the model weights.

• Retains general knowledge of the base model.
• Enables targeted domain-specific improvements.
• Very efficient: requires less memory and computing power.
• While allowing high quality specialization.
• Ideal for fine-tuning large models in real-world applications.

Demo — Building a domain-specific LLM for fitness

In this demo, a fine-tuned model is built based on health and fitness data. Mistral-7B is used with LoRA and PEFT to construct responses from a fitness-related chatbot. A Hugging Face dataset with instruction-output pairs relevant to fitness conversations is used.

Script: fine-tuned Mistral-7B.py

import torch
import os
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling,
    TrainerCallback
)
from peft import LoraConfig, get_peft_model, TaskType
from datasets import load_dataset

# ─────────────────────────────────────────────────────────────────
# Setup GPU
# Vérification de la disponibilité GPU et affichage du device utilisé
# ─────────────────────────────────────────────────────────────────
device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 1 : Chargement du dataset
# Dataset Hugging Face avec paires instruction-output pour les conversations fitness
# ─────────────────────────────────────────────────────────────────
dataset = load_dataset("your-fitness-dataset")  # Dataset Hugging Face fitness

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 2 : Chargement du modèle pré-entraîné et du tokenizer
# AutoTokenizer et AutoModelForCausalLM avec trust_remote_code=True
# pour charger le modèle Mistral-7B instruct
# ─────────────────────────────────────────────────────────────────
model_name = "mistralai/Mistral-7B-Instruct-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    trust_remote_code=True,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
    device_map="auto"
)

# Désactivation du cache (nécessaire pour le training)
model.config.use_cache = False

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 3 : Définition de la configuration LoRA
# Spécification du rang LoRA, des facteurs de scaling, du dropout, et du type de tâche
# ─────────────────────────────────────────────────────────────────
lora_config = LoraConfig(
    r=16,                          # Rang LoRA
    lora_alpha=32,                 # Facteur de scaling
    lora_dropout=0.05,             # Dropout
    bias="none",
    task_type=TaskType.CAUSAL_LM   # Type de tâche : modèle de langage causal
)

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 4 : Application des adaptations LoRA au modèle
# ─────────────────────────────────────────────────────────────────
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 5 : Tokenisation du dataset
# ─────────────────────────────────────────────────────────────────
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,
        padding="max_length"
    )

tokenized_dataset = dataset.map(tokenize_function, batched=True)

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 6 : Définition du data collator
# Assure les labels et le padding corrects pour les tâches de modélisation du langage
# ─────────────────────────────────────────────────────────────────
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False  # Causal LM (pas Masked LM)
)

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 7 : Définition des arguments d'entraînement
# batch size, logging steps, etc.
# fp16 activé sur GPU pour un entraînement plus rapide
# ─────────────────────────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    warmup_steps=100,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=500,          # Log toutes les 500 étapes
    save_steps=500,             # Sauvegarde toutes les 500 étapes
    fp16=(device == "cuda"),    # fp16 sur GPU pour un entraînement plus rapide
)

# Callback personnalisé pour afficher la progression d'entraînement toutes les 500 étapes
# Permet de suivre la perte (loss) et l'étape d'entraînement courante
class PrintProgressCallback(TrainerCallback):
    def on_log(self, args, state, control, logs=None, **kwargs):
        if state.global_step % 500 == 0:
            print(f"Étape {state.global_step}: loss = {logs.get('loss', 'N/A')}")

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 8 : Initialisation du trainer
# Crée le Hugging Face Trainer avec le modèle, les données et les callbacks
# ─────────────────────────────────────────────────────────────────
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    data_collator=data_collator,
    callbacks=[PrintProgressCallback()]
)

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 9 : Entraînement du modèle
# Durée approximative : 90 minutes sur une NVIDIA 4090
# ─────────────────────────────────────────────────────────────────
trainer.train()

# Sauvegarde du nouveau modèle fine-tuné
model.save_pretrained("mistral7B_fitness_llm_finetuned")
tokenizer.save_pretrained("mistral7B_fitness_llm_finetuned")
print("Modèle fine-tuné sauvegardé : mistral7B_fitness_llm_finetuned")

Note: Training takes approximately 90 minutes on local setup with an NVIDIA 4090.

Test script: test_model.py

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

# ─────────────────────────────────────────────────────────────────
# Définition du chemin vers le modèle fine-tuné
# ─────────────────────────────────────────────────────────────────
model_path = "./mistral7B_fitness_llm_finetuned"

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 3 : Chargement du tokenizer
# ─────────────────────────────────────────────────────────────────
tokenizer = AutoTokenizer.from_pretrained(model_path)

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 4 : Chargement du modèle sur le device
# Vérification de la disponibilité GPU
# fp16 pour GPU, fp32 sinon
# ─────────────────────────────────────────────────────────────────
device = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
    device_map="auto"
)
model.eval()

# Étape optionnelle : mise à jour de la configuration du modèle
# Assure que l'ID du modèle est défini dans la configuration si ce n'est pas déjà le cas
# Aide à maintenir la cohérence lors de la génération de texte
if not hasattr(model.config, 'model_id'):
    model.config.model_id = model_path

# ─────────────────────────────────────────────────────────────────
# ÉTAPE 6 : Définition du prompt (ligne 28)
# Prompt hardcodé formaté avec le template de chat Mistral
# (balises d'instruction [INST] et [/INST])
# ─────────────────────────────────────────────────────────────────
prompt = "[INST] What are the benefits of weightlifting for muscle growth? [/INST]"

# Tokenisation de l'entrée et déplacement vers le device
# Convertit le prompt en token IDs et crée un masque d'attention
# Déplace tous les tenseurs d'entrée vers le device correct
inputs = tokenizer(prompt, return_tensors="pt", return_attention_mask=True).to(device)

# ─────────────────────────────────────────────────────────────────
# Génération de la réponse avec greedy decoding
# ─────────────────────────────────────────────────────────────────
with torch.no_grad():
    outputs = model.generate(
        input_ids=inputs["input_ids"],
        attention_mask=inputs["attention_mask"],
        max_new_tokens=200,
        do_sample=False  # Greedy decoding
    )

# ─────────────────────────────────────────────────────────────────
# Décodage et affichage de la réponse
# ─────────────────────────────────────────────────────────────────
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Réponse du modèle fine-tuné :")
print(response)

Observed result: When testing with the prompt “What are the benefits of weightlifting for muscle growth?”, the fine-tuned model responds with 4 well-articulated benefits, demonstrating successful specialization in the fitness field.

Lesson 4 — Compliance, confidentiality and continuous improvement

Regulatory frameworks

In today’s rapidly evolving digital landscape, data privacy and compliance have become paramount concerns for organizations around the world. Navigating complex regulatory frameworks presents significant challenges.

graph TD
    A[Cadres Réglementaires] --> B[GDPR\nGeneral Data Protection Regulation\nUnion Européenne]
    A --> C[HIPAA\nHealth Insurance Portability\nand Accountability Act\nÉtats-Unis - Santé]
    A --> D[CCPA\nCalifornia Consumer\nPrivacy Act\nCalifornie]

    style A fill:#c0392b,color:#fff
    style B fill:#2980b9,color:#fff
    style C fill:#27ae60,color:#fff
    style D fill:#8e44ad,color:#fff

Data protection policies

To face regulatory challenges, organizations are implementing robust strategies:

flowchart LR
    A[Stratégies de\nProtection des Données] --> B[Anonymisation\ndes données]
    A --> C[Techniques de\npolitique différentielle\nDifferential Privacy]
    A --> D[Contrôles d'accès\nbased sur les rôles\nRBAC]

    B --> E["Suppression des PII\n(Personally Identifiable\nInformation)"]
    C --> F[Ajout de bruit\nstatistique pour\nprotéger la vie privée]
    D --> G[Accès restreint\nselon les rôles\nutilisateurs]

1. Data anonymization: deletion or masking of PII (Personally Identifiable Information) such as emails and telephone numbers.
2. Differential Privacy Techniques: Adding statistical noise to protect privacy while maintaining data usefulness.
3. Role-Based Access Controls (RBAC): Restricting access to sensitive data based on user roles.

AI Model Governance

Establishing comprehensive AI model governance frameworks is also critical. The NIST AI Risk Management Framework provides a structured approach to ensuring that AI systems operate ethically and transparently.

mindmap
  root((Gouvernance AI))
    NIST AI Risk Management Framework
      Identification des risques
      Évaluation des risques
      Réponse aux risques
      Monitoring continu
    Éthique et transparence
      Explicabilité des décisions
      Auditabilité
      Équité et non-discrimination
    Conformité réglementaire
      GDPR
      HIPAA
      CCPA

Continuous learning and feedback loops

Embracing continuous learning and feedback loops is vital for the evolution of AI systems, especially for domain-specific models.

Three pillars of continuous improvement:

1. Human-in-the-loop: Integrating human-in-the-loop methodologies allows models to adapt based on real-time inputs, improving their accuracy and relevance.

2. User feedback for iterative fine-tuning: leveraging user feedback for iterative fine-tuning ensures that AI solutions remain aligned with user needs and expectations.

3. Auto-curation of domain-specific datasets: Automatic curation of domain-specific datasets from real-time interactions allows AI systems to stay up-to-date and contextually aware, leading to more informed and efficient results.

flowchart TD
    A[Modèle en Production] --> B[Interactions utilisateur]
    B --> C[Anonymisation des PII]
    C --> D[feedback_log.json]
    D --> E[create_feedback_dataset.py\nCréation du dataset de feedback]
    E --> F[fine_tune_model.py\nFine-tuning continu avec LoRA]
    F --> G[Nouveau modèle amélioré]
    G --> A

    H[Human-in-the-loop] --> F
    
    style A fill:#4a90d9,color:#fff
    style G fill:#27ae60,color:#fff

Demo — Privacy-Preserving Feedback Loop

In this demo, a privacy-preserving feedback loop is implemented. Anonymized data and continuous learning are used to update the domain-specific model. Libraries used: torch, transformers, datasets, peft, accelerate.

Script 1: preprocess_feedback.py — Anonymized logging

This script logs user interactions while ensuring confidentiality. It removes PII (emails and phone numbers), then stores anonymized user inputs, model responses and feedback in a new feedback_log.json file with timestamps.

import json
import re
from datetime import datetime

# ─────────────────────────────────────────────────────────────────
# Fonction d'anonymisation des PII
# Supprime les emails et numéros de téléphone pour protéger la vie privée
# ─────────────────────────────────────────────────────────────────
def anonymize_pii(text):
    """Supprime les PII (emails, numéros de téléphone) du texte."""
    # Suppression des adresses email
    text = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL_REMOVED]', text)
    # Suppression des numéros de téléphone (formats variés)
    text = re.sub(r'(\+?1?\s?)?(\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4})', '[PHONE_REMOVED]', text)
    return text

# ─────────────────────────────────────────────────────────────────
# Journalisation d'une interaction utilisateur
# ─────────────────────────────────────────────────────────────────
def log_interaction(user_input, model_response, feedback_score):
    """
    Journalise une interaction en anonymisant les données sensibles.
    Stocke dans feedback_log.json avec un timestamp.
    """
    # Anonymisation des PII dans les entrées et réponses
    anonymized_input = anonymize_pii(user_input)
    anonymized_response = anonymize_pii(model_response)
    
    # Création de l'entrée de log
    log_entry = {
        "timestamp": datetime.now().isoformat(),
        "user_input": anonymized_input,
        "model_response": anonymized_response,
        "feedback": feedback_score
    }
    
    # Chargement du log existant ou création d'une nouvelle liste
    try:
        with open("feedback_log.json", "r") as f:
            logs = json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        logs = []
    
    # Ajout de la nouvelle entrée et sauvegarde
    logs.append(log_entry)
    with open("feedback_log.json", "w") as f:
        json.dump(logs, f, indent=2)
    
    print(f"Interaction journalisée à {log_entry['timestamp']}")

# Exemple d'utilisation
if __name__ == "__main__":
    log_interaction(
        user_input="What exercises help with john.doe@email.com lower back pain?",
        model_response="For lower back pain, consider these exercises...",
        feedback_score=4
    )
    print("Script preprocess_feedback.py exécuté. Fichier feedback_log.json créé.")

Result: The script creates the feedback_log.json file to track model performance and improve responses over time. Here is an example of the generated structure:

[
  {
    "timestamp": "2024-01-15T10:30:45.123456",
    "user_input": "What exercises help with lower back pain?",
    "model_response": "For lower back pain, consider these exercises...",
    "feedback": 4
  },
  {
    "timestamp": "2024-01-15T11:15:22.654321",
    "user_input": "How many calories should I eat to gain muscle?",
    "model_response": "Caloric intake for muscle gain depends on...",
    "feedback": 5
  }
]

Script 2: create_feedback_dataset.py — Creating the feedback dataset

This script converts logged user feedback into a structured dataset for fine-tuning. It reads feedback_log.json, extracts user inputs and model responses, then formats them correctly for subsequent fine-tuning.

from datasets import Dataset
import json
import os

# ─────────────────────────────────────────────────────────────────
# Chargement du feedback journalisé
# ─────────────────────────────────────────────────────────────────
def create_feedback_dataset():
    """
    Lit feedback_log.json, extrait les paires input/response,
    et les formate pour le fine-tuning.
    Le dataset est sauvegardé dans /datasets/feedback_dataset.
    """
    with open("feedback_log.json", "r") as f:
        logs = json.load(f)
    
    # Extraction des paires instruction-réponse
    # Filtrage optionnel sur les feedbacks positifs (score >= 3)
    training_data = []
    for entry in logs:
        if entry.get("feedback", 0) >= 3:  # Garder uniquement les bons feedbacks
            training_data.append({
                "text": f"[INST] {entry['user_input']} [/INST] {entry['model_response']}"
            })
    
    print(f"Nombre d'exemples d'entraînement extraits : {len(training_data)}")
    
    # Création et sauvegarde du dataset Hugging Face
    dataset = Dataset.from_list(training_data)
    output_path = "./datasets/feedback_dataset"
    os.makedirs(output_path, exist_ok=True)
    dataset.save_to_disk(output_path)
    
    print(f"Dataset de feedback sauvegardé : {output_path}")
    return dataset

if __name__ == "__main__":
    create_feedback_dataset()

Result: The feedback dataset is saved in /datasets/feedback_dataset, ready for training and continuous improvement of the model.

Script 3: fine_tune_model.py — Continuous fine-tuning with feedback

This script extends the domain-specific fine-tuning approach from section 3 by incorporating continuous learning via real user feedback. Instead of training a model from scratch, it fine-tunes the existing fine-tuned Mistral LLM using Low-Rank Adaptation (LoRA), making updates efficient and light on memory.

import torch
import os
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    TrainingArguments,
    Trainer,
    DataCollatorForLanguageModeling
)
from peft import LoraConfig, get_peft_model, TaskType, PeftModel
from datasets import load_from_disk

# ─────────────────────────────────────────────────────────────────
# Chargement du modèle fine-tuné existant (issu de la section 3)
# Au lieu de partir de zéro, on affine le modèle existant
# ─────────────────────────────────────────────────────────────────
base_model_path = "./mistral7B_fitness_llm_finetuned"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(base_model_path)
model = AutoModelForCausalLM.from_pretrained(
    base_model_path,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
    device_map="auto"
)
model.config.use_cache = False

# Configuration LoRA pour le fine-tuning continu
# Clé : utilisation de LoRA pour des mises à jour efficaces en mémoire
lora_config = LoraConfig(
    r=8,          # Rang réduit pour le fine-tuning incrémental
    lora_alpha=16,
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM
)
model = get_peft_model(model, lora_config)

# Chargement du dataset de feedback (créé par create_feedback_dataset.py)
feedback_dataset = load_from_disk("./datasets/feedback_dataset")

def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        truncation=True,
        max_length=512,
        padding="max_length"
    )

tokenized_feedback = feedback_dataset.map(tokenize_function, batched=True)
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# Arguments d'entraînement pour le fine-tuning continu
training_args = TrainingArguments(
    output_dir="./results_feedback",
    num_train_epochs=1,           # Moins d'époques pour le fine-tuning incrémental
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    logging_steps=50,
    save_steps=100,
    fp16=(device == "cuda"),
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_feedback,
    data_collator=data_collator,
)

trainer.train()

# ─────────────────────────────────────────────────────────────────
# LIGNE 68 : Création du nouveau modèle mis à jour
# Incorpore le feedback utilisateur pour amélioration continue
# ─────────────────────────────────────────────────────────────────
new_model_path = "./mistral7B_fitness_llm_updated"
model.save_pretrained(new_model_path)
tokenizer.save_pretrained(new_model_path)
print(f"Nouveau modèle mis à jour sauvegardé : {new_model_path}")

Key difference from section 3: Rather than curating static datasets, this process dynamically logs user interactions, anonymizes them, and uses feedback-driven fine-tuning to improve responses over time.

At line 68, a new model /mistral7B_fitness_llm_updated is created, incorporating this feedback.

Script 4: test_model.py — Comparison old vs new model

This script compares the responses of the old fine-tuned model with the updated model after additional training. It loads both models, processes a test request, and generates responses to evaluate improvements.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

def load_model(model_path, device):
    """Charge un modèle et son tokenizer depuis le chemin donné."""
    tokenizer = AutoTokenizer.from_pretrained(model_path)
    model = AutoModelForCausalLM.from_pretrained(
        model_path,
        torch_dtype=torch.float16 if device == "cuda" else torch.float32,
        device_map="auto"
    )
    model.eval()
    return model, tokenizer

def generate_response(model, tokenizer, prompt, device, max_tokens=200):
    """Génère une réponse pour un prompt donné."""
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_tokens,
            do_sample=False
        )
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Configuration
device = "cuda" if torch.cuda.is_available() else "cpu"
test_prompt = "[INST] What are the benefits of weightlifting for muscle growth? [/INST]"

# Chargement des deux modèles
print("Chargement de l'ancien modèle...")
old_model, old_tokenizer = load_model("./mistral7B_fitness_llm_finetuned", device)

print("Chargement du nouveau modèle (mis à jour avec feedback)...")
new_model, new_tokenizer = load_model("./mistral7B_fitness_llm_updated", device)

# Génération et comparaison des réponses
print("\n" + "="*60)
print("RÉPONSE — ANCIEN MODÈLE :")
print("="*60)
old_response = generate_response(old_model, old_tokenizer, test_prompt, device)
print(old_response)

print("\n" + "="*60)
print("RÉPONSE — NOUVEAU MODÈLE (après fine-tuning feedback) :")
print("="*60)
new_response = generate_response(new_model, new_tokenizer, test_prompt, device)
print(new_response)

Observed result: The new model’s response is extended with additional information, demonstrating that the model continually learns and adapts for domain-specific tasks.

By running the same prompt on both versions, it helps determine whether the new model produces better, more refined responses after fine-tuning. This ensures that the model continuously learns and adapts for domain-specific tasks.

4. Summary and key concepts

mindmap
  root((LLM\nDomain-Specific))
    Données
      Collecte diversifiée
      Nettoyage et déduplication
      Formatage structuré
      Annotation domain-specific
      MIT SuperCloud Dataset
    Fine-tuning
      Full SFT
        Tous les poids
        Ressources intensives
      LoRA
        Couches sélectionnées
        Mémoire réduite
      PEFT
        Adaptateurs légers
        Très efficace
    Modèles
      Llama-2-7b-chat-hf
      Mistral-7B-Instruct
      TSLAM-4B Télécoms
    Conformité
      GDPR
      HIPAA
      CCPA
      NIST AI Framework
    Amélioration continue
      Anonymisation PII
      Feedback Loop
      Human-in-the-loop
      Auto-curation

Summary of scripts developed in the course

Complete flow: from raw data to model in production

flowchart TD
    A[Données brutes\nEx: dcgm.csv - 96 000 entrées GPU] --> B[data_prep.py\nNettoyage, formatage, annotation]
    B --> C[prepared_dcgm.csv\nDataset structuré]
    
    D[LLM Généraliste\nMistral-7B / Llama] --> E[fine-tuned_Mistral-7B.py\nFine-tuning avec LoRA + PEFT]
    C --> E
    E --> F[mistral7B_fitness_llm_finetuned\nModèle domain-specific v1]
    
    F --> G[Déploiement en production]
    G --> H[Interactions utilisateur]
    H --> I[preprocess_feedback.py\nAnonymisation PII]
    I --> J[feedback_log.json]
    J --> K[create_feedback_dataset.py]
    K --> L[Dataset de feedback]
    L --> M[fine_tune_model.py\nFine-tuning continu LoRA]
    M --> N[mistral7B_fitness_llm_updated\nModèle domain-specific v2+]
    N --> G

    style A fill:#95a5a6,color:#000
    style F fill:#3498db,color:#fff
    style N fill:#27ae60,color:#fff
    style G fill:#e67e22,color:#fff

5. Glossary

Search Terms

domain-specific · llm · agents · ai · orchestration · artificial · intelligence · generative · script · data · fine-tuning · feedback · model · continuous · comparison · generalist · raw · testmodel.py