Open-Source LLMs: Introduction

Table of Contents

• Module 1: Understanding Open-source LLMs
  • What Are Open-source LLMs?
  • The Rise of the Open-source LLM Movement
  • Key Characteristics of Open-source LLMs
  • Benefits for Businesses and Developers
• Module 2: Customizing and Deploying Open-source LLMs
  • Inspecting and Modifying Open-source LLMs
  • Understanding LLM Architecture and Customization Options
  • Fine-tuning Open-source LLMs
  • Deploying LLMs on Your Own Infrastructure
  • Industry-Specific Customization Demonstration: Building a Customer Service Chatbot
• Module 3: Surveying Leading Open-source LLMs and Their Use Cases
  • Llama, BLOOM, DeepSeek, and Mistral Overview
  • Architecture, Scale, and Benchmarks
  • Specialized Models and Optimal Use Cases
  • Model Strengths, Weaknesses, and When to Use Each
• Summary

Module 1: Understanding Open-source LLMs

What Are Open-source LLMs?

Generative AI is one of the most impactful technologies to have emerged in the last decade. Many companies are racing to provide the best large language models (LLMs), both to capture market share and to solve real-world problems. Like every emergent field, there are many competing players.

On one side are large software companies that develop closed-source models and sell products built on top of them — OpenAI with ChatGPT, or Anthropic with Claude. On the other side are tech companies, research institutions, and collaborative communities that deliver open-source large language models. Like any open-source community, this ecosystem thrives on contributions from developers and researchers worldwide, fostering innovation and accessibility.

The main advantage of open-source models is that you can host them locally — on your own computer or on your organization’s infrastructure. This means more privacy and more control over the model’s capabilities.

Open-source vs. closed-source, defined:

An open-source LLM is a language model whose model weights and code are publicly available. Anyone can download them, study how they work, and change them to better fit their needs. Most of the time, they also come with permissive licenses that allow free use in research, in personal projects, or even in commercial applications.

This is very different from proprietary models like ChatGPT or Claude. With those, the company controls the technology — you don’t get access to the model weights. Instead, you interact with the model through an API or a web interface and must follow strict usage rules. You use the model as a service, not as something you fully own or can adapt.

Open-source models stand out in four main ways:

For developers, the benefits are significant: freedom to experiment without heavy restrictions, avoidance of vendor lock-in, and the power to adapt a model to exact needs — whether building a company tool, testing research ideas, or creating AI agents with unique behaviors.

flowchart LR
    subgraph Closed["Closed-source Models"]
        C1[OpenAI - ChatGPT]
        C2[Anthropic - Claude]
    end
    subgraph Open["Open-source Models"]
        O1[Public weights]
        O2[Public code]
        O3[Permissive licenses]
    end
    C1 --> API[Access only via API / Web UI]
    C2 --> API
    O1 --> Local[Download & run locally]
    O2 --> Local
    O3 --> Local
    Local --> Control[Full control: privacy, customization, no vendor lock-in]
    API --> Restricted[Usage governed by provider's rules]

The Rise of the Open-source LLM Movement

The rise of open-source large language models didn’t happen overnight. A short history of the technology:

• 2017 — Google introduced the transformer architecture in the paper Attention Is All You Need.
• 2018 — The first widely known language model, BERT (Bidirectional Encoder Representations from Transformers), had 340 million parameters. Parameters are the weights and biases a model learns during training to understand human-like text. Google used BERT extensively to interpret complex search queries. That same year, OpenAI released GPT-1.
• 2019 — GPT-2 caught the public’s attention, though OpenAI initially refused to release it fully to the public for fear of misuse.
• 2020 — GPT-3 was released — a 175-billion-parameter model.
• 2022 — ChatGPT launched and became wildly popular: after only 5 days, 1 million users had already tried it. OpenAI has since released further iterations, with GPT-5 being the latest at the time of recording.
• 2022 (in parallel) — A few source-available models were launched, such as BLOOM and Llama. These are not considered truly open source because their licenses don’t meet the criteria defined by the Open Source Initiative’s open-source definition. A key issue is that the licenses discriminate against certain groups and fields of endeavor — for example, Llama 2 and later versions prohibit entities with more than 700 million monthly active users from using the model without a separate license from Meta, granted at Meta’s sole discretion.
• Models with more permissive licenses (such as Apache 2.0) began to emerge, including Mistral AI’s Mistral 7B and Mixtral 8x7B. This marked a turning point: for the first time, researchers, startups, and individual developers could build with cutting-edge AI tools without the restrictions imposed by large corporations — mirroring how Linux and Apache challenged proprietary software decades earlier.

As commercial players like OpenAI, Anthropic, and Google continued releasing more powerful but closed models, the open-source community responded with alternatives that quickly caught up in quality:

timeline
    title Evolution of Large Language Models
    2017 : Transformer architecture (Attention Is All You Need)
    2018 : BERT (340M params) : GPT-1
    2019 : GPT-2 public release
    2020 : GPT-3 (175B params)
    2022 : ChatGPT launch : BLOOM and Llama (source-available)
    2023-2024 : Apache-licensed models emerge (Mistral, Mixtral)
    2024-2025 : Gemma, Qwen, DeepSeek v3/R1 mature the open ecosystem

Key Characteristics of Open-source LLMs

How exactly do open-source LLMs differentiate from closed-source models, and when should you choose which?

1. Public access to model weights. Weights are the result of intensive training and allow a model to generate text, write code, or answer questions. With proprietary models such as GPT-4 or Gemini, the only way to use the model is through a paid API — there is no access to the underlying technology. In contrast, open models like Qwen and Mixtral make their weights downloadable for local deployment, so you can use, study, and fine-tune them without gatekeeping.

2. Ability to modify architecture and retrain/fine-tune. Developers can adjust configurations to better suit their hardware, retrain with industry-specific data, or experiment with new algorithms. DeepSeek’s mixture-of-experts architecture is a good example — only the most relevant parts of the model activate for a given input, making computation faster and more targeted. Because code and training pipelines are open, researchers can learn from, improve on, and even fork projects to create new variants.

3. Transparency. Open-source LLMs provide public documentation about how models are trained, what data is used, and how they perform on benchmarks. This enables reproducing results, auditing for bias, and verifying compliance with safety standards. Qwen, for example, releases extensive benchmarks, training logs, and multilingual data-coverage details; Mixtral posts clear performance reports on reasoning, code generation, and conversation. This clarity also helps organizations meet privacy and regulatory requirements more easily than with opaque closed systems.

4. Community-driven development. Improvements, bug fixes, and new features often come from global collaboration, so open models evolve rapidly — sometimes outpacing proprietary competitors. Mixtral’s code and weights have been adopted worldwide, leading to specialized variants for healthcare, education, and finance. DeepSeek and Qwen have active forums and GitHub repositories where the community contributes solutions, suggests optimizations, and tests new releases. This culture of shared progress spreads best practices — more efficient quantization, safer agent deployment, smarter function calling — rather than siloing knowledge.

mindmap
  root((Open-source LLM Characteristics))
    Public Weights
      Downloadable
      No gatekeeping
      Local deployment
    Modifiable Architecture
      Retraining
      Fine-tuning
      Forking projects
    Transparency
      Public benchmarks
      Training logs
      Bias auditing
    Community-driven
      GitHub collaboration
      Rapid iteration
      Domain-specific forks

Benefits for Businesses and Developers

1. Greater control over model behavior. A generic API gives you a one-size-fits-all solution — optimized for everyone, but for no one in particular, with few levers to adjust its behavior. With an open-source model, you have the source code, so you can fine-tune it on your own data. For example, a legal tech company can take a base model and train it further on legal documents, court cases, and contracts — producing a specialist rather than a generalist. This level of customization isn’t possible with a closed API (retrieval-augmented generation and prompting can approximate it, but without access to the model’s internals).

2. Enhanced data privacy and security via on-premises deployment. In sensitive industries — healthcare, finance, legal, government — you often cannot legally or ethically send user data to a third-party API, where it might be used for training or exposed in a breach. Running an open-source model on your own infrastructure (company servers or a private cloud instance) means data never leaves your ecosystem — input, processing, and output all happen behind your firewall. For many projects this isn’t a nice-to-have, it’s an absolute requirement.

3. Cost savings. A proprietary API looks cheap at first — no upfront cost, pay per query — but as an application scales, the recurring bill grows without bound; every call costs money, and millions of calls a day become a massive expense. Self-hosting requires upfront investment in hardware or cloud GPUs and technical expertise to set up, but once running, the marginal cost per query approaches zero because the compute has already been paid for. For high-volume applications, total cost of ownership for a self-run open-source model can be orders of magnitude lower than an API — you invest in capital instead of burning operational funds.

4. Faster iteration cycles. Development against a remote API can be slow: rate limits throttle testing, a black-box model makes debugging weird outputs difficult, and if the provider changes something server-side, your application can break unexpectedly. With open-source models you can test as aggressively as you want without per-test charges, A/B test different fine-tuned versions, and have full visibility to diagnose and fix issues — all independent of a third party’s roadmap, stability, or pricing changes.

To recap: open-source models give you the control to build a tailored AI, the security to handle sensitive data, the economics to scale profitably, and the agility to outpace competitors. It requires more upfront technical effort, but the long-term strategic advantages are profound.

flowchart TD
    A[Choosing an LLM deployment strategy] --> B{Sensitive data or\nstrict compliance needs?}
    B -->|Yes| C[Self-host open-source model]
    B -->|No| D{High-volume, cost-sensitive\nat scale?}
    D -->|Yes| C
    D -->|No| E{Need deep domain\nspecialization?}
    E -->|Yes| C
    E -->|No| F[Proprietary API may be sufficient\nfor quick prototyping]

Module 2: Customizing and Deploying Open-source LLMs

Inspecting and Modifying Open-source LLMs

So you’ve downloaded an open-source large language model — now what do you actually have, and how do you start making it your own?

What you get when you download a model. It’s not a single magical file — it’s typically a collection of files. The most common place to find them is Hugging Face Hub, which acts like a “GitHub for AI models.” A downloaded model usually includes three key asset types:

1. Model weights — files with a .safetensors or .bin extension. These are the model’s learned knowledge: a numerical representation of the patterns found in training data. Without the architecture to structure them, weights are just a huge pile of numbers; combined with the right architecture, they become a reasoning engine.
2. Configuration file (config.json) — the blueprint that tells frameworks like Hugging Face Transformers how to reassemble the model architecture to correctly load the weights. It contains critical parameters: number of layers, hidden layer size, number of attention heads, vocabulary size, and more. Inspecting this file is the first step to understanding a model’s structure before even loading it.
3. Tokenizer files (e.g. tokenizer.json, vocab.json) — define how the model breaks text into smaller pieces (“tokens”). The tokenizer has its own vocabulary and rules and must match the model exactly; using the wrong tokenizer will completely garble the model’s understanding.

Inspecting the config file. You can simply open config.json in a text editor — it’s a structure of key-value pairs. You can note the model’s dimensionality (e.g. hidden_size: 4096) or how many transformer layers it has (e.g. num_hidden_layers: 32), which tells you the scale of the model you’re working with.

Inspecting the weights programmatically. You can’t read the raw numbers directly, but you can load them into Python using a library like PyTorch or TensorFlow. The following script (provided as a course resource) evaluates weights layer-by-layer and produces a detailed weight analysis file:

import torch
from transformers import AutoModel, AutoTokenizer
import os

# 1. Define the model name (from Hugging Face) or path to local files
model_name = "microsoft/DialoGPT-small"  # Example model - replace with your model
# Alternatively, for a local model:
# model_name = "./path/to/your/local/model"

# 2. Create output directory for inspection results
output_dir = "model_inspection"
os.makedirs(output_dir, exist_ok=True)

# 3. Load the model and tokenizer
print("Loading model and tokenizer...")
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 4. Basic model information
print(f"\nModel architecture: {type(model).__name__}")
print(f"Number of parameters: {sum(p.numel() for p in model.parameters()):,}")
print(f"Model device: {next(model.parameters()).device}")

# 5. Inspect model structure
print("\nModel structure:")
for name, module in model.named_children():
    print(f"- {name}: {type(module).__name__}")

# 6. Detailed weight inspection
print("\nInspecting weights by layer...")

# Open a file to save detailed analysis
with open(f"{output_dir}/weight_analysis.txt", "w") as f:
    f.write("Detailed Weight Analysis\n")
    f.write("=" * 50 + "\n\n")

    total_params = 0
    for name, param in model.named_parameters():
        if param.requires_grad:
            f.write(f"Layer: {name}\n")
            f.write(f"  Shape: {tuple(param.shape)}\n")
            f.write(f"  Size: {param.numel():,} parameters\n")
            f.write(f"  Mean: {param.mean().item():.6f}\n")
            f.write(f"  Std: {param.std().item():.6f}\n")
            f.write(f"  Min: {param.min().item():.6f}\n")
            f.write(f"  Max: {param.max().item():.6f}\n")
            f.write("-" * 30 + "\n")

            total_params += param.numel()

            # Print first few layers to console
            if len(name.split('.')) < 3:
                print(f"{name}: {tuple(param.shape)} ({param.numel():,} parameters)")

    f.write(f"\nTotal trainable parameters: {total_params:,}\n")

# 7. Save a small sample of actual weight values
print("\nSaving weight samples...")
with open(f"{output_dir}/weight_samples.txt", "w") as f:
    f.write("Weight Value Samples\n")
    f.write("=" * 50 + "\n\n")

    for name, param in model.named_parameters():
        if param.requires_grad and param.numel() < 1000:
            f.write(f"{name} (shape {tuple(param.shape)}):\n")
            f.write(f"Values: {param.data.flatten()[:10].tolist()}\n")
            f.write("-" * 40 + "\n")

# 8. Save model configuration
print("Saving model configuration...")
model.config.to_json_file(f"{output_dir}/config.json")

# 9. Create a simple summary report
with open(f"{output_dir}/summary.txt", "w") as f:
    f.write("Model Inspection Summary\n")
    f.write("=" * 50 + "\n\n")
    f.write(f"Model: {model_name}\n")
    f.write(f"Architecture: {type(model).__name__}\n")
    f.write(f"Total parameters: {sum(p.numel() for p in model.parameters()):,}\n")
    f.write(f"Trainable parameters: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}\n")
    f.write(f"Model layers: {len(list(model.named_parameters()))}\n")

print(f"\nInspection complete! Results saved to '{output_dir}' directory")
print("\nFiles created:")
for file in os.listdir(output_dir):
    print(f"- {file}")

Running this script produces a detailed breakdown for each layer — for example, the token embedding layer is the model’s lookup table converting input token IDs into meaningful vectors (the first layer of the model), and the position embedding layer provides information about a token’s position in a sequence (crucial, since transformers don’t inherently know word order). A stat such as “sequences up to 1024 tokens long, each position having its own 768-dimensional vector” tells you exactly how the model represents position. You can inspect the model’s structure programmatically — see all layers by name, check tensor shapes, and even extract specific parts of the model such as just the embedding layer or a specific attention head. If you’re curious to learn more, you can feed the generated output files into an LLM and ask it to explain them step by step.

Modifying the model. Changes range from small tweaks to large-scale overhauls, and don’t always require full retraining (full retraining is covered in the fine-tuning clip):

• Generation configuration — the simplest modification. This doesn’t change the model itself, only how it behaves when generating text: adjusting max_length to control output size, temperature to influence randomness, or top_p to narrow word choices. These parameters are set on every call, giving immediate control without touching a single weight.
• Architecture modification via the config file — for example, reducing num_hidden_layers from 32 to 24 to create a smaller, faster model. The catch: existing downloaded weights are shaped for the original 32-layer architecture and won’t fit the new 24-layer one. To make this work, you’d need to strategically copy weights from the original model (skipping some layers) and then extensively retrain to regain performance.
• Direct weight/layer manipulation — the most advanced modification. Research has shown you can sometimes improve performance in a specific area (like math or reasoning) by applying a small, targeted change to the weights of just a few layers — akin to “precise brain surgery.” This requires identifying a key layer, extracting its weights, applying a mathematical transformation, and plugging them back in — deep analysis and a hypothesis about what that layer does are prerequisites.

The key takeaway: an open-source model is not a static black box. It’s a collection of files — weights, configs, tokenizers — that you can open, read, and understand. You can start with simple configuration changes and progress to advanced architectural edits. This ability to inspect and modify every component is what sets open-source AI apart.

flowchart TD
    A[Downloaded Model] --> B[model.safetensors / .bin<br/>Weights]
    A --> C[config.json<br/>Architecture blueprint]
    A --> D[tokenizer.json / vocab.json<br/>Tokenizer]
    B --> E{What do you want to change?}
    E -->|Just behavior| F[Generation config:<br/>temperature, top_p, max_length]
    E -->|Model size| G[Edit config.json layers<br/>+ re-copy/retrain weights]
    E -->|Targeted capability| H[Direct weight/layer surgery]
    E -->|New domain knowledge| I[Fine-tuning - see next section]

Understanding LLM Architecture and Customization Options

Every LLM is built from four fundamental components, each mapped to real downloaded files:

1. The tokenizer. Not part of the neural network itself, but the critical translator between human language and machine numbers (tokenizer.json). It breaks a sentence into tokens — words, sub-words, or punctuation — and converts each into a unique ID. The model only ever sees a sequence of numbers; on the way out, the model predicts token IDs which the tokenizer converts back to readable text. Swapping tokenizers can completely break a model, since its internal logic is tied to the exact way it was trained to segment text.

2. Embedding layers. Where the model starts to find meaning, visible in weight files as layers like wte.weight (word token embedding) and wpe.weight (word position embedding).

• The token embedding layer is a giant lookup table: it takes token IDs and finds a corresponding dense vector (hundreds of numbers) representing that token’s meaning.
• The position embedding layer adds information about each token’s position in the sequence, since the core transformer architecture doesn’t inherently understand word order.
• Output: a stream of vectors, each representing a word plus its position, ready for deeper processing.

3. Transformer blocks. Where the attention mechanism happens and where the model does its actual reasoning — visible as layers inside an h module (h.0, h.1, … h.11 for a 12-layer model). Each block typically consists of:

• Multi-head attention — lets the model look at every other word in the sequence and decide which ones matter most for understanding the current word (solving problems like pronoun resolution: “The chef ate the pizza because he was hungry” → the model learns to connect “he” to “chef”).
• Feed-forward network — a smaller dense network that further processes the attention output.
• Layer normalization and residual connections — stabilizers that keep signals well-behaved as they flow through a deep network, critical for training stability. Data passes sequentially through each block, with each layer refining and building on the previous one’s understanding.

4. Output layer. After the data passes through all transformer blocks, a final linear layer projects the last hidden state back into the size of the model’s vocabulary, producing a probability score for every token. The model then selects the highest-scoring token (or samples from the top candidates) to generate the next word.

flowchart LR
    Input[Input text] --> Tok[Tokenizer]
    Tok --> Emb[Token + Position Embeddings]
    Emb --> T0["Transformer Block h.0"]
    T0 --> T1["Transformer Block h.1"]
    T1 --> Tn["... h.N"]
    Tn --> Out[Output Linear Layer]
    Out --> Prob[Probability distribution over vocabulary]
    Prob --> NextTok[Next token selected/sampled]
    NextTok --> Tok2[Tokenizer decodes to text]

The customization spectrum. Where you start depends on your goals and resources:

flowchart TD
    A[Customization Spectrum] --> B[Prompt Engineering<br/>easiest]
    B --> C[Inference Parameters]
    C --> D[Retrieval-Augmented Generation]
    D --> E[Fine-tuning]
    E --> F[Architectural Changes<br/>most complex]

Fine-tuning Open-source LLMs

Fine-tuning takes a powerful pre-trained model — with broad language understanding from internet-scale training — and continues its training on a smaller, specialized dataset. This adapts the model’s knowledge to a specific domain: legal documents, medical journals, a company’s internal style guide, and so on.

Full fine-tuning. The traditional, straightforward approach: run additional training steps on the new dataset, updating every parameter in the model.

• Advantage: potential for the deepest performance gains, since every parameter can adapt to the nuances of the new data.
• Disadvantages: requires enormous GPU memory and compute — essentially the same as the original training, just for less time. It’s expensive and slow, and carries a significant risk of catastrophic forgetting, where the model becomes so specialized it loses general abilities (e.g., a legal expert that can no longer write a simple email).

Parameter-efficient fine-tuning (PEFT). To solve full fine-tuning’s problems, PEFT methods keep the pre-trained model frozen (read-only) and add a small number of new trainable parameters — only these are updated during training.

The most popular and effective PEFT method is LoRA (Low-Rank Adaptation). The key idea: the important changes needed during fine-tuning can be represented by a low-rank matrix — a small, efficient approximation. In practice, for each key layer in the transformer (like attention projections), a pair of small matrices is injected; these are the only trainable parameters, while the original massive weights remain untouched.

Because of these advantages, LoRA has become the default choice for most practical fine-tuning scenarios.

flowchart TD
    subgraph Full["Full Fine-tuning"]
        FA[All parameters updated] --> FB[High GPU/compute cost]
        FB --> FC[Risk: catastrophic forgetting]
    end
    subgraph PEFT["Parameter-Efficient Fine-tuning: LoRA"]
        PA[Base model weights: frozen] --> PB[Small LoRA adapter matrices injected]
        PB --> PC[Only adapter weights trained]
        PC --> PD[Tiny portable adapter file]
    end

Preparing the dataset. No matter the method, high-quality data is essential:

1. Format. For instruction-following tasks, the best format is a simple JSON structure with two fields: instruction (the prompt/task) and output (the desired completion).
2. Size. Fine-tuning doesn’t need nearly as much data as one might think, thanks to the model’s pre-existing knowledge. A few hundred high-quality examples can produce remarkable results; a few thousand is often enough for a significant transformation. LoRA is especially good at learning from smaller datasets.
3. Quality. The dataset must be accurate, consistent, and representative:
   • Accuracy — outputs must be correct; the model learns any bias or error present.
   • Consistency — if there are multiple valid ways to respond, the dataset should reflect a single preferred style (mixing formal and slang tones confuses the model).
   • Coverage — examples must cover the breadth of scenarios expected (e.g., a legal bot needs examples for contract review, legal advice, and disclaimer messages).

This process of building a high-quality dataset is often called curation, and typically involves domain experts writing and reviewing examples to meet the required standard.

flowchart LR
    A[Domain experts write examples] --> B[Format as instruction/output JSON]
    B --> C[Review for accuracy & consistency]
    C --> D[Ensure scenario coverage]
    D --> E[Curated fine-tuning dataset]
    E --> F[LoRA fine-tuning run]

Deploying LLMs on Your Own Infrastructure

Deploying an open-source model on your own infrastructure gives you:

• Full data privacy — conversations never leave your network.
• Reliability — no dependency on another service’s rate limits or downtime.
• Predictable costs — a fixed cost of ownership instead of surprise usage-based bills.

Just a few years ago, this was a complex task reserved for machine learning engineers. Today, thanks to accessible tooling, any developer can do it.

Ollama. A lightweight, powerful tool that creates a local server for LLMs. It handles loading the model, managing context, and providing a standard API.

# Install Ollama (single command for Mac, Linux, or Windows)
# See https://ollama.com for the platform-specific installer

# Pull and run a model interactively
ollama run llama3

Running ollama run llama3 downloads the model and drops you into an interactive chat session directly in the terminal.

Ollama also runs as a local server in the background, exposing an API endpoint at localhost by default. You can send requests to it with curl or from any application code:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Write a short poem about open-source AI.",
  "stream": false
}'

Open WebUI. While developers are comfortable with the command line, most users need a friendly interface. Open WebUI is an open-source web application that connects to an Ollama back end — essentially a private, self-hosted ChatGPT-style interface. It’s easily installed as a Docker container. Once running, you point your browser to it and get a familiar chat interface where you can select from locally downloaded models, adjust parameters like temperature, hold multi-turn conversations, use prompt templating, manage users, and upload documents for context.

Hardware requirements. For optimal performance you need a GPU — VRAM is the most important factor, since it’s the workspace where the model is loaded.

Ollama automatically pulls a version of the model optimized to run well on the available hardware, whether GPU or CPU, using quantization.

Scalability. For a small team, a single machine running Ollama might be sufficient. As demand grows:

• Set up Ollama on a dedicated, more powerful server, and point application code at the central server’s API instead of a local one.
• For larger scale, use orchestration tools like Kubernetes to containerize the model and scale replica counts up and down based on demand.

The flexibility of this approach lets you start small on a laptop for development, deploy to a single GPU server for production, and scale out to a cluster as needs grow.

flowchart TD
    A[Laptop / Dev Machine<br/>Ollama local] --> B[Single GPU Server<br/>Ollama as central API]
    B --> C[Kubernetes Cluster<br/>Containerized replicas, autoscaling]
    subgraph Stack["Local LLM Stack"]
        Ollama[Ollama server<br/>localhost:11434] --> OpenWebUI[Open WebUI<br/>Docker container]
        Ollama --> CustomApp[Custom app via curl / HTTP API]
    end

sequenceDiagram
    participant User
    participant OpenWebUI as Open WebUI
    participant Ollama
    participant Model as Local LLM (e.g. llama3)

    User->>OpenWebUI: Open chat interface in browser
    OpenWebUI->>Ollama: HTTP request (prompt, model, params)
    Ollama->>Model: Load weights (GPU/CPU), run inference
    Model-->>Ollama: Generated tokens
    Ollama-->>OpenWebUI: JSON response
    OpenWebUI-->>User: Rendered chat message

Industry-Specific Customization Demonstration: Building a Customer Service Chatbot

This walkthrough ties together inspection, fine-tuning, and deployment by building a customer service chatbot for a fictional company, Globomantics, from persona definition through data preparation, fine-tuning, and a Streamlit interface.

Step 1 — Define the bot’s persona. The chatbot should be helpful, patient, and professional, with a slightly friendly edge. It can:

• Explain subscription tiers for Globomantics.
• Guide users through basic troubleshooting (e.g., resetting a password).
• Escalate issues to a human agent.

It must never guess at internal company information or make financial guarantees, and must always defer to human agents for complex issues.

Step 2 — Build the training dataset. The persona directly informs the dataset design. Only a few hundred high-quality examples are needed for a LoRA fine-tune, using the instruction/output format introduced earlier. Example entry (login-issue handling for the Globomantics application):

{"instruction": "I can't log into my Globomantics account, it says my password is incorrect.", "output": "I'm sorry to hear you're having trouble logging in! Let's get this sorted out. You can reset your password by clicking 'Forgot Password' on the login screen and following the instructions sent to your registered email. If you continue to have issues after resetting, I can connect you with a support specialist who can look into your account further."}

If there is no proprietary information the chatbot needs to know, it’s often better to handle behavior via system prompts rather than fine-tuning — the base LLM already has the necessary general knowledge and doesn’t need retraining for generic cases. Fine-tuning is reserved for teaching the model company- or domain-specific knowledge and response patterns.

Once assembled, the dataset file is saved, e.g. as globomantics_training_data.jsonl.

Step 3 — Set up the fine-tuning environment. Prerequisites: a machine with an Nvidia GPU (at least 16 GB VRAM for an 8B model) or a cloud GPU instance (Lambda Labs, RunPod, AWS), plus Python.

# Create a clean Python environment
python -m venv globomantics-finetune-env
source globomantics-finetune-env/bin/activate  # on Windows: globomantics-finetune-env\Scripts\activate

# Install Unsloth and required libraries
pip install unsloth torch datasets trl transformers

Step 4 — The fine-tuning script. The script below (from the course’s demo files) loads the base model with Unsloth optimization (up to 4x faster loading, ~70% less memory), adds LoRA adapters, loads the JSON training data, formats it for instruction-tuning, configures training arguments, and trains:

from unsloth import FastLanguageModel
import torch
from datasets import load_dataset
from trl import SFTTrainer
from transformers import TrainingArguments

# Load the base model with Unsloth optimization
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/Meta-Llama-3.1-8B",
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
)

# Add LoRA adapters - tells Unsloth which layers to adapt for LoRA
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", # Target attention layers
                      "gate_proj", "up_proj", "down_proj",], # Target FFN layers
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_gradient_checkpointing = True,
    random_state = 3407,
)

# Load the dataset (the JSON file prepared previously)
dataset = load_dataset("json", data_files="globomantics_training_data.jsonl", split="train")

# Format the training data - crucial for instruction tuning
def formatting_prompts_func(examples):
    instructions = examples["instruction"]
    outputs = examples["output"]
    texts = []
    for instruction, output in zip(instructions, outputs):

        text = f"<|start_header_id|>user<|end_header_id|>\n\n{instruction}<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n{output}<|eot_id|>"
        texts.append(text)
    return { "text" : texts, }
dataset = dataset.map(formatting_prompts_func, batched = True)

# Configure training arguments
training_args = TrainingArguments(
    output_dir = "outputs",
    per_device_train_batch_size = 2,
    gradient_accumulation_steps = 4,
    warmup_steps = 5,
    num_train_epochs = 3,
    learning_rate = 2e-4,
    fp16 = not torch.cuda.is_bf16_supported(),
    bf16 = torch.cuda.is_bf16_supported(),
    logging_steps = 1,
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407,
    save_strategy = "epoch",
)

# Initialize the trainer and train the model
trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "text",
    max_seq_length = 2048,
    args = training_args,
)

trainer.train()

# Save the LoRA adapter (a small safetensors file, not the full model)
model.save_pretrained("globo_assist_lora_adapter")

The last step saves only the LoRA adapter as a small .safetensors file — not a full copy of the model.

Step 5 — Load the fine-tuned adapter into Ollama. Create a Modelfile telling Ollama how to combine the base model with the new adapter:

FROM llama3.1:8b
ADAPTER ./globo_assist_lora_adapter/adapter_model.safetensors

SYSTEM """
You are GlomoAssist, the official customer support assistant for Globomantics.
You explain subscription tiers, help troubleshoot common issues like password resets,
and escalate anything complex to a human support agent. Never guess at internal
company information or make financial guarantees.
"""

PARAMETER temperature 0.1

The adapter path points to the generated safetensors file, and the FROM line specifies the base model. The SYSTEM block defines the chatbot’s name and the key information/behavior it should follow. The temperature parameter dictates creativity, where 0 is least creative — keeping it close to 0 limits hallucinations.

# Build the custom model in Ollama from the Modelfile
ollama create globo-assist -f Modelfile

This produces a custom model called globo-assist running locally, ready to be used from Open WebUI or a custom interface.

Step 6 — Build a custom chat interface with Streamlit. Rather than only using Open WebUI, a bespoke interface can be built:

import streamlit as st
import requests

st.set_page_config(page_title="GloboAssist", page_icon="🤖")
st.title("GloboAssist Support Bot")

OLLAMA_URL = "http://localhost:11434/api/generate"

if "messages" not in st.session_state:
    st.session_state.messages = []

for message in st.session_state.messages:
    with st.chat_message(message["role"]):
        st.markdown(message["content"])

if prompt := st.chat_input("How can I help you today?"):
    st.chat_message("user").markdown(prompt)
    st.session_state.messages.append({"role": "user", "content": prompt})

    request_data = {
        "model": "globo-assist",
        "prompt": prompt,
        "stream": False
    }

    try:
        response = requests.post(OLLAMA_URL, json=request_data)
        response.raise_for_status()
        full_response = response.json()
        assistant_msg = full_response['response']

    except Exception as e:
        assistant_msg = f"Sorry, I'm experiencing a technical issue: {e}"

    with st.chat_message("assistant"):
        st.markdown(assistant_msg)
    st.session_state.messages.append({"role": "assistant", "content": assistant_msg})

Walking through the code: st.set_page_config/st.title set the page title and icon (translated by Streamlit into HTML). The app connects to the local Ollama API, initializes and displays chat history, reacts to user input by rendering it in a chat message container and appending it to history, then prepares the request payload — noting that "model": "globo-assist" targets the fine-tuned model specifically. The request is sent to the locally running Ollama server, and the response is displayed in its own chat message container.

# Run the Streamlit interface (not a plain python command)
streamlit run interface.py

The result: a general-purpose model transformed into a specialized, compliant, and helpful customer service agent for a specific industry use case.

flowchart TD
    A[1. Define persona & guardrails] --> B[2. Build instruction/output JSON dataset]
    B --> C[3. Set up Python + GPU environment]
    C --> D[4. Fine-tune base model with Unsloth + LoRA]
    D --> E[5. Create Ollama Modelfile with adapter + system prompt]
    E --> F[6. ollama create globo-assist]
    F --> G[7. Build Streamlit chat UI]
    G --> H[Deployed: globo-assist customer service bot]

Module 3: Surveying Leading Open-source LLMs and Their Use Cases

Llama, BLOOM, DeepSeek, and Mistral Overview

Picking the wrong open-source model can mean wasted time, higher costs, and a solution that doesn’t fit the actual need. Here’s an overview of the leading model families:

Llama (Meta). The Llama family, currently at Llama 4, is not fully open source in the traditional sense due to its custom commercial license — but its open-weight approach has been revolutionary. Meta releases the weights for research and commercial use, but access requires agreeing to their terms (notably restricting entities with more than 700M monthly active users, per the licensing note above). Llama sets a high bar for general-purpose performance: versatile and strong across writing, reasoning, and many other tasks, and has become the base for countless fine-tuned derivative models.

BLOOM (BigScience workshop). A truly open-source, collaborative, community-driven project involving over 1,000 researchers (unlike Llama, which came from a single tech giant). Its key differentiator is massive multilingual coverage — trained on a dataset spanning 46 human languages, making it a powerhouse for global applications. It uses a permissive Responsible AI License (RAIL), which promotes open use but prohibits certain harmful applications. While it may not top every benchmark against newer models, BLOOM remains important as a transparent, internationally focused alternative to corporate-backed projects.

DeepSeek. This family has aggressively pushed the boundaries of open-weight models. The base DeepSeek models (e.g. the 67-billion-parameter version) have outperformed much larger models like Llama 4 70B on reasoning, coding, and math benchmarks, while also having native strength in Chinese. DeepSeek uses a custom commercial license.

Mistral AI (Mistral 7B). Emerging from France, Mistral competed on efficiency rather than sheer size. Its 7-billion-parameter model was designed to be smaller, faster, and cheaper to run — yet it outperformed models twice its size, like Llama 2 13B, across the board. A key to its rapid adoption is licensing: Mistral 7B uses the Apache 2.0 license, one of the most permissive and commercially friendly licenses available, making it attractive for businesses that want to self-host and modify models without legal complexity.

Each project emerged from a different background with different goals, and licensing reflects those philosophies — understanding why a model was built and for whom is the first step in choosing the right one.

mindmap
  root((Leading Open-source LLM Families))
    Llama - Meta
      Open-weight, custom license
      General-purpose versatility
      Huge fine-tuning community
    BLOOM - BigScience
      Truly community-driven
      46-language multilingual focus
      RAIL license
    DeepSeek
      Outperforms larger models on benchmarks
      Native Chinese strength
      Custom commercial license
    Mistral AI
      Efficiency over size
      Apache 2.0 license
      Business-friendly

Architecture, Scale, and Benchmarks

Architectural innovations. The classic transformer architecture (used by Llama 4 and BLOOM) is proven and powerful, but newer models introduce tweaks for efficiency and specialization:

• Mistral 7B uses two key techniques:

  • Sliding window attention — each token attends only to a subset of previous tokens rather than the entire sequence, processing long sequences more efficiently.
  • Grouped-query attention — reduces the memory footprint during inference without significant performance loss. These innovations explain why a 7B model can compete with models nearly twice its size.

• DeepSeek v3 employs a mixture-of-experts (MoE) architecture — not a single massive engine, but a team of specialized experts. For any input, a routing network selects only a few relevant expert subnetworks to activate. Total parameter count is 671 billion, but only ~37 billion are activated per token — maintaining the knowledge capacity of an enormous model while achieving the speed/cost profile of a much smaller one at inference time.

flowchart TB
    subgraph Sliding["Mistral 7B: Sliding Window Attention"]
        S1[Token N] --> S2["Attends only to a window of recent tokens"]
    end
    subgraph MoE["DeepSeek v3: Mixture of Experts"]
        M1[Input token] --> M2[Routing network]
        M2 --> M3[Expert 1]
        M2 --> M4[Expert 2]
        M2 -.->|not activated| M5[Expert N]
        M3 --> M6[Combined output]
        M4 --> M6
    end

Scale (parameter counts).

Raw size alone is a poor indicator of performance — architectural and training-data quality often matter more.

Benchmarks. Standardized tests measuring proficiency in reasoning, coding, and language understanding:

Results are often surprising: the DeepSeek 67B base model, despite having fewer parameters, has outperformed the much larger Llama 4 70B model on key benchmarks — demonstrating superior performance in complex reasoning, math, and code generation. This shows that training data quality and architectural efficiency can matter more than sheer parameter count. Similarly, when Mistral 7B was released, it outperformed the larger Llama 2 13B across all tested benchmarks — a clear demonstration that a smarter, more optimized architecture can achieve more with less. This reinforces that, for many practical applications, a smaller well-designed model beats a larger, more generic one.

xychart-beta
    title "Illustrative benchmark comparison (higher is better)"
    x-axis ["Mistral 7B", "Llama 2 13B", "DeepSeek 67B", "Llama 4 70B"]
    y-axis "Relative benchmark score" 0 --> 100
    bar [72, 60, 88, 80]

Specialized Models and Optimal Use Cases

Beyond general-purpose base models, the open-source ecosystem includes models fine-tuned for specific high-impact tasks:

DeepSeek-R1 — advanced reasoning. Not a single model but a series (including R1 and the more recent R10). What sets these apart is the training methodology: instead of being trained solely on next-token prediction, they use reinforcement learning, specifically group relative policy optimization (GRPO). In simpler terms, they were taught to think through problems step by step, much like a human, before arriving at an answer — making them exceptionally strong at complex logical puzzles, mathematical problems, and planning tasks. Their primary strength lies in research and applications requiring chain-of-thought reasoning rather than general chat.

DeepSeek Coder v2 — software development. A code-specialized model applying the mixture-of-experts architecture directly to programming. It supports 338 programming languages and features an extended 128,000-token context window, enabling it to process entire codebases rather than small snippets. Benchmarks show it rivaling GPT-4 Turbo’s coding capabilities for code completion, bug fixing, generating entire functions from natural-language descriptions, and refactoring large code sections — essentially a dedicated AI software engineer.

Mistral 7B — practical deployment. Its optimal use case is defined not by a single task like reasoning or coding but by a set of constraints: efficiency, speed, and legal simplicity. Its lightweight, efficient architecture runs effectively on a single consumer-grade GPU or even a powerful CPU. Its killer feature for businesses is the Apache 2.0 license — no usage restrictions, no requirement to disclose modifications, and no legal ambiguity — making it the go-to choice for companies that want to self-host, embed into commercial products, or modify for internal tools without licensing complications.

flowchart TD
    Start{What's the primary requirement?} -->|Multi-step reasoning| R1[DeepSeek-R1]
    Start -->|Software development / coding| Coder[DeepSeek Coder v2]
    Start -->|Speed, low cost, legal simplicity| Mistral[Mistral 7B]

Model Strengths, Weaknesses, and When to Use Each

Llama 4 — the reliable all-arounder.

• Strengths: versatility and strong general-purpose performance across writing, summarization, and question-answering; a massive community means abundant fine-tuned variants and support.
• Weaknesses: licensing — the custom Meta license, while allowing commercial use, requires agreeing to specific terms and requesting access, adding administrative friction versus fully open-source alternatives; not the top performer in any single specialized domain like coding or reasoning.
• Use it when: you need a proven jack-of-all-trades model for general-purpose applications and the licensing terms are acceptable for your organization.

BLOOM — the multilingual community project.

• Strengths: performance across 46 languages, making it excellent for truly global applications; as a community-driven project it holds a special place in the open-source ethos for transparency.
• Weaknesses: as an earlier model, overall performance on standard reasoning/coding benchmarks may lag newer, more specialized models; its RAIL license is open but includes use-case restrictions aimed at preventing harm.
• Use it when: the primary requirement is robust, equitable multilingual support and alignment with open/responsible AI principles.

DeepSeek — the high-performance challenger.

• Strengths: excels at reasoning, mathematics, coding, and Chinese-language comprehension; models like DeepSeek 67B outperform much larger competitors on these specific tasks.
• Weaknesses: accessibility and deployment complexity — larger models require significant compute, and mixture-of-experts models, while efficient at inference, add deployment complexity; licensing is a custom commercial license similar to Llama’s.
• Use it when: project success hinges on top-tier performance in reasoning or coding, and you have the infrastructure to support it.

Mistral 7B — the efficiency and commercial-ready expert.

• Strengths: best-in-class performance for its size, delivering strong capability in a small, fast, cost-effective package; the Apache 2.0 license is the gold standard for permissive, commercially friendly open source, making it the safest and easiest choice for integration, self-hosting, and modification without legal concerns.
• Weaknesses: scale — while efficient, it may not match the depth of knowledge or capability of much larger 70B+ models for highly complex tasks.
• Use it when: priorities are speed to market, low operational cost, and commercial simplicity — e.g., internal chatbots, content generation, or prototyping.

Three critical trade-offs to weigh:

1. Compute vs. capability. Need the absolute highest capability for a specialized task regardless of cost? A large DeepSeek model may be the answer. Constrained by budget and hardware? Mistral 7B’s efficiency is likely the best path.
2. Deployment complexity vs. specialization. A general-purpose model like Llama 4 is relatively straightforward to deploy. A massive mixture-of-experts model like DeepSeek-V3, while powerful, requires more sophisticated engineering — assess whether your team has the expertise to manage that complexity.
3. Licensing vs. openness. Mistral 7B’s Apache 2.0 license offers the highest level of commercial freedom. Llama’s and DeepSeek’s custom licenses are more restrictive but still allow commercial use. BLOOM’s RAIL license promotes openness but with specific ethical constraints. Legal and compliance teams are key stakeholders in this decision.

flowchart TD
    Q1{General chat, multilingual,\ncomplex reasoning, or code?} --> Q2{What is your compute budget?}
    Q2 --> Q3{What is your team's\ndeployment expertise?}
    Q3 --> Q4{Which license fits your\ncompany's policies?}
    Q4 --> Decision[Final model selection]

There is no single “best” open-source LLM — only the best model for a given set of requirements, budget, expertise, and licensing constraints.

Summary

Open-source large language models have evolved from early source-available releases (BLOOM, Llama) into a mature ecosystem of permissively licensed, highly capable models (Mistral, Qwen, DeepSeek, Gemma) that rival or exceed closed-source alternatives on many benchmarks. Their core value proposition rests on four pillars: accessibility (downloadable weights), transparency (open training/benchmark data), customizability (fine-tuning and architectural modification), and community-driven evolution.

For businesses and developers, self-hosting open-source models delivers greater behavioral control, stronger data privacy through on-premises deployment, better long-term economics at scale, and faster iteration cycles — at the cost of upfront infrastructure and technical investment.

Working with an open-source model in practice means understanding its three core downloaded assets (weights, config, tokenizer), knowing how to inspect and modify them, and applying parameter-efficient fine-tuning (LoRA) with a small, curated instruction dataset to specialize a general model for a specific domain — as demonstrated end-to-end with the Globomantics customer service chatbot (Unsloth + LoRA fine-tuning → Ollama Modelfile → Streamlit interface).

Finally, selecting the right model family — Llama, BLOOM, DeepSeek, or Mistral, and their specialized variants like DeepSeek-R1 or DeepSeek Coder v2 — requires balancing compute cost against capability, deployment complexity against specialization, and licensing against openness.

Quick-reference Table

Checklist for Adopting an Open-source LLM

• Define the use case and required capability (general chat, reasoning, coding, multilingual).
• Determine data-sensitivity and compliance requirements that may mandate self-hosting.
• Estimate available compute budget (GPU/VRAM or CPU/RAM) and match it to a model’s parameter count/quantization needs.
• Review each candidate model’s license (Apache 2.0, custom commercial, RAIL, MIT) against legal/compliance policy.
• Inspect the model’s config.json, weights, and tokenizer to understand its architecture before deployment.
• If specialization is needed, curate a small, high-quality instruction dataset and fine-tune with LoRA rather than full fine-tuning.
• Deploy via Ollama (or an equivalent local server) and expose it through Open WebUI or a custom interface.
• Plan for scale: single machine → dedicated server → Kubernetes-orchestrated cluster as demand grows.

Search Terms

open-source · llms · llm · application · development · artificial · intelligence · generative · ai · architecture · cases · customization · deploying