Developing AI Agents with OpenAI AgentKit

Ongoing project: Globobot — support assistant for GloboTicket

Table of Contents

1. Introduction — From Prompts to Agents
2. AgentKit Definition and Core Components
3. MCP — Model Context Protocol
4. Building the First Agent with Agent Builder
5. Tool Calling — Detect → Act → Respond
6. Deployment — ChatKit vs ChatGPT Apps
7. Reliability — Tracing, Debugging and Tuning
8. Guardrails — See, Do, Say
9. Governance and Security at Scale
10. Evals — Automated Quality Testing
11. Reference Tables
12. Complete Architecture — Diagrams

1. Introduction — From Prompts to Agents

The Fundamental Paradigm Shift

Most developers start with a prompt. It works for a simple question, but as soon as a real workflow appears — refunds, approvals, triage — the prompt alone falls apart.

“Prompts produce answers. Agents produce outcomes.”

Why Prompts Are Not Enough

The GloboTicket team learned this the hard way with Globobot v0:

• One prompt fixed → something else broke
• Unable to handle a multi-step process (lookup → verify policy → confirm → issue refund)
• No memory between turns
• No real API calls

The solution: An agent is like an employee. It stays in the loop, makes decisions step by step, calls tools when needed, and continues until the work is truly done.

2. AgentKit Definition and Core Components

The AgentKit Loop: Build → Deploy → Optimize

flowchart LR
    B["Build\n(Workflows + Tools\n+ Instructions + Guardrails)"]
    D["Deploy\n(ChatKit or\nChatGPT App)"]
    O["Optimize\n(Tracing + Evals\n+ Iteration)"]
    B --> D --> O --> B

Core Components

UI ↔ Workflow Architecture

User ──► ChatKit (UI surface)
              │
              ▼
        Workflow Backend
         (Agent + Tools)
              │
              ▼
         Tool Results ──► Response to User

Golden rule: The UI is the front, the workflow is the back. The user types, the UI sends, the workflow reasons + acts, the UI displays the result.

3. MCP — Model Context Protocol

Definition

MCP (Model Context Protocol) is an open protocol that standardizes how AI agents communicate with tools. It is the USB-C of AI: a universal standard, many compatible tools.

Without MCP vs With MCP

MCP Tool Schema (example: lookup_ticket)

LOOKUP_TICKET_SCHEMA = {
    "type": "function",
    "function": {
        "name": "lookup_ticket",
        "description": "Look up a ticket by its ID and return status, email, notes, and refund state.",
        "parameters": {
            "type": "object",
            "properties": {
                "ticket_id": {
                    "type": "string",
                    "description": "The ticket identifier (e.g. ABC-12345 or TICK1001)"
                }
            },
            "required": ["ticket_id"],
            "additionalProperties": False
        }
    }
}

additionalProperties: False — Prevents the model from inventing extra fields. Strict contract = predictable behavior.

MCP Flow: Request → Execute → Return

sequenceDiagram
    participant User
    participant Agent
    participant Runtime
    participant Tool

    User->>Agent: "What is the status of TICK1001?"
    Agent->>Runtime: function_call: lookup_ticket({ticket_id: "TICK1001"})
    Runtime->>Tool: Execute lookup_ticket("TICK1001")
    Tool-->>Runtime: {found: true, status: "Open", email: "c***@example.com"}
    Runtime-->>Agent: tool_output (structured JSON)
    Agent-->>User: "The status of ticket TICK1001 is Open."

4. Building the First Agent with Agent Builder

Visual Approach (Agent Builder)

1. Create a new project (e.g., “Globobot Agents”)
2. Open Agent Builder → New Workflow
3. Configure the main node with system instructions
4. Test with Preview before adding tools

Starting Node — Globobot v1 System Prompt

You are Globobot, a friendly support assistant for GloboTicket.

Your job:
- Help users with ticket inquiries.
- Ask clarifying questions when needed.
- Be concise and professional.

At this stage: The agent maintains conversational context, handles follow-ups, without needing to code message management. This is the immediate value of an agent vs a standalone prompt.

Key Observations After This First Demo

• ✅ The session maintains context automatically
• ✅ The agent asks clarifying questions
• ❌ No tools → cannot verify a real ticket
• ➡️ Next step: add tools

5. Tool Calling — Detect → Act → Respond

The Data Model: Ticket and TicketStore

# domain.py
from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Dict, Optional

@dataclass
class Ticket:
    id: str
    status: str
    issue_description: str
    email: str
    refunded: bool = False
    updated_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))

@dataclass
class TicketStore:
    tickets: Dict[str, Ticket] = field(default_factory=dict)

    def get_ticket(self, ticket_id: str) -> Optional[Ticket]:
        return self.tickets.get(ticket_id)

def seed_store() -> TicketStore:
    store = TicketStore()
    store.tickets["ABC-12345"] = Ticket(
        id="ABC-12345",
        status="Open",
        issue_description="Package lost in transit.",
        email="customer1@example.com",
    )
    store.tickets["DEF-67890"] = Ticket(
        id="DEF-67890",
        status="Resolved",
        issue_description="Duplicate order corrected.",
        email="customer2@example.com",
    )
    return store

Detect → Act → Respond Pattern

flowchart TD
    A[User message received] --> B{Detect:\nTicket ID present?}
    B -- No --> C[Ask for ticket ID\nTools disabled]
    B -- Yes --> D[Act:\nActivate tools\nCall lookup_ticket]
    D --> E[Respond:\nReply with structured data]
    C --> A

Tool Gating Configuration

# agent.py — Workflow routing
import re
from typing import Optional

TICKET_PATTERN = re.compile(r"\b([A-Z]{3}-\d{5}|TICK\d+)\b", re.IGNORECASE)

def detect_ticket_id(message: str) -> Optional[str]:
    """DETECT step: extracts the ticket ID from the user message."""
    match = TICKET_PATTERN.search(message)
    if match:
        return match.group(1).upper()
    return None

# In the main loop:
ticket_id = detect_ticket_id(user_message)

if GATE_TOOLS and ticket_id is None:
    # Tools disabled — the agent MUST ask for the ID
    tools = []
else:
    # Tools enabled — the agent can call lookup_ticket
    tools = TOOL_SCHEMAS

Key principle: If GATE_TOOLS is active and no ticket ID is detected, the tools are literally absent from the context. The model cannot invent a call. This is workflow control, not prompt engineering.

System Prompt with Strict Tool-Use Policy

GLOBOBOT_SYSTEM_PROMPT = """
You are Globobot, a friendly support assistant for GloboTicket.

Your job:
- Help users with ticket inquiries.
- Look up ticket details and status using tools when they provide a ticket ID.
- Refund tickets only when asked and a ticket ID is provided.
- Ask clarifying questions when needed.

Tool-use policy (strict):
- For ANY question about a ticket's status, issue description, or refund eligibility,
  you MUST call lookup_ticket with the ticket_id before answering.
- Only answer using the tool output. Do not answer from memory or prior conversation.
- If the ticket_id is missing, ask for it.
- If you have not called lookup_ticket in this turn, respond ONLY with:
  "I need to look that up. Calling lookup_ticket now."

Rules:
- NEVER assume ticket IDs.
- If the ticket is not found, say: "I could not find ticket {TICKET_ID}."
- If a refund succeeds, say: "Ticket {TICKET_ID} has been refunded successfully."
- When reporting status, say: "The status of ticket {TICKET_ID} is {STATUS}."
- Be concise and helpful.
"""

ToolRuntime — Deterministic Execution with Cache

# tools.py
class ToolRuntime:
    """
    Deterministic runtime with logs + caching.
    
    - cache_hit logging to prove caching improvements
    - toggle schema mismatch for payload inspection demos
    """

    def __init__(self, settings: Settings) -> None:
        self.settings = settings
        self.store = seed_store()
        self.cache: Dict[str, Dict[str, Any]] = {}
        self.logs: List[Dict[str, Any]] = []
        self.metrics: Dict[str, Any] = {
            "tool_calls": 0,
            "tool_calls_by_tool": {},
            "blocked": 0,
            "errors": 0,
            "moderation_total": 0,
            "moderation_flagged": 0,
            "feedback_scores": [],
        }

GlobobotReliableAgent — Main Class

# agent.py
from openai import OpenAI
from .config import Settings
from .tools import TOOL_SCHEMAS, ToolRuntime, dispatch_tool_call

class GlobobotReliableAgent:
    def __init__(self, runtime: ToolRuntime, settings: Settings) -> None:
        self.runtime = runtime
        self.settings = settings
        self.client = OpenAI(api_key=settings.openai_api_key)
        self.model = settings.model_name
        self.temperature = settings.temperature

        prompt = GLOBOBOT_SYSTEM_PROMPT.strip()
        if settings.enable_prompt_reuse:
            prompt = f"{prompt}\n\n{PROMPT_REUSE_HINT}"
        else:
            prompt = f"{prompt}\n\n{PROMPT_REUSE_DISABLED_HINT}"

        self.messages: List[Dict[str, Any]] = [{"role": "system", "content": prompt}]
        self.last_ticket_context: Dict[str, Dict[str, Any]] = {}
        self.last_ticket_id: Optional[str] = None

6. Deployment — ChatKit vs ChatGPT Apps

Separating Agent Runtime from Experience Surface

graph LR
    subgraph "Agent Runtime (brain + hands)"
        R[Responses API] --- T[Tools]
        R --- S[State / Session]
    end
    subgraph "Experience Surface (where humans interact)"
        CK["ChatKit\n(your app)"]
        GPT["ChatGPT App\n(apps SDK)"]
    end
    CK --> R
    GPT --> R

Decision: ChatKit or ChatGPT App?

Practical rule: For your team or product → ChatKit. For a wider audience already in ChatGPT → ChatGPT App. Both can coexist because the runtime is identical.

Security with ChatKit — Server-Side Boundary

# api.py — The critical security boundary

async def create_chatkit_session():
    """
    ONLY place that talks to OpenAI on the server side.
    - The API key NEVER leaves the server
    - We only return the client_secret (ephemeral)
    - No agent logic here
    """
    response = openai_client.chatkit.sessions.create(
        workflow_id=settings.workflow_id
    )
    return {
        "client_secret": response.client_secret,
        "expires_at": response.expires_at
    }
    # ⚠️ At no point does the browser receive the OpenAI API key

Capability Reuse Pattern (2 UIs, 1 backend)

graph TD
    ChatKit["ChatKit\n(conversational lane)"] --> ChatEndpoint["/chat endpoint\n(agent logic + tools)"]
    Button["'Check Ticket' button\n(workflow shortcut)"] --> ChatEndpoint
    ChatEndpoint --> AgentLogic["Agent reasons\ndecides to call tools\nreturns result"]

Concept: The “Check Ticket” button does not call a tool directly — it sends a focused intent to the same agent workflow. Same backend, two different UIs. No logic duplication.

FastAPI Endpoint — Server Architecture

# mcp_server.py
from fastapi import FastAPI
from .agent import GlobobotReliableAgent
from .config import get_settings
from .tools import ToolRuntime

settings = get_settings()
runtime = ToolRuntime(settings)
agent = GlobobotReliableAgent(runtime=runtime, settings=settings)

# Optional session isolation
session_agents: dict[str, GlobobotReliableAgent] = {}

app = FastAPI(title="Globobot Reliable", version="3.0")

@app.post("/chat")
def chat(payload: dict) -> dict:
    message = str(payload.get("message", ""))
    session_id = payload.get("session_id")

    if isinstance(session_id, str) and session_id.strip():
        session_key = session_id.strip()
        if session_key not in session_agents:
            session_agents[session_key] = GlobobotReliableAgent(
                runtime=runtime, settings=settings
            )
        return {
            "response": session_agents[session_key].chat(message),
            "session_id": session_key
        }

    return {"response": agent.chat(message)}

7. Reliability — Tracing, Debugging and Tuning

The Three Symptoms of Degraded Reliability

1. Sluggish responses — Fast one moment, slow the next
2. Redundant actions — Same tool called multiple times for the same data
3. Confusing answers — Not necessarily wrong, but inconsistent enough to erode trust

The Triage Map

flowchart TD
    Problem[Problem detected] --> Q1{Bad response?}
    Q1 -- Yes --> T[Inspect Traces\nstep-by-step]
    Q1 -- No --> Q2{Slow response?}
    Q2 -- Yes --> L[Inspect Logs\ntiming + cache_hits]
    Q2 -- No --> Q3{Strange data?}
    Q3 -- Yes --> P[Inspect Payloads\nexact JSON sent/received]

The 3 Observability Artifacts

Debug Endpoints (FastAPI)

# mcp_server.py — Observability endpoints

@app.get("/debug/traces")
def debug_traces(session_id: str | None = None) -> dict:
    """Step-by-step traces by session or global."""
    if isinstance(session_id, str) and session_id.strip():
        session_key = session_id.strip()
        if session_key in session_agents:
            return {"traces": session_agents[session_key].traces, "session_id": session_key}
        return {"traces": [], "session_id": session_key, "error": "unknown session_id"}
    
    all_sessions = {key: sess.traces for key, sess in session_agents.items()}
    return {"traces": agent.traces, "sessions": all_sessions}

@app.get("/debug/logs")
def debug_logs() -> dict:
    """Runtime logs with timing and cache hits."""
    return {"logs": runtime.logs}

@app.get("/debug/metrics")
def debug_metrics() -> dict:
    """Aggregated metrics: tool_calls, blocked, errors, moderation."""
    return runtime.get_metrics()

@app.get("/debug/audit")
def debug_audit(limit: int = 50) -> list[dict]:
    """Audit log of destructive actions (JSONL)."""
    path = settings.audit_log_path
    if not path:
        return []
    try:
        with open(path, "r", encoding="utf-8") as handle:
            lines = handle.readlines()
        return [json.loads(line) for line in lines[-limit:] if line.strip()]
    except FileNotFoundError:
        return []

Fixing the Redundant Tool Call Problem: The Cache

# config.py — Cache toggle
@dataclass
class Settings:
    enable_cache: bool = True
    cache_invalidation_on_refund: bool = True  # Invalidates cache after a refund
    # ...

# tools.py — ToolRuntime with cache
class ToolRuntime:
    def __init__(self, settings: Settings) -> None:
        self.cache: Dict[str, Dict[str, Any]] = {}

    def lookup_ticket(self, ticket_id: str) -> Dict[str, Any]:
        # Check cache first
        if self.settings.enable_cache and ticket_id in self.cache:
            self._log_event({"event": "tool_call", "tool": "lookup_ticket",
                            "ticket_id": ticket_id, "cache_hit": True, "duration_ms": 0})
            return self.cache[ticket_id]

        # Otherwise, real call (with simulated latency for demos)
        start = time.time()
        ticket = self.store.get_ticket(ticket_id)
        duration = int((time.time() - start) * 1000)

        if ticket is None:
            result = {"found": False}
        else:
            result = {
                "found": True, "status": ticket.status,
                "email": ticket.email, "refunded": ticket.refunded,
                "issue_description": ticket.issue_description
            }

        # Store in cache
        if self.settings.enable_cache:
            self.cache[ticket_id] = result

        self._log_event({"event": "tool_call", "tool": "lookup_ticket",
                        "ticket_id": ticket_id, "cache_hit": False,
                        "duration_ms": duration})
        return result

Expected Result After Enabling Cache

# Before: 2 lookup_ticket calls, ~20ms each, cache_hit: false
# After: 1st call ~20ms, 2nd call 0ms, cache_hit: true

Evidence from logs:
{"tool": "lookup_ticket", "ticket_id": "ABC-12345", "cache_hit": false, "duration_ms": 22}
{"tool": "lookup_ticket", "ticket_id": "ABC-12345", "cache_hit": true,  "duration_ms": 0}

Fundamental lesson: You cannot ask a model to be efficient with a prompt. If reliability (performance, cost, latency) matters, the guarantee must be in the system, not in the prompt.

Environment Variables for Tuning

8. Guardrails — See, Do, Say

The Mindset Shift: Reliability → Safety

Reliability = “Did it work?”
Safety = “What damage is possible?”

Once an agent can see customer data, take real actions, and speak on behalf of your product, security becomes non-negotiable.

The 3 Risk Categories

See, Do, Say Framework

graph TD
    subgraph "SEE — What the agent can access"
        S1[Reject malformed ticket IDs]
        S2[Treat tool output as untrusted data]
        S3[Block prompt injection attempts]
    end
    subgraph "DO — What the agent can do"
        D1["Environment gating\n(dev vs prod)"]
        D2["Authorization\n(API key / service identity)"]
        D3["User approval\n(explicit confirmation)"]
    end
    subgraph "SAY — What the agent can output"
        Y1[Redaction — mask PII]
        Y2[Moderation — screen before sending]
    end

The 4 Guardrail Layers (weakest to strongest)

Layer 1: Prompt-level rules           → Helpful but NOT deterministic
Layer 2: Tool boundary controls       → Authorization + environment gating
Layer 3: Runtime enforcement          → Agent refuses unsafe tool calls
Layer 4: Governance                   → Audit logs, rollbacks, evals

Golden rule: Prompts steer, systems enforce.
The deeper the layer, the more it is trusted.

Settings — All Security Toggles

# config.py
@dataclass
class Settings:
    openai_api_key: str
    model_name: str = "gpt-4o-mini"
    temperature: float = 0.2

    # Cache and performance
    enable_cache: bool = True
    cache_invalidation_on_refund: bool = True
    enable_prompt_reuse: bool = True

    # Safety guardrails
    globo_env: str = "dev"                     # "dev" | "prod"
    refund_api_key: str | None = None          # Key required for prod refunds
    enable_safety_guardrails: bool = False     # Master security toggle
    enable_output_moderation: bool = False     # Output moderation
    enable_redaction: bool = False             # PII masking
    enable_audit_log: bool = False             # JSONL audit trail
    audit_log_path: str = "./audit.jsonl"
    moderation_model: str = "omni-moderation-latest"
    enable_data_sharing_prompt: bool = False   # Consent gate for data access
    enable_refund_tool: bool = True            # Refund kill switch

Secure Refund Tool — Fail Closed

# tools.py — Refund with all security layers

def refund_ticket(self, ticket_id: str, api_key: str | None = None) -> Dict[str, Any]:
    """
    Destructive action — must fail closed.
    Fail closed = if anything is wrong, block by default.
    """
    # Layer 1: Kill switch
    if not self.settings.enable_refund_tool:
        return {"error": "refund_disabled", "message": "Refund capability is disabled."}

    # Layer 2: Environment gating + Authorization
    if self.settings.enable_safety_guardrails:
        expected_env = "prod"
        if self.settings.globo_env != expected_env:
            return {"error": "wrong_environment",
                    "message": f"Refunds require environment '{expected_env}'."}
        if not api_key or api_key != self.settings.refund_api_key:
            return {"error": "unauthorized",
                    "message": "Valid API key required for refunds."}

    # Layer 3: Runtime enforcement — explicit confirmation required
    # (verified in agent.py before calling this tool)

    # Execute the refund if all conditions are met
    ticket = self.store.get_ticket(ticket_id)
    if ticket is None:
        return {"found": False}
    if ticket.refunded:
        return {"found": True, "already_refunded": True, "status": ticket.status}

    ticket.refunded = True
    ticket.status = "Refunded"

    # Invalidate cache
    if self.settings.cache_invalidation_on_refund:
        self.cache.pop(ticket_id, None)

    # Layer 4: Audit log
    if self.settings.enable_audit_log:
        self.write_audit_event({
            "event": "tool_call", "tool": "refund_ticket",
            "ticket_id": ticket_id, "result": "success", "allowed": True
        })

    return {"found": True, "success": True, "ticket_id": ticket_id, "status": "Refunded"}

PII Redaction

# tools.py — Automatic masking of sensitive data

EMAIL_PATTERN = re.compile(r"\b([A-Z0-9._%+-]+)@([A-Z0-9.-]+\.[A-Z]{2,})\b", re.IGNORECASE)
DIGIT_PATTERN = re.compile(r"\b\d{8,}\b")

def redact_pii(value: Any) -> Any:
    """Recursive redaction on dict, list, tuple, str."""
    if isinstance(value, dict):
        return {key: redact_pii(val) for key, val in value.items()}
    if isinstance(value, list):
        return [redact_pii(item) for item in value]
    if isinstance(value, str):
        # Mask emails: customer1@example.com → c***@example.com
        value = EMAIL_PATTERN.sub(lambda m: f"{m.group(1)[0]}***@{m.group(2)}", value)
        # Mask long numbers: 1234567890 → ********90
        value = DIGIT_PATTERN.sub(lambda m: f"{'*' * (len(m.group(0)) - 2)}{m.group(0)[-2:]}", value)
        return value
    return value

# Usage: before returning any sensitive output
safe_output = redact_pii(tool_result) if settings.enable_redaction else tool_result

Data Consent Gate

# agent.py — Consent pattern

CONSENT_ACCESS_PATTERN = re.compile(
    r"\bconsent\s+(?:access|share)\s+([A-Z]{3}-\d{5}|TICK\d+)\b", re.IGNORECASE
)

def _has_data_consent(self, ticket_id: str) -> bool:
    """Checks if the user has given explicit consent for this ticket."""
    last_msg = getattr(self, "last_user_message", "")
    match = CONSENT_ACCESS_PATTERN.search(last_msg)
    if match:
        consented_id = match.group(1).upper()
        return consented_id == ticket_id.upper()
    return ticket_id in self.data_consents  # Set of already granted consents

# In the loop:
if self.settings.enable_data_sharing_prompt and not self._has_data_consent(ticket_id):
    return {
        "error": "requires_consent",
        "message": f"To access data for {ticket_id}, please say: 'consent access {ticket_id}'"
    }

9. Governance and Security at Scale

Governance = Observe + Rollback + Protect Over Time

flowchart LR
    O[Observe\nTraces + Logs + Metrics] --> R[Rollback\nif regression detected]
    R --> P[Protect\nEvals encode the rules]
    P --> O

Environment Variables for Production

# Production mode — all guardrails active
ENABLE_SAFETY_GUARDRAILS=true
GLOBO_ENV=prod
REFUND_API_KEY=your-secure-key-here
ENABLE_OUTPUT_MODERATION=true
ENABLE_REDACTION=true
ENABLE_AUDIT_LOG=true
AUDIT_LOG_PATH=/var/log/globobot/audit.jsonl

Governance Metrics (endpoint /debug/metrics)

{
  "tool_calls": 47,
  "tool_calls_by_tool": {
    "lookup_ticket": 38,
    "refund_ticket": 9
  },
  "blocked": 3,
  "errors": 1,
  "moderation_total": 47,
  "moderation_flagged": 0,
  "feedback_scores": [5, 4, 5, 3, 5]
}

Governance rule: Governance requires measurable signals, not assumptions. Metrics allow ops teams to monitor activity and risk indicators immediately.

10. Evals — Automated Quality Testing

Why Evals?

“Fixes are only useful if they stay fixed.”

Evals answer one question: “Is the system still behaving as expected?”

These are not phrasing or tone tests. These are behavioral tests.

Eval Structure

// test_cases.json
[
  {
    "name": "refund_status_consistency",
    "prompt": "Refund ticket ABC-12345. Now what is the status?",
    "expected_substring": "refunded"
  },
  {
    "name": "ticket_not_found",
    "prompt": "What is the status of TICK-99999?",
    "expected_substring": "could not find"
  },
  {
    "name": "missing_ticket_id",
    "prompt": "What is the status of my ticket?",
    "expected_substring": "ticket ID"
  }
]

Eval Harness (Python)

# eval_tests.py
import json
from pathlib import Path
from globobot.agent import GlobobotReliableAgent
from globobot.config import Settings
from globobot.tools import ToolRuntime

def run_evals(test_cases_path: str = "test_cases.json") -> None:
    settings = Settings(openai_api_key=os.environ["OPENAI_API_KEY"])
    runtime = ToolRuntime(settings)

    with open(test_cases_path) as f:
        test_cases = json.load(f)

    passed = 0
    failed = 0

    for case in test_cases:
        # Fresh agent per test case
        agent = GlobobotReliableAgent(runtime=ToolRuntime(settings), settings=settings)
        response = agent.chat(case["prompt"])

        expected = case["expected_substring"].lower()
        if expected in response.lower():
            print(f"✅ PASS: {case['name']}")
            passed += 1
        else:
            print(f"❌ FAIL: {case['name']}")
            print(f"   Expected: '{expected}'")
            print(f"   Got: '{response[:100]}'")
            failed += 1

    print(f"\n{passed}/{passed + failed} tests passed")
    if failed > 0:
        raise SystemExit(1)  # Fail CI if evals fail

Safety Evals

# safety_eval_tests.py

def test_redaction() -> None:
    """Verifies that PII is masked in outputs."""
    text = "Email customer1@example.com and call 1234567890"
    redacted = redact_pii(text)
    assert "customer1@example.com" not in redacted, "Email should be masked"
    assert "1234567890" not in redacted, "Long digit string should be masked"
    assert "c***@example.com" in redacted, "Masked email should keep domain"

def test_refund_kill_switch() -> None:
    """Verifies that the kill switch completely disables refunds."""
    settings = Settings(
        openai_api_key="test-key",
        enable_safety_guardrails=True,
        enable_refund_tool=False,     # Kill switch active
        enable_audit_log=False,
    )
    runtime = ToolRuntime(settings)
    result = runtime.refund_ticket("ABC-12345")
    assert result.get("error") == "refund_disabled", "Refund kill switch failed"

def test_data_consent_gate() -> None:
    """Verifies that the consent gate blocks access without explicit consent."""
    settings = Settings(
        openai_api_key="test-key",
        enable_safety_guardrails=True,
        enable_data_sharing_prompt=True,
    )
    runtime = ToolRuntime(settings)
    agent = GlobobotReliableAgent(runtime=runtime, settings=settings)
    agent.last_user_message = "CONSENT access ABC-12345"
    assert agent._has_data_consent("ABC-12345"), "Data-sharing consent pattern not detected"

Critical invariant: After a successful refund, the ticket status must show “refunded”. If the test fails, it is a behavioral regression, not a wording issue.

11. Reference Tables

AgentKit Components — Complete Overview

Tool Call Patterns

Environment Variables — Complete Reference

Pre-Production Security Checklist

• ENABLE_SAFETY_GUARDRAILS=true
• ENABLE_REDACTION=true — PII masked in all outputs
• ENABLE_OUTPUT_MODERATION=true — Response screening
• ENABLE_AUDIT_LOG=true — Trace of all destructive actions
• REFUND_API_KEY configured with a real secure key
• GLOBO_ENV=prod — Environment gating active
• Evals running in CI/CD
• Governance metrics monitored

12. Complete Architecture — Diagrams

Complete Globobot Agent Architecture

graph TB
    subgraph "Experience Surface"
        U[User]
        CK["ChatKit\n(Admin Portal)"]
        GPT["ChatGPT App\n(Apps SDK)"]
    end

    subgraph "API Gateway (FastAPI)"
        CE["/chat endpoint"]
        CS["/api/chatkit/session"]
        DT["/debug/traces"]
        DL["/debug/logs"]
        DM["/debug/metrics"]
        DA["/debug/audit"]
    end

    subgraph "Agent Runtime"
        A["GlobobotReliableAgent\n(Responses API)"]
        SP["System Prompt\n+ Tool-use Policy"]
        MSG["Messages History\n(Session State)"]
    end

    subgraph "Tool Runtime"
        TR["ToolRuntime\n(Cache + Logs + Metrics)"]
        LT["lookup_ticket"]
        RT["refund_ticket"]
        RD["redact_pii"]
        MOD["output_moderation"]
        AUD["audit_log (JSONL)"]
    end

    subgraph "Data Store"
        TS["TicketStore\n(in-memory)"]
    end

    U --> CK
    U --> GPT
    CK --> CE
    CK --> CS
    GPT --> CE
    CE --> A
    A --> SP
    A --> MSG
    A --> TR
    TR --> LT
    TR --> RT
    RT --> RD
    RT --> MOD
    RT --> AUD
    LT --> TS
    RT --> TS
    TR --> DT
    TR --> DL
    TR --> DM
    AUD --> DA

Tool Calling Flow with Guardrails

sequenceDiagram
    participant User
    participant Agent
    participant GuardRails
    participant ToolRuntime
    participant OpenAI

    User->>Agent: "Refund ticket ABC-12345"
    Agent->>GuardRails: Check: ticket ID detected?
    GuardRails-->>Agent: ✅ Ticket detected, tools enabled
    Agent->>OpenAI: Responses API (messages + tool schemas)
    OpenAI-->>Agent: function_call: refund_ticket({ticket_id: "ABC-12345"})
    Agent->>GuardRails: Check: explicit confirmation received?
    GuardRails-->>Agent: ❌ No confirmation
    Agent-->>User: "Please confirm: 'confirm refund ABC-12345'"
    User->>Agent: "confirm refund ABC-12345"
    Agent->>GuardRails: Check: prod env? valid API key?
    GuardRails-->>Agent: ✅ All conditions met
    Agent->>ToolRuntime: refund_ticket("ABC-12345", api_key=...)
    ToolRuntime->>ToolRuntime: Write audit log
    ToolRuntime-->>Agent: {success: true, status: "Refunded"}
    Agent->>GuardRails: redact_pii + moderate output
    GuardRails-->>Agent: Output safe
    Agent-->>User: "Ticket ABC-12345 has been refunded successfully."

Multi-Session Isolation

graph LR
    S1["Session A\n(Support Agent #1)"] --> SA["GlobobotReliableAgent\n(instance A)"]
    S2["Session B\n(Customer #2)"] --> SB["GlobobotReliableAgent\n(instance B)"]
    S3["Session C\n(Admin)"] --> SC["GlobobotReliableAgent\n(instance C)"]
    SA --> TR["ToolRuntime\n(shared)"]
    SB --> TR
    SC --> TR
    TR --> TS["TicketStore"]

Build → Deploy → Optimize Cycle

flowchart LR
    B1["Define Workflows"] --> B2["Create Tools\n(with MCP schemas)"]
    B2 --> B3["Write System Prompt\n+ Tool-use Policy"]
    B3 --> B4["Add Guardrails\n(See, Do, Say)"]
    B4 --> D1["Deploy ChatKit\n(your app)"]
    D1 --> D2["Deploy ChatGPT App\n(Apps SDK)"]
    D2 --> O1["Observe Traces\nand Logs"]
    O1 --> O2["Run Evals\n(CI/CD)"]
    O2 --> O3["Iterate:\nCache? Prompt?\nTool contract?"]
    O3 --> B1

Conclusion — The Era of AI Operators

“The teams that win with agents won’t be the ones with the cleverest prompts; they’ll be the ones who understand systems.”

The 5 Pillars of a Production Agent

The Production Loop

1. Observe — Reproduce the problematic behavior
2. Diagnose — Traces → logs → payloads
3. Fix — Change the system, not just the prompt
4. Verify — Evals confirm the fix holds
5. Repeat

What a Production Agent Is NOT

• ❌ A very long prompt
• ❌ A chatbot that guesses
• ❌ A system you “hope” is correct
• ❌ A demo you cannot explain

What a Production Agent IS

• ✅ An operational system with explicit contracts (MCP)
• ✅ A workflow with deterministic decisions (gating)
• ✅ An observable service with complete traces
• ✅ A governed software with automated evals
• ✅ A digital teammate, not just a chatbot

Search Terms

developing · ai · agents · openai · agentkit · orchestration · artificial · intelligence · generative · agent · tool · production · architecture · mcp · security · cache · chatkit · components · environment · evals · governance · reliability · variables · act