Frameworks for Developing LLM Agents

Demo Project: https://github.com/jzheaux/ai-frameworks

Table of Contents

1. Overview and Architecture
2. Module 1 — Refining Prompts to Improve Content and Format
   • 2.1 Why Use a Framework?
   • 2.2 The System Prompt
   • 2.3 Output Formatting
3. Module 2 — RAG Techniques to Enrich Agent Context
   • 3.1 Document Ingestion
   • 3.2 VectorStore and Embeddings
   • 3.3 The QuestionAnswerAdvisor — Restricting Prior Knowledge
   • 3.4 Connecting to Real-Time Data (Function Calling)
4. Module 3 — Improving Performance via the Feedback Loop
   • 4.1 Manual Feedback (Human-in-the-Loop)
   • 4.2 Automatic Feedback via Function Calling
5. Module 4 — Orchestration and Conversational Memory
   • 5.1 ChatMemory and Session Identifier
   • 5.2 PromptChatMemoryAdvisor
6. Complete Architecture — Final Overview
7. Spring Boot Project Structure
8. RAG Documents Used in the Demo
9. Summary and Best Practices

1. Overview and Architecture

The demo application is a tourist guide for students visiting San Francisco as part of a school trip. The AI agent plays the role of a chaperone who:

• Suggests activities taking into account the weather, the trip schedule, and school policies
• Refuses any suggestion inappropriate for minors
• Remembers previous conversations with each student
• Learns from user feedback

flowchart TB
    Student["👤 Student"]
    App["Spring Boot Application\n(Chaperone)"]
    OpenAI["🤖 OpenAI LLM\n(ChatClient)"]
    VectorDB["🗄️ VectorStore\n(RAG Documents)"]
    ChatMem["💬 ChatMemory\n(History)"]
    WeatherAPI["🌦️ Weather API\n(api.weather.gov)"]
    FeedbackFn["🔄 Feedback Function\n(Consumer)"]

    Student -->|"Text message"| App
    App -->|"System Prompt + User Message"| OpenAI
    OpenAI -->|"Function call"| WeatherAPI
    OpenAI -->|"Function call"| FeedbackFn
    VectorDB -->|"Relevant RAG context"| App
    ChatMem -->|"Conversation history"| App
    FeedbackFn -->|"Writes feedback"| VectorDB
    OpenAI -->|"Structured JSON response"| App
    App -->|"Activity suggestions"| Student

Request Processing Flow

sequenceDiagram
    participant S as Student
    participant C as Chaperone (Spring AI)
    participant A as Advisor Chain
    participant V as VectorStore
    participant M as ChatMemory
    participant L as OpenAI LLM
    participant W as Weather API

    S->>C: "What should I do Monday evening?"
    C->>A: Build the prompt
    A->>V: Retrieve relevant RAG context
    V-->>A: Policies + Brochure + Itinerary
    A->>M: Retrieve history (chatId)
    M-->>A: Previous messages
    A->>L: Full prompt (system + RAG + memory + message)
    L->>W: getWeatherForecast() [function call]
    W-->>L: San Francisco weather forecast
    L-->>C: Structured JSON response
    C-->>S: Appropriate activity suggestions

2. Module 1 — Refining Prompts to Improve Content and Format

2.1 Why Use a Framework?

“Can’t I just toss a prompt into ChatGPT and call it a day?”

The short answer: this entire application can be built in fewer than 100 lines of maintainable and testable Spring AI code.

Advantages of frameworks like Spring AI:

2.2 The System Prompt

The system prompt defines the agent’s role, context, and tone. It is the first layer of prompt engineering.

Analogy: Like a math tutor who must first identify a student’s fundamental gaps before tackling trigonometric identities — talking to an LLM is an exercise in awareness of the implicit assumptions one makes.

System Prompt Evolution (pedagogical progression)

Step 1 — Minimal (generic result):

// No system prompt → generic responses without context
this.chat = builder
    .build();

public String chat(String userInput) {
    return this.chat.prompt()
        .user(userInput)
        .call()
        .content();
}

Step 2 — Basic role:

this.chat = builder
    .defaultSystem("""
        Please act as if you are a chaperone for a group of high school 
        students from Brookside High School. They are visiting San Francisco 
        and have occasional free time where they will ask you for suggestions 
        for what to do.
    """)
    .build();

Step 3 — Added restrictions (minors, curfew):

.defaultSystem("""
    Please act as if you are a chaperone ...
    
    Do not suggest any activities inappropriate for minors.
    Remind the students to be back in the hotel by 10 PM since that's curfew.
""")

Step 4 — Complete system prompt (final version with all parameters):

// Excerpt from Chaperone.java
this.chat = builder
    .defaultSystem("""
        Please act as if you are a chaperone for a group of high school students 
        from Brookside High School. They are visiting San Francisco and have 
        occasional free time where they will ask you for suggestions for what to do.

        If you don't know the student's name, begin by asking, so that you can 
        retrieve any previous conversations with them.
        
        If you have provided suggestions to them in the past, ask them what 
        activities they did and how much they enjoyed them. When you receive
        feedback, make sure to use the appropriate function to store it and 
        improve later suggestions.

        To create a list of suggestions, take the following into account:
        
        * What day of their trip they are wanting suggestions for
        * What time of day they'd likely be doing the activity, given the trip 
          itinerary you were given (if you aren't sure, you can ask)
        * What the weather conditions are for that time period (use the provided 
          function to check the weather forecast)
        * If it is an activity they've told you they didn't like (you probably 
          shouldn't suggest that again)
        * If it is an activity they've told you they already did (you probably 
          shouldn't suggest that again) 
        * If it abides school policy. You MUST follow all school policies and 
          SHOULD stick to the list of pre-approved activities from the brochure 
          you were given.
        * If there the venue is open at that time and there is enough time to 
          travel there, enjoy the activity without being rushed, and get back to 
          the group for the rest of their itinerary.
        * NOTE: Sometimes they have a performance. Remember that they will need 
          extra time to get ready before and will be in tuxedos and dresses.

        IMPORTANT If the activity cannot be done during their free-time, either 
        due to time constraints, weather conditions, it's against school policy, 
        or the venue is not open DO NOT SUGGEST that activity.

        The first day of their trip is the upcoming Sunday. They will always have 
        an adult chaperone with them.

        Today is {current_date}. They've worked hard to get here, help them have fun!
    """)
    .defaultAdvisors(new PromptChatMemoryAdvisor(memory), new QuestionAnswerAdvisor(vectors))
    .defaultFunctions("getWeatherForecast", "saveStudentFeedback")
    .build();

Key point: The system prompt defines the role, constraints, and context. The more information you provide, the more precisely the agent can operate.

2.3 Output Formatting

Spring AI can automatically ask the LLM to format its response as JSON matching a Java record — useful for a front end to parse the results.

// Structured response record definition
private record Response(String response, List<Activity> activities) {}

private record Activity(
    String activityName, 
    String studentName, 
    Double activityCost, 
    List<String> dayOfWeek, 
    String timeOfDay, 
    String forecastDescription, 
    Integer forecastTemperature
) {}

// Usage in the chat() method
public String chat(String chatId, String userMessage) {
    Response response = this.chat.prompt()
        .system(s -> s.param("current_date", LocalDate.now().toString()))
        .user(userMessage)
        .advisors(a -> a
            .param(CHAT_MEMORY_CONVERSATION_ID_KEY, chatId)
            .param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))
        .call()
        .entity(Response.class);  // ← Spring AI instructs the LLM to return JSON
    
    if (response.activities() != null) {
        this.activities.addAll(response.activities());
    }
    return response.response;
}

Example of structured JSON output:

{
  "response": "I recommend visiting Chinatown and the Fortune Cookie Factory...",
  "activities": [
    {
      "activityName": "Chinatown & Fortune Cookie Factory",
      "studentName": "Alice",
      "activityCost": 2.0,
      "dayOfWeek": ["Monday"],
      "timeOfDay": "afternoon",
      "forecastDescription": "Partly Cloudy",
      "forecastTemperature": 65
    }
  ]
}

Best practice: Let the framework handle common formatting instructions, such as defining the role, constraining the LLM, and guiding the format so it is parseable by code.

3. Module 2 — RAG Techniques to Enrich Agent Context

3.1 Document Ingestion

RAG (Retrieval-Augmented Generation) merges the LLM with your external documents, enabling the AI to reliably cite or use up-to-date data.

Why not put everything in the system prompt?

graph LR
    A["Add everything<br/>to the system prompt"] -->|"Problem 1"| B["High cost per request<br/>(all tokens billed)"]
    A -->|"Problem 2"| C["Cannot update<br/>without redeployment"]
    A -->|"Problem 3"| D["Less focused context,<br/>less precise agent"]
    
    E["VectorStore RAG"] -->|"Solution"| F["Only relevant context<br/>is sent to the LLM"]
    E -->|"Solution"| G["Runtime updates<br/>without redeployment"]
    E -->|"Solution"| H["Token savings,<br/>more focused agent"]

RAG documents used in the demo:

Ingestion code at startup:

// AiFrameworksApplication.java — CommandLineRunner
@Override
public void run(String... args) throws Exception {
    // Load RAG documents at startup
    this.resources.forEach((r) -> {
        // 1. Read the text document
        List<Document> documents = new TextReader(r).read();
        
        // 2. Split into chunks (paragraphs/sections)
        //    for more efficient embedding
        documents = new TokenTextSplitter().transform(documents);
        
        // 3. Write to the VectorStore
        //    (automatic embedding by the configured model)
        this.vectors.write(documents);
    });
    
    runApp();
}

Injection of RAG resources:

public AiFrameworksApplication(
    Chaperone chaperone, 
    VectorStore vectors,
    @Value("classpath:rag/*.txt") List<Resource> resources  // ← all .txt files in rag/ folder
) {
    this.chaperone = chaperone;
    this.vectors = vectors;
    this.resources = resources;
}

Spring AI also supports PdfReader for PDF files and JsonReader for JSON, not just TextReader.

3.2 VectorStore and Embeddings

flowchart LR
    subgraph "Ingestion Phase (at startup)"
        D1["activity-brochure.txt"]
        D2["school-policies.txt"]
        D3["trip-itinerary.txt"]
        TR["TokenTextSplitter\n(split into chunks)"]
        EM["OpenAI Embedding Model\n(text → numeric vectors)"]
        VS["SimpleVectorStore\n(vector database)"]
        D1 & D2 & D3 --> TR --> EM --> VS
    end

    subgraph "Query Phase (on each prompt)"
        Q["User query"]
        QE["Query embedding"]
        SIM["Cosine similarity search"]
        CTX["Retrieved relevant chunks"]
        LLM["OpenAI LLM"]
        Q --> QE --> SIM
        VS --> SIM --> CTX --> LLM
        Q --> LLM --> REP["Contextualized response"]
    end

VectorStore configuration:

// AiConfig.java
@Configuration
public class AiConfig {
    
    @Bean
    VectorStore vectors(EmbeddingModel model) {
        // SimpleVectorStore = in-memory for demo
        // In production: ChromaDB, Pinecone, pgvector, etc.
        return new SimpleVectorStore(model);
    }

    @Bean
    ChatMemory memory() {
        return new InMemoryChatMemory();
        // In production: CassandraChatMemory, etc.
    }
}

VectorStore alternatives supported by Spring AI:

SimpleVectorStore    (in-memory, for demo/testing)
ChromaDB            (open-source vector database)
Pinecone            (cloud)
Weaviate            (open-source)
pgvector            (PostgreSQL extension)
Redis               (with Vector Similarity Search module)
Azure AI Search     (Microsoft cloud)

3.3 The QuestionAnswerAdvisor — Restricting Prior Knowledge

Spring AI uses the concept of Advisors — components that enrich the prompt before it is sent to the LLM.

flowchart LR
    UP["User Prompt"] --> AC
    subgraph AC["Advisor Chain"]
        direction TB
        PMA["PromptChatMemoryAdvisor\n(adds history)"]
        QAA["QuestionAnswerAdvisor\n(adds RAG context)"]
        PMA --> QAA
    end
    AC --> |"Enriched prompt"| LLM["LLM"]

Adding the QuestionAnswerAdvisor:

// In Chaperone.java
public Chaperone(ChatClient.Builder builder, VectorStore vectors, ChatMemory memory) {
    this.chat = builder
        .defaultSystem("...")
        .defaultAdvisors(
            new PromptChatMemoryAdvisor(memory),   // conversational memory
            new QuestionAnswerAdvisor(vectors)      // RAG context
        )
        .defaultFunctions("getWeatherForecast", "saveStudentFeedback")
        .build();
}

What the QuestionAnswerAdvisor does internally: It adds strict instructions to the prompt indicating that the LLM must not use prior knowledge and should rely solely on the documents provided by the VectorStore.

Automatically added instructions (internal example):

Use the following context to answer the user's question. 
DO NOT use any prior knowledge outside of this context.

CONTEXT:
{vector_store_context}

3.4 Connecting to Real-Time Data (Function Calling)

Function calling allows the LLM to call functions in our application to obtain dynamic data (weather, database, external API).

flowchart LR
    LLM["OpenAI LLM"]
    LLM -->|"I need to know the weather"| FC["Function Call:\ngetWeatherForecast()"]
    FC -->|"HTTP call"| WAPI["api.weather.gov\n/gridpoints/MTR/85,106/forecast/hourly"]
    WAPI -->|"Weather JSON"| FC
    FC -->|"7-day forecast"| LLM
    LLM -->|"Weather-aware suggestion"| RESP["Final response"]

WeatherTool implementation:

// WeatherTools.java
@Configuration
public class WeatherTools {
    
    @Bean
    @Description("Get Weather Forecast")  // ← Human-readable description for the LLM
    public Supplier<List<WeatherResponse>> getWeatherForecast() {
        return () -> {
            RestTemplate rest = new RestTemplate();
            String uri = "https://api.weather.gov/gridpoints/MTR/85,106/forecast/hourly";
            WeatherApiResponse response = rest.getForObject(URI.create(uri), WeatherApiResponse.class);
            return response.properties().periods();
        };
    }

    // Records for parsing the API response
    public record WeatherApiResponse(Properties properties) {}
    public record Properties(List<WeatherResponse> periods) {}
    public record WeatherResponse(
        ZonedDateTime startTime, 
        ZonedDateTime endTime, 
        Integer temperature, 
        String windSpeed,
        Precipitation probabilityOfPrecipitation, 
        String shortForecast
    ) {}
    public record Precipitation(Integer value) {}
}

Function types supported by Spring AI:

Declaring functions at the ChatClient level:

// The @Bean name is the function name registered with OpenAI
.defaultFunctions("getWeatherForecast", "saveStudentFeedback")

Testing the weather API:

// AiFrameworksApplicationTests.java
@Test
void whenGetWeatherThenReturnsHourly() {
    RestTemplate rest = new RestTemplate();
    String uri = "https://api.weather.gov/gridpoints/MTR/85,106/forecast/hourly";
    WeatherApiResponse response = rest.getForObject(URI.create(uri), WeatherApiResponse.class);
    assertThat(response.properties().periods()).isNotNull();
}

4. Module 3 — Improving Performance via the Feedback Loop

4.1 Manual Feedback (Human-in-the-Loop)

A human operator (teacher, administrator) can update the VectorStore in real time, without restarting the application.

flowchart TB
    subgraph "Normal flow"
        S["Student"] --> APP["AI Agent"]
        APP --> S
    end
    
    subgraph "Human feedback"
        ADM["👩‍🏫 Administrator / Teacher"] -->|"POST /feedback"| FB["FeedbackTools\n(REST endpoint)"]
        FB -->|"write(new Document(...))"| VS["VectorStore"]
        VS -->|"Enriched context on next request"| APP
    end
    
    subgraph "Monitoring"
        ADM2["👨‍💼 Operator"] -->|"GET /activities"| ACT["List of suggested activities"]
        ACT -->|"Anomalies detected"| ADM2
    end

Endpoint to view suggested activities:

// In Chaperone.java
@GetMapping("/activities")
public List<Activity> activitiesSuggested() {
    return this.activities;  // List of activities recommended by the AI
}

VectorStore update endpoint:

// FeedbackTools.java
@Configuration
public class FeedbackTools {
    private final VectorStore vectors;
    
    public FeedbackTools(VectorStore vectors) {
        this.vectors = vectors;
    }

    // Manual feedback: called via a REST endpoint by an operator
    // (e.g., "New info: SFMOMA is closed next Monday")
    public void addManualFeedback(String feedbackText) {
        List<Document> docs = List.of(new Document(feedbackText));
        docs = new TokenTextSplitter().transform(docs);
        this.vectors.write(docs);
    }
    
    // Automatic feedback via function calling (see section 4.2)
    @Bean
    @Description("Save Student Feedback")
    Consumer<Feedback> saveStudentFeedback() {
        return (feedback) -> this.vectors.write(
            List.of(new Document(feedback.feedback()))
        );
    }

    private record Feedback(String feedback) {}
}

Real-world use cases for manual feedback:

• New exhibit at SFMOMA → update the brochure
• Last-minute curfew adjustment
• Temporary venue closure
• Parent feedback after a visit

4.2 Automatic Feedback via Function Calling

The LLM detects on its own when a user gives feedback and automatically calls the function to store it.

sequenceDiagram
    participant S as Student
    participant L as OpenAI LLM
    participant F as saveStudentFeedback()
    participant V as VectorStore

    S->>L: "I loved the dim sum in Chinatown,\nbut I'm not really into bubble tea"
    Note over L: LLM detects feedback\nin the message
    L->>F: saveStudentFeedback({feedback: "Student likes dim sum, dislikes bubble tea"})
    F->>V: write(new Document("..."))
    V-->>F: OK
    L-->>S: "Got it! I'll keep your preferences in mind\nfor future suggestions."
    
    Note over V: On this student's next request,\nthe VectorStore will return this context

Instruction in the system prompt to enable automatic feedback:

.defaultSystem("""
    ...
    If you have provided suggestions to them in the past, ask them what 
    activities they did and how much they enjoyed them. When you receive
    feedback, make sure to use the appropriate function to store it and 
    improve later suggestions.
    ...
""")
.defaultFunctions("getWeatherForecast", "saveStudentFeedback")

Complete FeedbackTools implementation:

// FeedbackTools.java
@Configuration
public class FeedbackTools {
    private final VectorStore vectors;
    
    public FeedbackTools(VectorStore vectors) {
        this.vectors = vectors;
    }

    @Bean
    @Description("Save Student Feedback")
    Consumer<Feedback> saveStudentFeedback() {
        return (feedback) -> this.vectors.write(
            List.of(new Document(feedback.feedback()))
        );
    }

    // The LLM will format its call using this record
    private record Feedback(String feedback) {}
}

Power of this approach: LLMs are naturally good at understanding human language. By delegating feedback detection to the LLM, there’s no need to add ”👍 / 👎” buttons to the interface — it understands the nuances of free text.

Possible enhancements:

• Ask the LLM to score the sentiment of the feedback (positive/negative/neutral)
• Store feedback in a persistent external database
• Implement an automated evaluation loop (LLM-as-judge)

5. Module 4 — Orchestration and Conversational Memory

5.1 ChatMemory and Session Identifier

Problem: A student discusses a visit to the Golden Gate Bridge on Tuesday for Friday. When they resume the conversation on Friday, the agent must remember that request.

stateDiagram-v2
    [*] --> Welcome: Application starts
    Welcome --> StudentIdentification: Student Name >>>
    StudentIdentification --> ChatSession: chatId = unique UUID per student
    
    state ChatSession {
        [*] --> Conversation
        Conversation --> MemorySaved: Each message stored
        MemorySaved --> Conversation: Next interaction
        Conversation --> Done: "DONE"
    }
    
    Done --> StudentIdentification: Next student

Session management in the application:

// AiFrameworksApplication.java
public class AiFrameworksApplication implements CommandLineRunner {
    private String chatId = UUID.randomUUID().toString();
    private Map<String, String> chats = new LinkedHashMap<>();  // name → chatId

    private void runApp() {
        System.out.print("""
            Welcome to Our AI Tour Guide App! Begin by entering your name and 
            the AI will greet you. Once you are all done, type the word DONE 
            so another student can use it.
        """);
        
        try (Scanner scanner = new Scanner(System.in)) {
            while (true) {
                System.out.print("Student Name >>> ");
                String name = scanner.nextLine();
                
                // Retrieve or create a chatId for this student
                this.chatId = this.chats.computeIfAbsent(
                    name, 
                    (n) -> UUID.randomUUID().toString()
                );
                
                System.out.println(this.chaperone.chat(this.chatId, "hi, my name is " + name));
                
                while (true) {
                    System.out.print(">>> ");
                    String message = scanner.nextLine();
                    if ("DONE".equals(message)) break;
                    System.out.println(this.chaperone.chat(this.chatId, message));
                }
            }
        }
    }
}

Difference between VectorStore and ChatMemory:

┌─────────────────────────────────────────────────────────────────┐
│                      VectorStore                                 │
│  • Information shared across ALL users                           │
│  • School policies, brochure, itinerary                         │
│  • Semantic similarity search                                   │
│  • Can be updated at runtime                                    │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                      ChatMemory                                  │
│  • Context specific to ONE conversation / ONE user              │
│  • Message history (questions + responses)                      │
│  • Identified by chatId (UUID per student)                      │
│  • Retrieval of the last N messages                             │
└─────────────────────────────────────────────────────────────────┘

5.2 PromptChatMemoryAdvisor

// AiConfig.java
@Bean
ChatMemory memory() {
    return new InMemoryChatMemory();
    // Persistent alternative: CassandraChatMemory (Spring AI)
}

// Chaperone.java — Usage in the advisor chain
public Chaperone(ChatClient.Builder builder, VectorStore vectors, ChatMemory memory) {
    this.chat = builder
        .defaultSystem("...")
        .defaultAdvisors(
            new PromptChatMemoryAdvisor(memory),      // ← Adds history to the prompt
            new QuestionAnswerAdvisor(vectors)
        )
        .build();
}

public String chat(String chatId, String userMessage) {
    Response response = this.chat.prompt()
        .system(s -> s.param("current_date", LocalDate.now().toString()))
        .user(userMessage)
        .advisors(a -> a
            .param(CHAT_MEMORY_CONVERSATION_ID_KEY, chatId)     // ← Session identifier
            .param(CHAT_MEMORY_RETRIEVE_SIZE_KEY, 100))         // ← Last 100 messages
        .call()
        .entity(Response.class);
    
    if (response.activities() != null) {
        this.activities.addAll(response.activities());
    }
    return response.response;
}

What the PromptChatMemoryAdvisor adds to the prompt:

Use the conversation history below to provide context-aware responses.

CONVERSATION HISTORY:
[User]: hi, my name is Alice
[Assistant]: Hello Alice! I'm delighted to welcome you...
[User]: What can I do Monday afternoon?
[Assistant]: Monday afternoon, you have free time from 1pm to 5pm...
[User]: What about Friday?
...

6. Complete Architecture — Final Overview

flowchart TB
    subgraph "Application Startup"
        BOOT["SpringApplication.run()"]
        LOAD["CommandLineRunner.run()"]
        DOC1["activity-brochure.txt"]
        DOC2["school-policies.txt"]
        DOC3["trip-itinerary.txt"]
        TSP["TokenTextSplitter"]
        EMB["OpenAI Embedding Model"]
        VS[("VectorStore\n(SimpleVectorStore)")]
        
        BOOT --> LOAD
        DOC1 & DOC2 & DOC3 -->|"TextReader"| TSP
        TSP -->|"chunks"| EMB
        EMB -->|"vectors"| VS
    end
    
    subgraph "Conversation with a Student"
        STU["👤 Student"]
        APP["Chaperone.chat(chatId, message)"]
        PROMPT["Prompt Building"]
        PMA["PromptChatMemoryAdvisor"]
        QAA["QuestionAnswerAdvisor"]
        MEM[("ChatMemory\nby chatId")]
        
        STU -->|"message"| APP
        APP --> PROMPT
        PROMPT --> PMA
        PMA -->|"Retrieves history"| MEM
        PROMPT --> QAA
        QAA -->|"Semantic search"| VS
    end
    
    subgraph "LLM Call"
        OPENAI["🤖 OpenAI GPT\n(ChatClient)"]
        WFC["getWeatherForecast()\n→ api.weather.gov"]
        SFC["saveStudentFeedback()\n→ VectorStore"]
        RESP["JSON Response\n{response, activities}"]
        
        PROMPT -->|"enriched prompt"| OPENAI
        OPENAI -->|"function call"| WFC
        OPENAI -->|"function call"| SFC
        SFC -->|"new document"| VS
        WFC -->|"weather forecast"| OPENAI
        OPENAI --> RESP
    end
    
    RESP -->|"text response"| STU
    RESP -->|"activities"| ACT["GET /activities\n(operator monitoring)"]
    MEM -.->|"saves messages"| RESP

Complete flow in 6 steps:

1. RAG LOADING
   └─ Brochure + Policies + Itinerary → TokenTextSplitter → VectorStore

2. NEW CONVERSATION
   └─ Student identifies themselves → unique UUID chatId created/retrieved

3. DOCUMENT CONSTRAINTS
   └─ QuestionAnswerAdvisor retrieves: curfew, approved activities, schedule

4. REAL-TIME DATA
   └─ LLM calls getWeatherForecast() → San Francisco weather forecast

5. FEEDBACK LOOP
   └─ Student gives feedback → LLM calls saveStudentFeedback() → VectorStore

6. CONVERSATIONAL MEMORY
   └─ ChatMemory stores each exchange → available in future conversations

7. Spring Boot Project Structure

ai-frameworks-main/
├── build.gradle                    ← Gradle dependencies
├── src/
│   ├── main/
│   │   ├── java/.../ai_frameworks/
│   │   │   ├── AiFrameworksApplication.java  ← Entry point + RAG ingestion
│   │   │   ├── AiConfig.java                 ← VectorStore + ChatMemory beans
│   │   │   ├── Chaperone.java                ← Main agent logic
│   │   │   ├── WeatherTools.java             ← Function calling: weather
│   │   │   └── FeedbackTools.java            ← Function calling: feedback
│   │   └── resources/
│   │       ├── application.properties        ← OpenAI API key config
│   │       └── rag/
│   │           ├── activity-brochure.txt     ← 10 San Francisco activities
│   │           ├── school-policies.txt       ← Brookside High School policies
│   │           └── trip-itinerary.txt        ← Schedule Sunday → Saturday
│   └── test/
│       └── .../AiFrameworksApplicationTests.java  ← Weather API test

Gradle dependencies:

// build.gradle
plugins {
    id 'java'
    id 'org.springframework.boot' version '3.4.0'
    id 'io.spring.dependency-management' version '1.1.6'
}

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(17)
    }
}

ext {
    set('springAiVersion', "1.0.0-M4")
}

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter-web'
    implementation 'org.springframework.ai:spring-ai-openai-spring-boot-starter'
    
    testImplementation 'org.springframework.boot:spring-boot-starter-test'
    testRuntimeOnly 'org.junit.platform:junit-platform-launcher'
}

Configuration:

# application.properties
spring.application.name=ai-frameworks
spring.ai.openai.api-key={{your-openai-key}}

8. RAG Documents Used in the Demo

Activity Brochure (activity-brochure.txt)

10 pre-approved activities for Brookside High students in San Francisco:

School Policies (school-policies.txt)

Key Brookside High School rules applicable during the trip:

• Curfew: Return to The Inn at Union Square hotel before 10:00 PM
• Adult supervision: All off-site activities must have an approved chaperone
• Appropriate activities: No adult-only venues (bars, casinos, clubs)
• Dress code: Adhere to school dress code
• Transportation: Only school-approved vehicles
• Free time: Pre-approved activities only, within designated areas

Trip Itinerary (trip-itinerary.txt)

7-day schedule (Sunday → Saturday) — Band trip (music tour):

Day 1 (Sunday)    — Arrival
  3:00 PM : Check-in at The Inn at Union Square
  7:00 PM–10:00 PM : Free time

Day 2 (Monday)    — Rehearsals begin
  8:00 AM–12:00 PM : Music rehearsal
  12:00 PM–1:00 PM : ★ FREE TIME (lunch)
  1:00 PM–5:00 PM  : ★ FREE TIME (afternoon)
  5:00 PM+         : ★ FREE TIME (evening)

Day 3 (Tuesday)   — Group activity #1
  8:00 AM–12:00 PM : Rehearsal
  12:00 PM–1:00 PM : ★ FREE TIME
  1:00 PM–5:00 PM  : Music workshop + Backstage tour (San Francisco Opera / SF Symphony)
  5:00 PM+         : ★ FREE TIME

Day 4 (Wednesday) — Mid-week rehearsals
  8:00 AM–12:00 PM : Rehearsal
  12:00 PM–1:00 PM : ★ FREE TIME
  1:00 PM–5:00 PM  : ★ FREE TIME (SFMOMA, shopping...)
  5:00 PM+         : ★ FREE TIME

Day 5 (Thursday)  — Group activity #2
  8:00 AM–12:00 PM : Rehearsal
  12:00 PM–1:00 PM : ★ FREE TIME
  1:00 PM–5:00 PM  : Studio masterclass or team-building
  5:00 PM+         : ★ FREE TIME

Day 6 (Friday)    — 🎭 Evening performance
  8:00 AM–12:00 PM : Final rehearsal
  1:00 PM–3:00 PM  : Stage dress rehearsal
  3:00 PM–5:00 PM  : ★ FREE TIME (light rest)
  7:00 PM–9:00 PM  : PERFORMANCE #1
  10:00 PM         : Curfew

Day 7 (Saturday)  — 🎭 Two performances
  9:30 AM–11:00 AM : ★ FREE TIME
  12:00 PM–2:00 PM : MATINEE
  2:00 PM–5:00 PM  : ★ FREE TIME (⚠️ stay in concert attire)
  6:00 PM–8:00 PM  : EVENING PERFORMANCE
  10:00 PM         : Curfew

Special case: On performance days, students are in tuxedos/dresses. Free time between the two Saturday performances should not include activities requiring a change of clothes or going far.

9. Summary and Best Practices

The 4 Pillars Covered in This Course

mindmap
  root((Frameworks<br/>LLM Agents))
    Prompts
      System Prompt
      Role and context
      Constraints and restrictions
      Structured Output
    RAG
      Document ingestion
      TokenTextSplitter
      VectorStore
      QuestionAnswerAdvisor
    Feedback
      Human-in-the-loop
      REST endpoint
      Automatic function calling
      Continuous improvement
    Orchestration
      ChatMemory
      chatId per session
      PromptChatMemoryAdvisor
      Conversation state

Summary of Spring AI Components Used

Best Practices

1. Invest in your prompts — A good system prompt is the foundation of any effective agent
2. Use RAG for business knowledge rather than putting everything in the system prompt
3. Split your documents with TokenTextSplitter for more precise embedding
4. Separate VectorStore and ChatMemory — one for global knowledge, the other for conversational state
5. Use function calling for dynamic data (weather, inventory, databases)
6. Implement feedback loops — both manual and automatic — to continuously improve the agent
7. Create monitoring endpoints (/activities) to observe AI behavior in production
8. Test your integrations — external API calls (weather) must have unit tests

In Production — What Would Need to Change

// DEV (in-memory, reset on every restart)
return new SimpleVectorStore(model);
return new InMemoryChatMemory();

// PRODUCTION (persistent, distributed)
return new ChromaVectorStore(chromaClient, embeddingModel);
return new CassandraChatMemory(cassandraSession);
// or
return new PgVectorStore(jdbcTemplate, embeddingModel);
return new RedisChatMemory(redisTemplate);

Complete source code: https://github.com/jzheaux/ai-frameworks
Step-by-step branch: Branch steps to follow the progressive build

Search Terms

frameworks · developing · llm · agents · ai · orchestration · artificial · intelligence · generative · feedback · architecture · calling · function · prompt · rag · spring · system · via