Amazon Bedrock

Complete guide to Amazon Bedrock: architecture, model invocation, RAG, inference parameters, and model customization.

Table of Contents

• 1. Amazon Bedrock’s Role in AWS Architectures
  • 1.1 Foundation Models in Generative AI Applications
  • 1.2 Amazon Bedrock as a Managed Inference Service
  • 1.3 Bedrock vs SageMaker
  • 1.4 Bedrock in the Application
• 2. Request-Response Flow for Model Invocation
  • 2.1 The Request-Response Flow with Bedrock
  • 2.2 Transforming Requests into Model Inputs
  • 2.3 Returning and Consuming Model Responses
• 3. Architectural Patterns for Bedrock Use Cases
  • 3.1 Common Foundation Model Use Cases
  • 3.2 Invocation Patterns
  • 3.3 Architectural Anti-Patterns
• 4. Retrieval-Augmented Generation (RAG)
  • 4.1 Foundation Model Limitations
  • 4.2 What Is RAG?
  • 4.3 Retrieval Components in Bedrock
  • 4.4 When to Use (or Not Use) RAG
• 5. Choosing the Right Foundation Model
  • 5.1 Model Families and Selection Criteria
  • 5.2 The Converse API and Model Portability
• 6. Configuring Inference Parameters
  • 6.1 Temperature, Top-P and Top-K
  • 6.2 Max Tokens, Stop Sequences and Length Control
  • 6.3 Prompt Engineering and System Prompts
• 7. Model Customization and Fine-Tuning
  • 7.1 Introduction to Customization in Bedrock
  • 7.2 Deploying and Evaluating a Fine-Tuned Model
• 8. Code Examples
  • 8.1 Synchronous Invocation and Streaming
  • 8.2 Chatbot with Conversation History (LangChain)
  • 8.3 RAG with Bedrock Knowledge Bases

1. Amazon Bedrock’s Role in AWS Architectures

1.1 Foundation Models in Generative AI Applications

Every generative AI application is powered by one or more foundation models. Each model has strengths for different use cases:

┌─────────────────────────────────────────────────────────────┐
│              Available Foundation Models                     │
├──────────────────────┬──────────────────────────────────────┤
│ Anthropic Claude     │ Reasoning, agentic workflows         │
│ Amazon Nova          │ Text, image, video, extraction       │
│ Meta Llama           │ Open-weight, flexible, customizable  │
│ Mistral AI           │ Multilingual, edge to frontier       │
│ OpenAI, DeepSeek...  │ Additional models                    │
└──────────────────────┴──────────────────────────────────────┘

The problem without Bedrock:

graph TD
    App[Application] --> |API Key 1| Claude[Anthropic Claude]
    App --> |API Key 2| Llama[Meta Llama]
    App --> |API Key 3| Nova[Amazon Nova]
    App --> |API Key 4| Mistral[Mistral AI]
    
    style App fill:#ff9999
    style Claude fill:#ffcc99
    style Llama fill:#ffcc99
    style Nova fill:#ffcc99
    style Mistral fill:#ffcc99

Challenges: multiple subscriptions, multiple API keys, different prompt formats, variable response structures, multiple billing.

The solution with Bedrock:

graph TD
    App[Application] --> |AWS Credentials| Bedrock[Amazon Bedrock\nUnified API]
    Bedrock --> Claude[Anthropic Claude]
    Bedrock --> Llama[Meta Llama]
    Bedrock --> Nova[Amazon Nova]
    Bedrock --> Mistral[Mistral AI]
    
    style App fill:#99ccff
    style Bedrock fill:#99ff99

Definition: Amazon Bedrock is a fully managed generative AI service by AWS. It provides secure access to leading foundation models through a unified set of APIs.

1.2 Amazon Bedrock as a Managed Inference Service

flowchart LR
    Prompt[Prompt\ncreated by app] --> API[Bedrock API]
    API --> Inference[Managed\nInference]
    Inference --> Response[Response:\ntext, image,\nembeddings...]

Key advantages:

When to choose which AWS AI service:

flowchart TD
    Start([What do I need?]) --> Q1{Use AI\nimmediately?}
    Q1 -->|Yes| AmazonQ[Amazon Q\nor Kiro IDE]
    Q1 -->|No| Q2{Build an app\nwithout managing\ninfrastructure?}
    Q2 -->|Yes| Bedrock[Amazon Bedrock]
    Q2 -->|No| Q3{Full control\nof ML lifecycle?}
    Q3 -->|Yes| SageMaker[Amazon SageMaker]
    Q3 -->|No| GPU[GPU / Trainium\n/ Inferentia]
    
    style Bedrock fill:#99ff99
    style SageMaker fill:#ffcc99
    style AmazonQ fill:#99ccff

1.3 Bedrock vs SageMaker

1.4 Bedrock in the Application

sequenceDiagram
    participant App as Application
    participant Bedrock as Amazon Bedrock
    participant FM as Foundation Model

    App->>Bedrock: API request (prompt + model_id + parameters)
    Bedrock->>Bedrock: IAM access control
    Bedrock->>Bedrock: Request validation
    Bedrock->>Bedrock: Guardrails (optional)
    Bedrock->>Bedrock: Knowledge Base (optional)
    Bedrock->>FM: Transformed prompt (model-specific format)
    FM->>Bedrock: Generated response (tokens)
    Bedrock->>Bedrock: Response guardrails (optional)
    Bedrock->>App: Structured JSON response

Application responsibilities:

• Create and send requests to Bedrock APIs
• Format data and generate the prompt
• Select the right inference parameters
• Handle responses and extract generated text/image
• Integrate the result into the user experience
• Handle errors and retries

Bedrock responsibilities:

• Host and manage foundation models on AWS infrastructure
• Route requests to the appropriate model (via model ID)
• Invoke the model to process input and generate output
• Manage and scale underlying infrastructure
• Apply optional components (guardrails, knowledge bases)
• Return inference results in the specified format

2. Request-Response Flow for Model Invocation

2.1 The Request-Response Flow with Bedrock

flowchart TD
    App[Application] --> RuntimeAPI[Bedrock Runtime API]
    RuntimeAPI --> Auth[IAM Authentication]
    Auth --> FM[Foundation Model]
    FM --> Response[Generated Response]
    Response --> App

    subgraph Optional Components
        Guardrails[Guardrails\ninput/output evaluation]
        KB[Knowledge Bases\nseparate API - fetch data]
        Agents[Bedrock Agents\nmulti-step workflows]
    end

    RuntimeAPI -.-> Guardrails
    RuntimeAPI -.-> KB
    RuntimeAPI -.-> Agents

The JSON request includes:

• model_id: identifier of the foundation model to use
• prompt: text, image, or video depending on the model
• Inference parameters: temperature, top_p, response length

2.2 Transforming Requests into Model Inputs

flowchart LR
    Req[JSON Request\nfrom application] --> AC[IAM\nAccess Control]
    AC --> Val[Parameter\nValidation]
    Val --> Guard[Guardrails\n-optional-]
    Guard --> KB2[Knowledge Base\nenrichment\n-optional-]
    KB2 --> Trans[Model Format\nTransformation]
    Trans --> FM[Foundation Model\nInference]

Layer separation:

┌─────────────────────────────────────────────────────┐
│                   APPLICATION                        │
│  • Business logic and prompt construction            │
│  • Model-agnostic (interacts via Bedrock APIs)       │
│  • No changes needed when switching models           │
├─────────────────────────────────────────────────────┤
│                   AMAZON BEDROCK                     │
│  • Centralized access control (IAM)                  │
│  • Security policies and guardrails                  │
│  • Routing to the model                              │
│  • Model-specific format translation                 │
├─────────────────────────────────────────────────────┤
│               FOUNDATION MODELS                      │
│  • Inference: output generation                      │
│  • Replaceable (evaluate new models)                 │
│  • Swappable by changing the model identifier        │
└─────────────────────────────────────────────────────┘

2.3 Returning and Consuming Model Responses

flowchart TD
    FM[Foundation Model\nGenerates response] --> Bedrock[Amazon Bedrock]
    Bedrock --> Guard{Guardrails\nconfigured?}
    Guard -->|No| Format[Transformation\nto structured JSON]
    Guard -->|Yes| Eval[Content\nEvaluation]
    Eval --> Block{Violation\ndetected?}
    Block -->|Yes| Msg[Pre-configured\nmessage\nor masking]
    Block -->|No| Format
    Msg --> App[Application]
    Format --> Delivery{Delivery\nmode}
    Delivery -->|Synchronous| Full[Complete response\nin one payload]
    Delivery -->|Streaming| Chunks[Incremental\nresponse\nin chunks]
    Full --> App
    Chunks --> App

Response delivery modes:

Important note: Streaming does not make the model faster. Total generation time remains similar. What changes is how the response is delivered to the application and user.

Stop reasons:

• Complete response (natural end)
• Token limit reached
• Stop sequence encountered

3. Architectural Patterns for Bedrock Use Cases

3.1 Common Foundation Model Use Cases

┌───────────────────────────────────────────────────────────────┐
│         Use Cases and Characteristics                          │
├────────────────────┬──────────────┬────────────┬──────────────┤
│ Use Case           │ Input Size   │ Output Size│ Latency      │
├────────────────────┼──────────────┼────────────┼──────────────┤
│ Summarization      │ Large        │ Small      │ Tolerable    │
│ Classification     │ Small        │ Structured │ Low          │
│ Chat / Q&A         │ Variable     │ Variable   │ Low          │
│ Creative generation│ Small        │ Large      │ Moderate     │
│ Reasoning          │ Variable     │ Variable   │ High         │
└────────────────────┴──────────────┴────────────┴──────────────┘

Model selection by task:

3.2 Invocation Patterns

flowchart TD
    Start([Use Case]) --> RT{Real-time\nresponse?}
    RT -->|Yes| Len{Output\nlength?}
    Len -->|Short| Sync[Synchronous\nInvocation]
    Len -->|Long| Stream[Streaming\nInvocation]
    RT -->|No| Vol{Volume?}
    Vol -->|Individual| Async[Asynchronous\nInvocation]
    Vol -->|Mass| Batch[Batch\nInvocation]

    style Sync fill:#99ccff
    style Stream fill:#99ff99
    style Async fill:#ffcc99
    style Batch fill:#ff9999

Synchronous invocation:

App ──────────────────────────────► Bedrock
     ◄────────── [WAITING] ──────────
     ◄──────── Complete response ────

• Use cases: classification, short Q&A, information extraction
• Advantage: simplicity
• Disadvantage: blocks during generation

Streaming invocation:

App ──────────────────────────────► Bedrock
     ◄── token ◄── token ◄── token ── (progressive)

• Use cases: chatbots, long summaries, creative generation
• Advantage: reduced perceived latency, better UX
• Note: same total time as synchronous

Asynchronous invocation:

App ──────────────────────────────► Bedrock
     ◄── Acknowledgment ───────────

     (later)

Bedrock ──────────────────────────► S3 bucket
                                    (result)

• Use cases: pipelines, long tasks, video generation, batch processing
• Advantage: eliminates timeouts and persistent connections
• Disadvantage: not suitable for real-time responses

Batch invocation:

• Upload a dataset of prompts to S3
• Submit a single processing job
• Examples: mass summarization, document classification

3.3 Architectural Anti-Patterns

mindmap
  root((Bedrock\nAnti-patterns))
    Prompt as a database
      Context windows have limits
      More tokens = higher costs
      Manage state externally
    Conditional logic in the prompt
      Unreliable results
      Business logic belongs in the application
      Hard to unit test
    Wrong invocation pattern
      Synchronous for thousands of documents
      Asynchronous for real-time chat
    Unvalidated user input
      Prompt injection possible
      Validate before sending to model
    Ignoring capacity tiers
      On-demand vs provisioned
      Same tier for POC and production

4. Retrieval-Augmented Generation (RAG)

4.1 Foundation Model Limitations

graph TD
    FM[Foundation Model] --> L1[No access to\nproprietary data]
    FM --> L2[Possible\nhallucinations]
    FM --> L3[Training data\ncutoff date]
    
    L1 --> U1[Users expect\ndomain-specific\nresponses]
    L2 --> U2[Users need to\ntrust the\noutputs]
    L3 --> U3[Users expect\nup-to-date data]

The 3 major limitations:

1. Missing proprietary data: The model has never seen your internal data (orders, products, HR policies…)
2. Hallucinations: The probability-based generation mechanism can produce outputs that seem true but aren’t
3. Training cutoff: The model’s knowledge is limited to its training end date

4.2 What Is RAG?

Analogy: RAG turns your foundation model into a student taking an open-book exam. Instead of relying solely on what’s in memory (closed-book exam), the model can consult a reference book for precise answers.

Definition: Retrieval-Augmented Generation is the process of:

1. Retrieve: Query and fetch information from a data source
2. Augment: Enrich the prompt with that information
3. Generate: Send the enriched prompt to the foundation model for a more accurate, grounded response

flowchart LR
    Q[User\nQuestion] --> R[RETRIEVE\nSearch in\nKnowledge Base]
    R --> A[AUGMENT\nPrompt\nenrichment]
    A --> G[GENERATE\nGeneration by\nthe foundation model]
    G --> Resp[Grounded\nresponse +\ncitations]

    KB[(Knowledge Base\nProprietary data\nUp-to-date data)] --> R

    style R fill:#99ccff
    style A fill:#ffcc99
    style G fill:#99ff99

Grounded response: A response where the model output is anchored in an authoritative source, rather than generated solely from training data.

4.3 Retrieval Components in Bedrock

flowchart TD
    App[Application\ne.g.: order status 101] --> Bedrock

    subgraph Bedrock
        direction TB
        KBQuery[Knowledge Base\nQuery]
        Retrieve[Retrieve\nrelevant documents]
        Augment[Prompt\nenrichment]
        Invoke[Foundation Model\nInvocation]
    end

    Bedrock --> KB[(Knowledge Base\nOrders\nProducts\nShipments)]
    KB --> Bedrock
    Bedrock --> App2[Application\nResponse + Citations]

Bedrock Knowledge Bases — Phase 1: Preprocessing (ingestion):

flowchart LR
    DS[Data sources\nS3, Confluence, SharePoint] --> Chunk[Splitting\ninto chunks]
    Chunk --> Embed[Converting to\nvector embeddings\nvia embedding model]
    Embed --> VStore[(Vector Store\nOpenSearch Serverless\nor other)]

Bedrock Knowledge Bases — Phase 2: Runtime (query):

flowchart LR
    Q[User\nQuery] --> QEmbed[Convert to\nembedding]
    QEmbed --> Search[Vector search\nin Vector Store]
    Search --> Top[Top relevant\nchunks]
    Top --> Augment[Prompt\nenrichment]
    Augment --> FM[Foundation Model\nGeneration]
    FM --> Resp[Grounded\nresponse]

Two RAG patterns in Bedrock:

Knowledge Base configuration (chunking):

{
  "chunkingStrategy": "FIXED_SIZE",
  "fixedSizeChunkingConfiguration": {
    "maxTokens": 512,
    "overlapPercentage": 20
  }
}

OpenSearch Serverless configuration:

{
  "settings": {
    "index.knn": "true",
    "number_of_shards": 1,
    "knn.algo_param.ef_search": 512,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "vector": {
        "type": "knn_vector",
        "dimension": 1536,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2"
        }
      },
      "text": { "type": "text" },
      "text-metadata": { "type": "text" }
    }
  }
}

4.4 When to Use (or Not Use) RAG

Use RAG when:

graph TD
    U1[App requires proprietary\nor internal data] 
    U2[Factual accuracy critical\nhallucination reduction]
    U3[Compliance: citations\ntraced to authorized sources]
    U4[Data that changes frequently\nwithout retraining the model]

Do NOT use RAG when:

graph TD
    N1[General facts already covered\nby training data]
    N2[Classification, tagging,\nsentiment analysis]
    N3[Critical latency or\nminimal cost is priority]
    N4[Summarizing entire documents\nRAG retrieves chunks, not everything]

Trade-off comparison:

5. Choosing the Right Foundation Model

5.1 Model Families and Selection Criteria

Model families available in Bedrock:

┌─────────────────────────────────────────────────────────────────────┐
│                    Bedrock Model Families                            │
├──────────────────┬──────────────────────────────────────────────────┤
│ Amazon Nova      │ Nova Micro, Lite, Pro, Premier, Nova 2           │
│                  │ Text, image, video, speech                       │
├──────────────────┼──────────────────────────────────────────────────┤
│ Anthropic Claude │ Haiku, Sonnet, Opus                              │
│                  │ Strong reasoning, long context, tool use         │
├──────────────────┼──────────────────────────────────────────────────┤
│ Meta Llama       │ Llama family (open-weight)                       │
│                  │ Flexible, customizable, cost-effective            │
├──────────────────┼──────────────────────────────────────────────────┤
│ Mistral AI       │ Large 3, Mistral 3                               │
│                  │ Multilingual, agentic, edge to frontier          │
├──────────────────┼──────────────────────────────────────────────────┤
│ Cohere           │ Command R, R+, Embed v3                          │
│                  │ Optimized for RAG and enterprise search          │
├──────────────────┼──────────────────────────────────────────────────┤
│ Others           │ AI21, DeepSeek, OpenAI, Qwen...                 │
└──────────────────┴──────────────────────────────────────────────────┘

Selection criteria:

mindmap
  root((Model\nSelection))
    Capabilities
      Reasoning quality
      Context window size
      Multimodal support
      Agentic capabilities
    Latency
      Time to first token
      Total response time
      Chain-of-thought impact
    Cost
      Input token price
      Output token price
      Cached tokens
      Provisioned throughput

Token comparison — Nova 2 Lite vs Claude Sonnet:

Prompt: "Give me three names for a developer productivity tool..."

┌─────────────────┬────────────────┬─────────────────────────┐
│                 │ Nova 2 Lite    │ Claude Sonnet           │
├─────────────────┼────────────────┼─────────────────────────┤
│ Input tokens    │ 84             │ 48                      │
│ Output tokens   │ 101            │ 128                     │
│ Latency         │ ~1 sec         │ ~3 sec                  │
└─────────────────┴────────────────┴─────────────────────────┘

Each model uses its own tokenizer!

5.2 The Converse API and Model Portability

The Converse API provides a consistent request/response format for all text models in Bedrock. It enables swapping models without changing application code.

flowchart LR
    App[Application\nCode unchanged] --> ConverseAPI[Converse API]
    ConverseAPI --> Claude[Claude Haiku]
    ConverseAPI --> Nova[Nova Pro]
    ConverseAPI --> Llama[Meta Llama]
    
    style ConverseAPI fill:#99ff99

Converse API trade-offs:

6. Configuring Inference Parameters

6.1 Temperature, Top-P and Top-K

How a model chooses the next token:

Each time a model generates a word, it computes a probability distribution over all possible next tokens.

Example: “The field was full of ___”

Candidate tokens and probabilities:
┌───────────┬─────────────┐
│ Token     │ Probability │
├───────────┼─────────────┤
│ horses    │ 60%         │
│ zebras    │ 20%         │
│ unicorns  │ 10%         │
│ cats      │ 5%          │
│ clouds    │ 3%          │
│ other     │ 2%          │
└───────────┴─────────────┘

Temperature:

graph LR
    T0[Temperature = 0\nDeterministic\nConsistent\nPredictable] 
    T1[Temperature = 1\nCreative\nVaried\nSurprising]
    
    T0 -->|"0.0 → 0.3\nClassification\nCode\nFacts"| Mid
    Mid -->|"0.5 → 0.7\nChat\nConversation"| T1
    T1 -->|"0.7 → 1.0\nCreative writing\nBrainstorming"| End[...]

Top-P (Nucleus Sampling):
Limits candidate tokens to those collectively representing X% of the probability mass.

• Top-P = 0.9 → only tokens forming 90% of probability are considered

Top-K:
Specifies the number of most probable eligible tokens.

• Top-K = 50 → only the top 50 most probable tokens are eligible

Quick guide by task type:

6.2 Max Tokens, Stop Sequences and Length Control

flowchart LR
    Gen[Model\nGeneration] --> Check{Which stop\nfirst?}
    Check -->|Natural end| End1[Complete response\n✅]
    Check -->|Stop sequence\nreached| End2[Stop at the\ndefined sequence\n✅]
    Check -->|Max tokens\nreached| End3[Stop at limit\n⚠️ may cut\nmid-sentence]

Max tokens:

• Defines the upper limit on the number of tokens in the response
• Output tokens are the most expensive — important for cost control
• ⚠️ May stop mid-sentence

Stop sequences:

• Character strings that signal the model to stop generating immediately
• Very useful for structured outputs (JSON, XML)
• Examples: "}" for JSON, "###" to delimit sections

Best practices for response length:

Best practice: Always set both max tokens AND stop sequences as a double safety net.

6.3 Prompt Engineering and System Prompts

Well-structured prompt with XML tags:

<ticket>
  Subject: Billing discrepancy after plan upgrade
  Body: I upgraded from Basic to Pro on March 1st. My invoice shows the full 
  Pro rate for the entire month, but I only upgraded mid-cycle. I expected a 
  prorated charge. Account ID: 8834-XQ.
</ticket>

<instructions>
  Classify this ticket. Only use information from the ticket above. 
  Do not assume or invent any company policies. If policy information is 
  needed to resolve the issue, set needs_policy_lookup to true.
</instructions>

<output_schema>
  {
    "category": "string",
    "priority": "Low | Medium | High",
    "summary": "string",
    "needs_policy_lookup": true/false,
    "reasoning": "string"
  }
</output_schema>

Impact of a well-defined system prompt:

Comparing two system prompts for a support assistant:

❌ Verbose system prompt:
"You are a highly experienced, professional, and empathetic customer support 
specialist working for CloudTools Pro, a leading SaaS platform for developer 
productivity tools. Your primary responsibility is to assist customers with 
questions related to billing, account management, technical troubleshooting, 
and feature requests. You should always provide thorough, comprehensive, and 
detailed responses that cover all possible angles..."

✅ Concise, structured system prompt:
"You are a support assistant for CloudTools Pro. 
Respond in JSON with fields: answer, confidence (high/medium/low), 
needs_escalation (true/false). Be concise."

The concise prompt gives more predictable, less expensive, and more programmatically parseable responses.

7. Model Customization and Fine-Tuning

7.1 Introduction to Customization in Bedrock

flowchart TD
    Custom[Customization\nin Bedrock] --> FT[Supervised\nFine-tuning]
    Custom --> CPT[Continued\nPre-training]
    
    FT --> FTDesc[Adapt a model to a\nspecific task with\nlabeled data]
    CPT --> CPTDesc[Continue training\non your unlabeled\ndata]
    
    FT --> UseCase1[Domain-specific classification\nGeneration with particular style\nBusiness data extraction]
    CPT --> UseCase2[Enrich model with\nproprietary knowledge\nSpecific jargon and terminology]

When to customize vs use RAG:

Training data format for fine-tuning:

{
  "prompt": "Classify the sentiment of the following review: 'The product arrived quickly but the packaging was damaged.'",
  "completion": "Mixed - Positive: fast delivery, Negative: damaged packaging"
}

7.2 Deploying and Evaluating a Fine-Tuned Model

flowchart TD
    Train[Fine-tuned model\nready] --> PThroughput[Purchase Provisioned\nThroughput]
    PThroughput --> Support[AWS Support Case\nApproved in a few\nbusiness days]
    Support --> Deploy[Model deployed\nand hosted]
    Deploy --> Validate[Validation via\nPlayground or API]
    Validate --> Eval[Bedrock\nEvaluation]
    
    subgraph Bedrock Evaluation
        Prog[Programmatic\npredefined metrics]
        Judge[Model as a Judge\nmodel evaluates responses]
        HumanAWS[Human - AWS\nwork team]
        HumanBYO[Human - Bring\nyour own team]
    end
    
    Eval --> Prog
    Eval --> Judge
    Eval --> HumanAWS
    Eval --> HumanBYO

Using the custom model ARN in the API:

# Base model
model_id = "amazon.nova-lite-v1:0"

# Replace with your fine-tuned custom model ARN
model_id = "arn:aws:bedrock:us-east-1:123456789:provisioned-model/your-custom-model-arn"

Available evaluation types:

8. Code Examples

8.1 Synchronous Invocation and Streaming

Synchronous invocation with Boto3 (Converse API):

import boto3

bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')

# Synchronous invocation
response = bedrock_client.converse(
    modelId="amazon.nova-micro-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "text": "Which team won the FIFA World Cup in 2022?"
                }
            ]
        }
    ],
    inferenceConfig={
        "maxTokens": 512,
        "temperature": 0.2,
        "topP": 0.9
    }
)

# Extract the response
output_text = response['output']['message']['content'][0]['text']
print(output_text)

# Metrics
usage = response['usage']
print(f"Input tokens: {usage['inputTokens']}")
print(f"Output tokens: {usage['outputTokens']}")
print(f"Stop reason: {response['stopReason']}")

Streaming invocation:

import boto3

bedrock_client = boto3.client('bedrock-runtime', region_name='us-east-1')

# Streaming invocation
response = bedrock_client.converse_stream(
    modelId="amazon.nova-micro-v1:0",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "text": "Write fifteen sentences explaining why bees are beneficial for the garden."
                }
            ]
        }
    ],
    inferenceConfig={
        "maxTokens": 1024,
        "temperature": 0.7
    }
)

# Process the stream token by token
for event in response['stream']:
    if 'contentBlockDelta' in event:
        text = event['contentBlockDelta']['delta'].get('text', '')
        print(text, end='', flush=True)
    elif 'messageStop' in event:
        print(f"\nStop reason: {event['messageStop']['stopReason']}")

Asynchronous (batch) invocation:

import boto3

bedrock_client = boto3.client('bedrock', region_name='us-east-1')

# Submit a batch job (prompts in S3)
response = bedrock_client.create_model_invocation_job(
    roleArn="arn:aws:iam::123456789:role/BedrockBatchRole",
    modelId="amazon.nova-micro-v1:0",
    jobName="batch-summarization-job",
    inputDataConfig={
        "s3InputDataConfig": {
            "s3Uri": "s3://my-bucket/input-prompts/",
            "s3InputFormat": "JSONL"
        }
    },
    outputDataConfig={
        "s3OutputDataConfig": {
            "s3Uri": "s3://my-bucket/output-results/"
        }
    }
)

job_arn = response['jobArn']
print(f"Job submitted: {job_arn}")

8.2 Chatbot with Conversation History (LangChain)

import boto3
import os
from langchain_core.chat_history import InMemoryChatMessageHistory
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables.history import RunnableWithMessageHistory
from langchain_aws import ChatBedrock

# Create the Bedrock client
bedrock_client = boto3.client(
    'bedrock-runtime',
    region_name=os.environ.get("AWS_DEFAULT_REGION", "us-east-1")
)

# Configure the model with LangChain
chat_model = ChatBedrock(
    model_id="meta.llama3-8b-instruct-v1:0",
    client=bedrock_client
)

# Define the prompt template with conversation history
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "Answer the following questions as best you can."),
        ("placeholder", "{chat_history}"),
        ("human", "{input}"),
    ]
)

# Create in-memory history
history = InMemoryChatMessageHistory()

def get_history():
    return history

# Chain the components
chain = prompt | chat_model | StrOutputParser()

wrapped_chain = RunnableWithMessageHistory(
    chain,
    get_history,
    history_messages_key="chat_history",
)

# Example conversation
response = wrapped_chain.invoke({"input": "What is LangChain?"})
print(response)

# Follow-up question — the model remembers the context
response = wrapped_chain.invoke({"input": "Tell me more about its key features"})
print(response)

# Display history
print(history)

8.3 RAG with Bedrock Knowledge Bases

Create a Knowledge Base and ingest data:

import boto3
import json
import random

boto3_session = boto3.session.Session()
region_name = boto3_session.region_name
bedrock_agent_client = boto3_session.client('bedrock-agent', region_name=region_name)

suffix = random.randrange(200, 900)
bucket_name = f'bedrock-kb-{suffix}'
index_name = f"bedrock-sample-index-{suffix}"

# Define OpenSearch Serverless configuration
opensearchServerlessConfiguration = {
    "collectionArn": "arn:aws:aoss:us-east-1:123456789:collection/my-collection",
    "vectorIndexName": index_name,
    "fieldMapping": {
        "vectorField": "vector",
        "textField": "text",
        "metadataField": "text-metadata"
    }
}

# Chunking strategy
chunkingStrategyConfiguration = {
    "chunkingStrategy": "FIXED_SIZE",
    "fixedSizeChunkingConfiguration": {
        "maxTokens": 512,
        "overlapPercentage": 20
    }
}

# Create the Knowledge Base
create_kb_response = bedrock_agent_client.create_knowledge_base(
    name=f'bedrock-kb-demo-{suffix}',
    description='Demo knowledge base for RAG',
    roleArn="arn:aws:iam::123456789:role/BedrockKBRole",
    knowledgeBaseConfiguration={
        "type": "VECTOR",
        "vectorKnowledgeBaseConfiguration": {
            "embeddingModelArn": "arn:aws:bedrock:us-east-1::foundation-model/amazon.titan-embed-text-v1"
        }
    },
    storageConfiguration={
        "type": "OPENSEARCH_SERVERLESS",
        "opensearchServerlessConfiguration": opensearchServerlessConfiguration
    }
)

knowledge_base_id = create_kb_response['knowledgeBase']['knowledgeBaseId']

# Query the Knowledge Base (Retrieve and Generate)
bedrock_agent_runtime = boto3_session.client(
    'bedrock-agent-runtime',
    region_name=region_name
)

response = bedrock_agent_runtime.retrieve_and_generate(
    input={
        "text": "What is the status of order 101?"
    },
    retrieveAndGenerateConfiguration={
        "type": "KNOWLEDGE_BASE",
        "knowledgeBaseConfiguration": {
            "knowledgeBaseId": knowledge_base_id,
            "modelArn": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0"
        }
    }
)

print(response['output']['text'])

# Citations
for citation in response.get('citations', []):
    for ref in citation.get('retrievedReferences', []):
        print(f"Source: {ref['location']['s3Location']['uri']}")

Search Terms

amazon · bedrock · aws · ai · machine · web · services · model · foundation · rag · invocation · architectural · cases · customization · flow · inference · patterns · request-response