Azure Knowledge Mining and Document Intelligence Solutions

Target Certification: AI-102 — Designing and Implementing a Microsoft Azure AI Solution
Scope: Azure AI Search · Azure AI Document Intelligence

Table of Contents

1. Course Overview
2. Azure AI Search — Knowledge Mining
   • 2.1 Context and Problem to Solve
   • 2.2 What Azure AI Search Offers
   • 2.3 High-Level Process
   • 2.4 Creating a Search Resource
   • 2.5 Replicas, Partitions and Search Units
   • 2.6 Billing Model
   • 2.7 Index Attributes
   • 2.8 Suggesters
   • 2.9 Demo — Create an Index
   • 2.10 Demo — Query an Index
   • 2.11 Custom Skills
   • 2.12 Knowledge Stores
   • 2.13 Semantic and Vector Search
3. Azure AI Document Intelligence
   • 3.1 Context and Problem to Solve
   • 3.2 Service Overview
   • 3.3 Processing Models
   • 3.4 Sending Requests to the Service
   • 3.5 Demo — Process Documents with Pre-built Models
   • 3.6 Custom Models — Template and Neural
   • 3.7 Composed Models
   • 3.8 Demo — Train a Custom Model
   • 3.9 Integration with Azure AI Search
4. AI-102 Exam Tips
5. Quick Reference — Model IDs

1. Course Overview

This course covers two high-level Azure AI services used in the AI-102 certification:

Prerequisites: Comfortable with Python or C# and navigating the Azure portal.

2. Azure AI Search — Knowledge Mining

Note on Renaming

Azure Cognitive Search  →  Azure AI Search
(old name)                   (new name)

Older training materials refer to Azure Cognitive Search, but it is the same service with the same concepts.

2.1 Context and Problem to Solve

Organizations face recurring challenges with their data:

• Large quantities of data in various formats (text, images, structured databases, etc.)
• Difficulty accessing and searching across these heterogeneous sources
• Need to enrich raw data with additional context (sentiment, language, entities, etc.)

┌──────────────┐  ┌──────────────┐  ┌──────────────┐
│  Text        │  │   Images     │  │  SQL         │
│  files       │  │              │  │  databases   │
└──────┬───────┘  └──────┬───────┘  └──────┬───────┘
       │                 │                 │
       └─────────────────┼─────────────────┘
                         │
                         ▼
              ┌──────────────────┐
              │  Hard to         │
              │  search and      │
              │  leverage!       │
              └──────────────────┘

2.2 What Azure AI Search Offers

Azure AI Search provides a unified search experience:

• Index across multiple sources — single pane of glass for all search needs
• Support for multiple data types — text, images, structured databases
• Filtering for more relevant results
• Data enrichment via AI capabilities (image descriptions, sentiment, language, etc.)

2.3 High-Level Process

The process unfolds in two main steps:

flowchart LR
    DS["🗄️ Data Source(s)\n(Azure + non-Azure)"]
    IDX["⚙️ Indexer\n(indexing engine)"]
    SK["🧠 Skillset\n(AI enrichment)"]
    INDEX["📋 Search Index\n(structured JSON)"]
    QUERY["🔍 Query / Search\n(filter by attributes)"]

    DS --> IDX
    IDX --> SK
    SK --> IDX
    IDX --> INDEX
    INDEX --> QUERY

Step 1 — Create the searchable index:

• Internal Azure data sources (Cosmos DB, Azure Blob) or external
• Apply a skillset for AI enrichment
• The indexer acts as the indexing engine and produces the index in JSON format

Step 2 — Query the index:

• Search and filter by field attributes
• Available attributes: key, searchable, filterable, sortable, facetable, retrievable

💡 It is also possible to write directly to the index without going through a data source, though this is not the most common use case.

2.4 Creating a Search Resource

Pricing Tier (SKU)

Unlike other Azure AI services, the SKU chosen has a major impact on service capabilities and cannot be changed after creation.

⚠️ Important: Once the SKU is chosen and the resource created, it is impossible to change. A new resource must be created and data migrated.

Authentication Options

flowchart TD
    A[Azure AI Search Authentication] --> B["API Key\n(Primary / Secondary)"]
    A --> C["Microsoft Entra ID\n(Managed Identity + RBAC)"]
    C --> D["✅ Recommended for production"]

2.5 Replicas, Partitions and Search Units

Definitions

Search Unit Calculation

$$\text{Search Units} = \text{Replicas} \times \text{Partitions}$$

Example: Standard S1 (max 36 Search Units)

  Replicas  ×  Partitions  =  Search Units
     3      ×      3       =       9
     6      ×      6       =      36  (max)
    12      ×      3       =      36  (max)
     1      ×      1       =       1  (default)

flowchart LR
    subgraph Replica1["Replica 1 (VM Node)"]
        P1["Partition 1\n(1/3 of index)"]
        P2["Partition 2\n(1/3 of index)"]
        P3["Partition 3\n(1/3 of index)"]
    end
    subgraph Replica2["Replica 2 (VM Node)"]
        P4["Partition 1\n(1/3 of index)"]
        P5["Partition 2\n(1/3 of index)"]
        P6["Partition 3\n(1/3 of index)"]
    end
    INDEX["📋 Full Index"] --> Replica1
    INDEX --> Replica2

High Availability

✅ Unlike the SKU, replicas and partitions can be modified after creation via the scale configuration.

2.6 Billing Model

Azure AI Search uses a pay-per-allocation model (not pay-per-consumption):

┌──────────────────────────────────────────────┐
│  Azure AI Search Billing Model               │
│                                              │
│  ✗ NOT billed by usage                       │
│    (unlike other AI services)                │
│                                              │
│  ✓ Billed by allocation:                     │
│    You pay for compute + storage             │
│    EVEN if you don't use the service         │
│                                              │
│  ➜ Remember to delete unused resources!      │
└──────────────────────────────────────────────┘

2.7 Index Attributes

Each document in the index is represented as JSON with fields that have specific attributes.

Example indexed document

{
  "@search.score": 1,
  "listingId": "OTM4NjQyOQ2",
  "beds": 3,
  "baths": 3,
  "description": "This is a bachelor apartment and is perfect for entertaining.",
  "sqft": 5832,
  "daysOnMarket": 249,
  "status": "active",
  "source": "Lucas Roberts Homes",
  "city": "Bellevue",
  "region": "wa",
  "price": 1650456,
  "tags": [
    "bachelor apartment",
    "entertaining",
    "great views"
  ]
}

The 5 Main Attributes

flowchart LR
    F[Field] --> S["🔍 searchable\nFull-text search\nMulti-word tokenized"]
    F --> R["👁 retrievable\nVisible in results\n(≠ searchable)"]
    F --> FL["🔽 filterable\nExact match filter\n(no tokenization)"]
    F --> SO["↕️ sortable\nAscending/descending sort"]
    F --> FA["📊 facetable\nCount per category"]

Searchable vs Retrievable Comparison

Attribute Summary Table

⚠️ Warning: Field attributes are locked at index creation. Only the retrievable attribute can be modified after the fact on certain fields.

2.8 Suggesters

A suggester is a special attribute that provides autocomplete / type-ahead functionality (similar to Google suggestions).

User types: "dub"
                ↓
     Suggester active on "city" field
                ↓
     Suggestions: "Dubai", "Dublin", "Dubrovnik"

• Configured per field when creating the index
• Determines which fields benefit from autocomplete

2.9 Demo — Create an Index

Demo Prerequisites

• An Azure AI Search resource (Basic SKU or higher)
• An Azure Storage Account with a container containing PDF files
• Optional: an Azure AI multi-service resource for higher quality AI enrichment

Steps via the Azure portal import wizard

Step 1: Connect data
  ├── Source: Azure Blob Storage
  ├── Source name: margies-travel
  └── Connection string → container "margies"

Step 2: Configure cognitive skills (Skillset)
  ├── Link an Azure AI service (multi-service recommended)
  ├── Enable OCR (merge all text into merged_content)
  └── Selected skills:
      ├── Extract location names (locations)
      ├── Extract key phrases (key phrases)
      ├── Detect language
      ├── Generate image tags
      └── Generate image captions

Step 3: Customize target index
  ├── Name: margies-index
  └── Field attributes:
      ├── metadata_storage_size   → retrievable, filterable, sortable
      ├── metadata_storage_name   → retrievable, filterable, sortable, searchable
      ├── metadata_author         → retrievable, filterable, sortable, facetable, searchable
      ├── locations               → retrievable, filterable, searchable
      ├── keyphrases              → retrievable, filterable, searchable
      └── language                → retrievable, filterable, searchable

Step 4: Create the indexer
  ├── Name: margies-indexer
  ├── Execution: once
  └── Advanced options: Base64 encode keys ✓

Example index configuration (JSON)

{
  "name": "margies-index",
  "fields": [
    {
      "name": "metadata_storage_name",
      "type": "Edm.String",
      "retrievable": true,
      "filterable": true,
      "sortable": true,
      "searchable": true
    },
    {
      "name": "metadata_author",
      "type": "Edm.String",
      "retrievable": true,
      "filterable": true,
      "sortable": true,
      "facetable": true,
      "searchable": true
    },
    {
      "name": "locations",
      "type": "Collection(Edm.String)",
      "retrievable": true,
      "filterable": true,
      "searchable": true
    },
    {
      "name": "keyphrases",
      "type": "Collection(Edm.String)",
      "retrievable": true,
      "filterable": true,
      "searchable": true
    }
  ]
}

Example Skillset with OCR, vision and extraction (JSON)

{
  "name": "search-skillset",
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Vision.OcrSkill",
      "name": "ocrSkill",
      "context": "/document/normalized_images/*",
      "lineEnding": "Space",
      "defaultLanguageCode": "en",
      "detectOrientation": true,
      "inputs": [{ "name": "image", "source": "/document/normalized_images/*" }],
      "outputs": [
        { "name": "text",       "targetName": "text" },
        { "name": "layoutText", "targetName": "layoutText" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Text.MergeSkill",
      "name": "mergeSkill",
      "context": "/document",
      "inputs": [
        { "name": "text",          "source": "/document/content" },
        { "name": "itemsToInsert", "source": "/document/normalized_images/*/text" },
        { "name": "offsets",       "source": "/document/normalized_images/*/contentOffset" }
      ],
      "outputs": [
        { "name": "mergedText", "targetName": "merged_content" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Text.LanguageDetectionSkill",
      "name": "languageDetectionSkill",
      "context": "/document",
      "inputs": [
        { "name": "text", "source": "/document/content" }
      ],
      "outputs": [
        { "name": "languageCode", "targetName": "language" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Text.KeyPhraseExtractionSkill",
      "name": "keyPhraseExtractionSkill",
      "context": "/document/content",
      "defaultLanguageCode": "en",
      "inputs": [
        { "name": "text",         "source": "/document/content" },
        { "name": "languageCode", "source": "/document/language" }
      ],
      "outputs": [
        { "name": "keyPhrases", "targetName": "keyPhrases" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Vision.ImageAnalysisSkill",
      "name": "imageAnalysisSkill",
      "context": "/document/normalized_images/*",
      "visualFeatures": ["tags", "description"],
      "inputs": [{ "name": "image", "source": "/document/normalized_images/*" }],
      "outputs": [
        { "name": "tags",    "targetName": "imageTags" },
        { "name": "objects", "targetName": "imageObjects" }
      ]
    }
  ]
}

2.10 Demo — Query an Index

Available Query Methods

1. Azure portal interface (Search Explorer) — simple view or JSON
2. REST API — direct HTTP requests
3. SDK (.NET, Python, JavaScript, Java)

Supported Query Syntaxes

flowchart TD
    Q[Query types] --> S["Simple Syntax\n(default, interpretive)"]
    Q --> A["Lucene / Full Syntax\n(advanced)"]
    S --> S1["Common use cases\nBasic wildcards"]
    A --> A1["Wildcards, fuzzy, proximity\nRegex, term boosting"]

Operators — simple syntax

Operators — advanced syntax (Lucene)

Example JSON queries (portal)

Return all documents with a count:

{
  "search": "*",
  "count": true
}

Select specific fields:

{
  "search": "*",
  "count": true,
  "select": "metadata_storage_name, metadata_author, locations"
}

Phrase search:

{
  "search": "San Francisco",
  "select": "content, metadata_storage_name, keyphrases"
}

Search with author filter:

{
  "search": "San Francisco",
  "filter": "metadata_author eq 'Reviewer'",
  "select": "content, metadata_storage_name"
}

REST API Call

POST https://{service-name}.search.windows.net/indexes/{index-name}/docs/search?api-version=2024-07-01
Content-Type: application/json
api-key: <YOUR_API_KEY>

{
  "search": "San Francisco",
  "filter": "metadata_author eq 'Reviewer'",
  "select": "content, metadata_storage_name",
  "count": true,
  "top": 10
}

.NET Application — Querying the Index

using Azure;
using Azure.Search.Documents;
using Azure.Search.Documents.Models;

string searchEndpoint = "https://contoso-search.search.windows.net";
string indexName      = "margies-index";
string apiKey         = "<YOUR_API_KEY>";

var credential = new AzureKeyCredential(apiKey);
var client = new SearchClient(new Uri(searchEndpoint), indexName, credential);

var options = new SearchOptions
{
    Size = 5,
    IncludeTotalCount = true,
    QueryType = SearchQueryType.Simple
};
options.Select.Add("metadata_storage_name");
options.Select.Add("metadata_author");
options.Select.Add("locations");

SearchResults<SearchDocument> response =
    await client.SearchAsync<SearchDocument>("San Francisco", options);

Console.WriteLine($"Total results: {response.TotalCount}");

await foreach (var result in response.GetResultsAsync())
{
    Console.WriteLine($"\n--- Score: {result.Score} ---");
    foreach (var field in new[] { "metadata_storage_name", "metadata_author", "locations" })
    {
        if (result.Document.TryGetValue(field, out var value))
            Console.WriteLine($"{field}: {value}");
    }
}

2.11 Custom Skills

Why Custom Skills?

Built-in skills cover most use cases, but some scenarios require domain-specific processing.

┌───────────────────────────────────────────────────────┐
│                Custom Skills — Flow                   │
│                                                       │
│  Data Source → Indexer → Skillset                     │
│                              │                        │
│                    ┌─────────▼────────┐               │
│                    │ Custom Web API   │               │
│                    │ Skill            │               │
│                    └─────────┬────────┘               │
│                              │  HTTPS                 │
│                              ▼                        │
│                    ┌─────────────────┐                │
│                    │ External endpoint│               │
│                    │ ┌─────────────┐ │                │
│                    │ │Azure Function│ │                │
│                    │ │Azure Container│ │               │
│                    │ │Instance     │ │                │
│                    │ │AKS, App Svc │ │                │
│                    │ └─────────────┘ │                │
│                    └─────────────────┘                │
│                              │                        │
│                              ▼                        │
│                        Search Index                   │
└───────────────────────────────────────────────────────┘

When to Use a Custom Skill?

Use Case — Legal Entity Extraction

Extraction of specific legal clauses not found in built-in skills:

• Legal clauses
• Regulation-specific terms and conditions
• Non-disclosure agreements (NDAs)
• Regulatory compliance

Integration in the Skillset (JSON)

{
  "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
  "name": "customLegalEntitySkill",
  "description": "Extracts specific legal entities",
  "uri": "https://{azure-function-url}/api/extract-legal-entities?code={function-key}",
  "httpMethod": "POST",
  "timeout": "PT30S",
  "context": "/document",
  "batchSize": 1,
  "inputs": [
    {
      "name": "text",
      "source": "/document/content"
    }
  ],
  "outputs": [
    {
      "name": "legalClauses",
      "targetName": "legalClauses"
    },
    {
      "name": "complianceFlags",
      "targetName": "complianceFlags"
    }
  ]
}

Example GenAI Skill (summarization via Azure OpenAI)

{
  "@odata.type": "#Microsoft.Skills.Custom.ChatCompletionSkill",
  "name": "Summarizer",
  "description": "Summarizes document content.",
  "context": "/document",
  "uri": "https://foundry.cognitiveservices.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2025-01-01-preview",
  "httpMethod": "POST",
  "timeout": "PT30S",
  "batchSize": 1,
  "inputs": [
    { "name": "text",          "source": "/document/content" },
    { "name": "systemMessage", "source": "='You are a concise AI assistant.'" },
    { "name": "userMessage",   "source": "='Summarize the following text:'" }
  ],
  "outputs": [
    { "name": "response", "targetName": "summary" }
  ],
  "commonModelParameters": {
    "temperature": 0.3,
    "maxTokens": 0
  }
}

2.12 Knowledge Stores

What is a Knowledge Store?

By default, the created index is stored within the Azure AI Search resource. A knowledge store is an optional secondary storage location for AI-enriched content.

flowchart TD
    DS["🗄️ Data Source"] --> IDX["⚙️ Indexer"]
    IDX --> SS["🧠 Skillset\n(with Shaper Skill)"]
    SS --> IDX
    IDX --> INDEX["📋 Search Index\n(in Azure AI Search)"]
    IDX --> KS["🏪 Knowledge Store\n(in Azure Storage)"]
    KS --> OBJ["📄 Object Projection\nBlob Container (JSON)"]
    KS --> FILE["🖼️ File Projection\nBlob Container (images/binaries)"]
    KS --> TABLE["📊 Table Projection\nAzure Table Storage"]

Projection Types

Knowledge Store Use Cases

1. Customer feedback analysis — sentiment analysis exported to Power BI
2. ETL pipelines — send to Azure Data Factory for additional transformations
3. Knowledge mining — independent exploration of the search index
4. Data modeling — integration with third-party tools

Example Skillset with Knowledge Store and Shaper Skill (JSON)

{
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Util.ShaperSkill",
      "name": "ShaperSkill",
      "context": "/document",
      "inputs": [
        { "name": "DocTitle",   "source": "/document/metadata_storage_name" },
        { "name": "DocContent", "source": "/document/merged_content" },
        { "name": "DocOcrText", "source": "/document/normalized_images/*/text" }
      ],
      "outputs": [
        { "name": "output", "targetName": "objectprojection" }
      ]
    }
  ],
  "knowledgeStore": {
    "storageConnectionString": "<STORAGE_CONNECTION_STRING>",
    "projections": [
      {
        "files": [
          {
            "storageContainer": "ocrimages",
            "generatedKeyName": "ocrimagesKey",
            "source": "/document/normalized_images/*"
          }
        ],
        "objects": [
          {
            "storageContainer": "contents",
            "generatedKeyName": "contentsKey",
            "source": "/document/objectprojection"
          }
        ],
        "tables": [
          {
            "tableName": "DocumentsTable",
            "generatedKeyName": "docKey",
            "source": "/document/objectprojection",
            "inputs": [
              { "name": "DocumentTitle", "source": "/document/objectprojection/DocTitle" },
              { "name": "DocOcrText",    "source": "/document/objectprojection/DocOcrText" }
            ]
          }
        ]
      }
    ]
  }
}

⚠️ Limitation: Two projections in the same group cannot share the same source. Use separate groups if needed.

2.13 Semantic and Vector Search

Comparison of Search Types

flowchart TD
    Q[Search types] --> KW["🔤 Keyword Search\n(full-text)"]
    Q --> SEM["💬 Semantic Search\nLLM-based re-ranking"]
    Q --> VEC["📐 Vector Search\nNumerical similarity (embeddings)"]
    Q --> HYB["🔀 Hybrid Search\nCombines all three"]

    SEM --> SEM1["'How to protect my network?'\n→ returns docs on Azure Firewall, NSG"]
    VEC --> VEC1["Search 'dog'\n→ also returns docs with 'canine'"]
    VEC --> VEC2["Multilingual search\n'dog' → returns docs with 'hund' (DE)"]

Vector Search — Pipeline

sequenceDiagram
    participant APP as Application
    participant OPENAI as Azure OpenAI
    participant INDEX as AI Search Index

    APP->>OPENAI: Query text ("wall of fire")
    OPENAI-->>APP: Numerical vector [0.023, -0.014, ...]
    APP->>INDEX: POST /docs/search (vector query)
    INDEX-->>APP: Similar documents (Azure Firewall docs!)

Skillset with Azure OpenAI Embedding Skill

{
  "@odata.type": "#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill",
  "name": "aoaiEmbedding",
  "description": "Generate embeddings for vector search",
  "context": "/document",
  "resourceUri": "https://my-openai.openai.azure.com",
  "deploymentId": "text-embedding-ada-002",
  "dimensions": 1536,
  "modelName": "text-embedding-ada-002",
  "inputs": [
    { "name": "text", "source": "/document/merged_content" }
  ],
  "outputs": [
    { "name": "embedding", "targetName": "contentVector" }
  ]
}

REST Request for Vector Search

# Step 1 — Get the query vector via Azure OpenAI
POST https://my-openai.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2024-02-01
Content-Type: application/json
api-key: <OPENAI_API_KEY>

{
  "input": "wall of fire protection"
}

# Step 2 — Vector search in AI Search
POST https://{service}.search.windows.net/indexes/{index}/docs/search?api-version=2024-07-01
Content-Type: application/json
api-key: <SEARCH_API_KEY>

{
  "vectorQueries": [
    {
      "kind": "vector",
      "vector": [0.023, -0.014, 0.007],
      "fields": "contentVector",
      "k": 5
    }
  ]
}

💡 Result: The search for "wall of fire" returns documents about Azure Firewall — vector semantics understands the concept!

3. Azure AI Document Intelligence

Note on Renaming

Azure Form Recognizer  →  Azure AI Document Intelligence
(old name)                 (new name)

Older materials refer to Form Recognizer, but it is the same service with the same concepts.

3.1 Context and Problem to Solve

• Manual data entry from documents is time-consuming and error-prone
• Scanned documents (image PDF, JPEG, PNG) require OCR (Optical Character Recognition)
• Businesses process large volumes of documents that need to be handled quickly and accurately

┌──────────────────────────────────────────────────────────┐
│              Challenge: Document Processing              │
│                                                          │
│  📄 Invoice PDF  📄 Receipt image  📄 Scanned form       │
│       │               │                │                │
│       └───────────────┼────────────────┘                │
│                       ▼                                  │
│          ❌ Manual entry → Slow and error-prone          │
│                       ▼                                  │
│          ✅ Azure AI Document Intelligence               │
│             → Automatic and accurate extraction          │
└──────────────────────────────────────────────────────────┘

3.2 Service Overview

Azure AI Document Intelligence allows you to:

• Analyze and extract data from documents (text or image)
• Perform OCR on scanned PDF files and images (JPEG, PNG, TIFF, BMP)
• Automate business workflows
• Use a wide variety of pre-built models (continuously expanding)
• Create a searchable index by integrating Azure AI Search
• Train custom models for non-standard forms

Possible Integrations

flowchart LR
    DI["⚙️ Azure AI\nDocument Intelligence"]
    DI --> SEARCH["🔍 Azure AI Search\n(searchable index)"]
    DI --> LOGIC["🔄 Logic Apps"]
    DI --> POWER["⚡ Power Automate"]
    DI --> FUNC["🛠️ Azure Functions"]
    DI --> APP["💻 Application\n(SDK C#, Python, JS, Java)"]

3.3 Processing Models

Two main categories of pre-built models

┌─────────────────────────────────────────────────────────────┐
│                   Pre-built Models                          │
│                                                             │
│  CATEGORY 1: Document Analysis (generalized)               │
│  ┌────────────┐  ┌──────────────────┐  ┌────────────────┐  │
│  │   Read     │  │ General Document │  │    Layout      │  │
│  │ Simple OCR │  │  Key-value pairs │  │ Tables +       │  │
│  │ Plain text │  │  Catch-all       │  │ checkboxes     │  │
│  └────────────┘  └──────────────────┘  └────────────────┘  │
│                                                             │
│  CATEGORY 2: Scenario-Specific (by document type)         │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐  │
│  │ Invoice  │ │ Receipt  │ │  ID Doc  │ │Business Card │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────────┘  │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐  │
│  │ US W2/   │ │ Mortgage │ │ Health   │ │  Marriage    │  │
│  │ 1098/1040│ │   Docs   │ │ Ins Card │ │  Certificate │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────────┘  │
└─────────────────────────────────────────────────────────────┘

Detail of Document Analysis Models

Key Differences Between Analysis Models

flowchart TD
    A[Which model to choose?] --> B{Text images\nonly?}
    B -- Yes --> C["📖 Read\nOCR + language detection"]
    B -- No --> D{Need to\nrecognize\nfields?}
    D -- No --> C
    D -- Yes --> E{Checkboxes\nor tables?}
    E -- Yes --> F["📐 Layout\nStructure + selection marks"]
    E -- No --> G["📋 General Document\nCatch-all key-value pairs"]
    G --> H{Known specific\nformat?}
    H -- Yes --> I["🎯 Scenario-Specific\n(Invoice, Receipt, etc.)"]
    H -- No --> G

Fields Extracted by Scenario-Specific Models

3.4 Sending Requests to the Service

Two-Step Process (Asynchronous)

Unlike other Azure AI services, Document Intelligence uses a two-request process:

sequenceDiagram
    participant APP as Application
    participant DI as Document Intelligence

    APP->>DI: 1️⃣ POST Analyze Document\n(submits the document)
    DI-->>APP: Response: Result ID\n(NOT the extracted results!)
    
    Note over APP,DI: Asynchronous processing in progress...
    
    APP->>DI: 2️⃣ GET Analyze Result\n(with the Result ID)
    DI-->>APP: Response: Extracted results (JSON)

Why This Two-Step Process?

┌──────────────────────────────────────────────────────────┐
│           Asynchronous Processing — Benefits             │
│                                                          │
│  ✅ Large documents take time to process                 │
│  ✅ You can submit a large volume without waiting        │
│  ✅ Better scalability — "drop & retrieve"               │
│                                                          │
│  Typical flow:                                           │
│  1. Submit 100 documents → get 100 Result IDs            │
│  2. Continue other tasks                                 │
│  3. Retrieve results later                               │
└──────────────────────────────────────────────────────────┘

Model IDs — Model Identifiers

Each request must include a Model ID identifying the model to use.

REST API Request URLs

Analyze Document (POST):

{endpoint}/documentintelligence/documentModels/{modelId}:analyze?api-version=2024-02-29-preview

Analyze Result (GET):

{endpoint}/documentintelligence/documentModels/{modelId}/analyzeResults/{resultId}?api-version=2024-02-29-preview

3.5 Demo — Process Documents with Pre-built Models

Configuration Steps

1. Create a Document Intelligence resource in the Azure portal
2. Retrieve the endpoint and API key (Keys and Endpoint)
3. Configure environment variables locally

Complete Example via curl (REST API)

Initialize variables:

# initialize_variables.sh
export SERVICE_KEY="your_api_key"
export ENDPOINT="https://your-service.cognitiveservices.azure.com"
echo "Variables set: KEY=${SERVICE_KEY:0:5}... ENDPOINT=$ENDPOINT"

Step 1 — Analyze Document (submit the document):

# analyze_document.sh
curl -i -X POST \
  "${ENDPOINT}/documentintelligence/documentModels/prebuilt-invoice:analyze?api-version=2024-02-29-preview" \
  -H "Content-Type: application/json" \
  -H "Ocp-Apim-Subscription-Key: ${SERVICE_KEY}" \
  --data-ascii "{'urlSource': 'https://raw.githubusercontent.com/Azure-Samples/cognitive-services-REST-api-samples/master/curl/form-recognizer/sample-invoice.pdf'}"

The response contains an apim-request-id (Result ID) to keep for the next step.

Step 2 — Analyze Result (retrieve results):

# get_analyze_result.sh
curl -i -X GET \
  "${ENDPOINT}/documentintelligence/documentModels/prebuilt-invoice/analyzeResults/{RESULT_ID}?api-version=2024-02-29-preview" \
  -H "Ocp-Apim-Subscription-Key: ${SERVICE_KEY}"

Example JSON Output (invoice result)

{
  "status": "succeeded",
  "modelId": "prebuilt-invoice",
  "analyzeResult": {
    "documents": [
      {
        "docType": "invoice",
        "confidence": 0.99,
        "fields": {
          "AmountDue": {
            "type": "currency",
            "valueCurrency": { "amount": 610.00, "currencyCode": "USD" },
            "confidence": 0.98
          },
          "BillingAddress": {
            "type": "address",
            "content": "123 Main Street, Redmond, WA 98052",
            "confidence": 0.97
          },
          "BillingAddressRecipient": {
            "type": "string",
            "content": "Microsoft Corporation",
            "confidence": 0.99
          },
          "CustomerAddress": {
            "type": "address",
            "content": "456 Tech Avenue, Seattle, WA 98101",
            "confidence": 0.96
          },
          "InvoiceDate": {
            "type": "date",
            "valueDate": "2024-01-15",
            "confidence": 0.98
          },
          "InvoiceTotal": {
            "type": "currency",
            "valueCurrency": { "amount": 610.00, "currencyCode": "USD" },
            "confidence": 0.97
          }
        }
      }
    ]
  }
}

Python SDK — Pre-built Receipt Model

from azure.core.credentials import AzureKeyCredential
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.ai.documentintelligence.models import AnalyzeDocumentRequest, AnalyzeResult

endpoint = "https://your-service.cognitiveservices.azure.com/"
key      = "your_api_key"

client = DocumentIntelligenceClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

receipt_url = "https://raw.githubusercontent.com/Azure/azure-sdk-for-python/main/sdk/formrecognizer/azure-ai-formrecognizer/tests/sample_forms/receipt/contoso-receipt.png"

# Step 1: Analyze Document
poller = client.begin_analyze_document(
    "prebuilt-receipt",
    AnalyzeDocumentRequest(url_source=receipt_url)
)

# Step 2: Analyze Result (via the poller object)
result: AnalyzeResult = poller.result()

for document in result.documents:
    print(f"Type: {document.doc_type}")
    
    if "MerchantName" in document.fields:
        merchant = document.fields["MerchantName"]
        print(f"Merchant: {merchant.content} (confidence: {merchant.confidence:.2f})")
    
    if "TransactionDate" in document.fields:
        date = document.fields["TransactionDate"]
        print(f"Date: {date.value_date} (confidence: {date.confidence:.2f})")
    
    if "Items" in document.fields:
        items = document.fields["Items"]
        for item in items.value_list:
            desc = item.value_dictionary.get("Description")
            price = item.value_dictionary.get("TotalPrice")
            if desc and price:
                print(f"  Item: {desc.content} — {price.value_currency.amount}")
    
    if "Total" in document.fields:
        total = document.fields["Total"]
        print(f"Total: {total.value_currency.amount} (confidence: {total.confidence:.2f})")

C# SDK — Pre-built Receipt Model

using Azure;
using Azure.AI.DocumentIntelligence;

string endpoint = "https://your-service.cognitiveservices.azure.com/";
string apiKey   = "your_api_key";

var client = new DocumentIntelligenceClient(new Uri(endpoint), new AzureKeyCredential(apiKey));

Uri receiptUri = new Uri("https://raw.githubusercontent.com/Azure/azure-sdk-for-python/main/sdk/formrecognizer/azure-ai-formrecognizer/tests/sample_forms/receipt/contoso-receipt.png");

// Step 1: Analyze Document
Operation<AnalyzeResult> operation =
    await client.AnalyzeDocumentAsync(WaitUntil.Completed, "prebuilt-receipt", receiptUri);

// Step 2: Analyze Result
AnalyzeResult receipts = operation.Value;

foreach (AnalyzedDocument receipt in receipts.Documents)
{
    if (receipt.Fields.TryGetValue("MerchantName", out DocumentField merchantField)
        && merchantField.FieldType == DocumentFieldType.String)
        Console.WriteLine($"Merchant: {merchantField.ValueString} (confidence: {merchantField.Confidence})");

    if (receipt.Fields.TryGetValue("TransactionDate", out DocumentField dateField)
        && dateField.FieldType == DocumentFieldType.Date)
        Console.WriteLine($"Date: {dateField.ValueDate} (confidence: {dateField.Confidence})");

    if (receipt.Fields.TryGetValue("Items", out DocumentField itemsField)
        && itemsField.FieldType == DocumentFieldType.List)
    {
        foreach (DocumentField item in itemsField.ValueList)
        {
            if (item.FieldType == DocumentFieldType.Dictionary)
            {
                if (item.ValueDictionary.TryGetValue("Description", out var desc))
                    Console.WriteLine($"  Item: {desc.ValueString}");
                if (item.ValueDictionary.TryGetValue("TotalPrice", out var price)
                    && price.FieldType == DocumentFieldType.Currency)
                    Console.WriteLine($"  Price: {price.ValueCurrency.Amount}");
            }
        }
    }

    if (receipt.Fields.TryGetValue("Total", out DocumentField totalField)
        && totalField.FieldType == DocumentFieldType.Currency)
        Console.WriteLine($"Total: {totalField.ValueCurrency.Amount}");
}

⚠️ Security best practice: Never hardcode API keys in production code. Use Azure Key Vault to store credentials.

3.6 Custom Models — Template and Neural

When to Use a Custom Model?

• Forms not covered by pre-built models
• Industry-specific terminology
• Need for more accurate and predictable results
• Documents with a structure unique to your organization

Two Types of Custom Extraction Models

Custom Model Training Process

flowchart TD
    A["📁 Prepare documents\n(minimum 5 examples)"] --> B["☁️ Upload to\nAzure Blob Storage"]
    B --> C["🏷️ Label fields\nin Document Intelligence Studio\nor programmatically"]
    C --> D["📄 Generated JSON files\nocr.json + labels.json + fields.json"]
    D --> E["⚙️ Launch training\n(neural or template)"]
    E --> F["📊 Evaluate accuracy scores\n(confidence per field)"]
    F --> G{Satisfactory score?}
    G -- No, add examples --> C
    G -- Yes --> H["🚀 Deploy the model\n→ Unique Model ID assigned"]
    H --> I["💻 Use via SDK / REST API\nwith the custom Model ID"]

Files Required for Training

azure-blob-container/
├── fields.json               ← Defines fields to extract
├── form-001.jpg              ← Example form
├── form-001.jpg.labels.json  ← Label bounding boxes
├── form-001.jpg.ocr.json     ← OCR result
├── form-002.jpg
├── form-002.jpg.labels.json
├── form-002.jpg.ocr.json
└── ... (minimum 5 examples)

Example fields.json

{
  "fields": {
    "Merchant": { "fieldKey": "Merchant", "fieldType": "string" },
    "PhoneNumber": { "fieldKey": "PhoneNumber", "fieldType": "string" },
    "Website": { "fieldKey": "Website", "fieldType": "string" },
    "Email": { "fieldKey": "Email", "fieldType": "string" },
    "PurchaseOrder": { "fieldKey": "PurchaseOrder", "fieldType": "string" },
    "DatedAs": { "fieldKey": "DatedAs", "fieldType": "date" }
  }
}

Python SDK — Custom Model

from azure.core.credentials import AzureKeyCredential
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.ai.documentintelligence.models import AnalyzeDocumentRequest

endpoint = "YOUR_FORM_RECOGNIZER_ENDPOINT"
key      = "YOUR_FORM_RECOGNIZER_KEY"
model_id = "YOUR_CUSTOM_BUILT_MODEL_ID"
formUrl  = "YOUR_DOCUMENT_URL"

client = DocumentIntelligenceClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

# Step 1: Analyze Document with custom model
poller = client.begin_analyze_document(
    model_id,
    AnalyzeDocumentRequest(url_source=formUrl)
)

# Step 2: Analyze Result
result = poller.result()

print(f"Model used: {result.model_id}")

for idx, document in enumerate(result.documents):
    print(f"\n--- Document #{idx + 1} ---")
    print(f"  Type       : {document.doc_type}")
    print(f"  Confidence : {document.confidence:.2f}")
    for name, field in document.fields.items():
        print(f"  Field '{name}': '{field.content}' (confidence: {field.confidence:.2f})")

# Pages
for page in result.pages:
    print(f"\nPage {page.page_number}:")
    for line in page.lines:
        print(f"  Line: {line.content}")

# Tables
for i, table in enumerate(result.tables):
    print(f"\nTable {i + 1} ({table.row_count} rows × {table.column_count} columns):")
    for cell in table.cells:
        print(f"  [{cell.row_index}][{cell.column_index}]: '{cell.content}'")

C# SDK — Custom Model

using Azure;
using Azure.AI.DocumentIntelligence;

string endpoint = "YOUR_FORM_RECOGNIZER_ENDPOINT";
string key      = "YOUR_FORM_RECOGNIZER_KEY";
string modelId  = "YOUR_CUSTOM_MODEL_ID";
Uri    fileUri  = new Uri("YOUR_DOCUMENT_URI");

var client = new DocumentIntelligenceClient(new Uri(endpoint), new AzureKeyCredential(key));

Operation<AnalyzeResult> operation =
    await client.AnalyzeDocumentAsync(WaitUntil.Completed, modelId, fileUri);

AnalyzeResult result = operation.Value;
Console.WriteLine($"Model used: {result.ModelId}");

foreach (AnalyzedDocument document in result.Documents)
{
    Console.WriteLine($"Document type: {document.DocumentType}");
    foreach (var fieldKvp in document.Fields)
    {
        Console.WriteLine($"  Field '{fieldKvp.Key}': '{fieldKvp.Value.Content}' " +
                          $"(confidence: {fieldKvp.Value.Confidence})");
    }
}

// Pages
foreach (DocumentPage page in result.Pages)
{
    Console.WriteLine($"\nPage {page.PageNumber}:");
    foreach (var line in page.Lines)
        Console.WriteLine($"  Line: {line.Content}");

    foreach (var mark in page.SelectionMarks)
        Console.WriteLine($"  Checkbox: '{mark.State}', confidence: {mark.Confidence}");
}

// Tables
for (int i = 0; i < result.Tables.Count; i++)
{
    Console.WriteLine($"\nTable {i + 1}:");
    foreach (var cell in result.Tables[i].Cells)
        Console.WriteLine($"  [{cell.RowIndex}][{cell.ColumnIndex}]: '{cell.Content}'");
}

3.7 Composed Models

Concept

A Composed Model is a parent endpoint that groups multiple custom models under a single Model ID.

flowchart TD
    DOC["📄 Incoming document\n(unknown type)"] --> CM["🔀 Composed Model\n(single Model ID)"]
    CM --> CLS["🏷️ Classifier\n(identifies document type)"]
    CLS --> M1["📋 Custom Model 1\n(e.g., Purchase Order)"]
    CLS --> M2["📋 Custom Model 2\n(e.g., Supplier Invoice)"]
    CLS --> M3["📋 Custom Model 3\n(e.g., Customer Contract)"]
    M1 --> JSON["📊 JSON Result\n(consistent structure)"]
    M2 --> JSON
    M3 --> JSON

Benefits

• Single endpoint for all document types
• Automatic routing to the correct model
• No need to manually choose the model for each request
• Consistent output regardless of document variant

Composed Model Creation Process

1. Train Custom Model A (e.g., purchase-order-model)
         ↓
2. Train Custom Model B (e.g., supplier-invoice-model)
         ↓
3. In Document Intelligence Studio:
   → Go to the custom training project
   → Select models to compose
   → Test and validate the composed model
   → A new Model ID is assigned to the Composed Model
         ↓
4. Use the Composed Model ID in requests
   → Automatic routing to the correct child model

Best Practices

✅ Use clear naming conventions for child models
✅ Align field names between models for consistent output
✅ Avoid preview versions for production systems
✅ Set a confidence threshold — recommended: 0.5

3.8 Demo — Train a Custom Model

Training Data Used

For this demonstration, 5 images of purchase orders from a fictional company “Hero Limited” are used, with the following fields:

Demo Flow in Document Intelligence Studio

1. Azure Portal → Document Intelligence → Open Studio
         ↓
2. Custom models → Custom extraction model → Create project
   ├── Name: custom-purchase-order
   ├── Link the Document Intelligence resource
   └── Connect the Blob Storage container (training data)
         ↓
3. Labeling interface:
   ├── Pre-generated JSON files (ocr.json, labels.json) are loaded
   ├── Colored bounding boxes appear on each document
   ├── Click on a text area → select the corresponding field
   └── Add new fields (ShippedFrom*) via "Add a field"
         ↓
4. Train the model:
   ├── "Train" button
   ├── Name the Model ID: custom-purchase-order
   ├── Build mode: Neural (recommended)
   └── Duration: ~45 minutes (depending on volume)
         ↓
5. Test the model:
   ├── "Test" button
   ├── Upload a sample form
   └── Check extracted fields and confidence scores

💡 Demo result: Even with low confidence scores (30-40%), fields are extracted correctly. The confidence score is not always correlated to actual accuracy.

3.9 Integration with Azure AI Search

Scenario

Create a searchable index of all documents processed by Document Intelligence, allowing authorized teams to search through the processing history.

Example: Search through the supplier invoice history by amount, date, or vendor.

flowchart TD
    DOC["📄 New document\n(invoice, purchase order)"] --> SEARCH["🔍 Azure AI Search\n(indexing service)"]
    SEARCH --> IDX["⚙️ Indexer"]
    IDX --> SS["🧠 Skillset\n(Custom Web API Skill)"]
    SS --> FUNC["🛠️ Azure Function\n(or other endpoint)"]
    FUNC --> DI["⚙️ Azure AI\nDocument Intelligence"]
    DI --> FUNC2["📋 Extracted results\n(form fields)"]
    FUNC2 --> INDEX["📋 Search Index\n(enriched with DI extraction)"]
    INDEX --> QUERY["🔍 Queries\n(search, filtering)"]

Custom Skill Example Linking AI Search to Document Intelligence

{
  "description": "Skillset that invokes Document Intelligence (formerly Form Recognizer)",
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
      "name": "formrecognizer",
      "description": "Extracts fields from a form via Document Intelligence",
      "uri": "[AzureFunctionEndpointUrl]/api/analyze-form?code=[AzureFunctionDefaultHostKey]",
      "httpMethod": "POST",
      "timeout": "PT30S",
      "context": "/document",
      "batchSize": 1,
      "inputs": [
        {
          "name": "formUrl",
          "source": "/document/metadata_storage_path"
        },
        {
          "name": "formSasToken",
          "source": "/document/metadata_storage_sas_token"
        }
      ],
      "outputs": [
        {
          "name": "address",
          "targetName": "address"
        },
        {
          "name": "recipient",
          "targetName": "recipient"
        }
      ]
    }
  ]
}

ℹ️ Note: Some programmatic resources still reference Form Recognizer in code. This is the old name of the Document Intelligence service — the concepts remain identical.

4. AI-102 Exam Tips

Azure AI Search — Key Points

Field Attributes — Key Differences

searchable  → Can be included in the search query
               (multiple words are tokenized)
retrievable → Appears in results
               (may NOT be searchable)
filterable  → Exact match filter
               (no word tokenization)
sortable    → Ascending/descending sort
facetable   → Total count per category (e.g., hotels per city)

⚠️ searchable ≠ retrievable:
   - A retrievable but non-searchable field: visible in results,
     but not directly queryable

Azure AI Document Intelligence — Key Points

5. Quick Reference — Model IDs

┌────────────────────────────────────────────────────────────┐
│              Model IDs — Document Intelligence             │
│                                                            │
│  DOCUMENT ANALYSIS (generalized)                          │
│  prebuilt-read          → OCR + language                  │
│  prebuilt-layout        → Text + tables + marks           │
│  prebuilt-document      → General key-value pairs         │
│                                                            │
│  FINANCIAL DOCUMENTS                                      │
│  prebuilt-invoice       → Invoices                        │
│  prebuilt-receipt       → Receipts                        │
│  prebuilt-creditCard    → Credit cards                    │
│                                                            │
│  IDENTITY DOCUMENTS                                       │
│  prebuilt-idDocument    → Driver's licenses, passports    │
│  prebuilt-businessCard  → Business cards                  │
│                                                            │
│  LEGAL AND MEDICAL DOCUMENTS                              │
│  prebuilt-contract                  → Contracts           │
│  prebuilt-healthInsuranceCard.us    → US health insurance │
│  prebuilt-marriageCertificate.us    → Marriage certificate│
│                                                            │
│  US MORTGAGE DOCUMENTS                                    │
│  prebuilt-mortgage.us.1003          → Form 1003           │
│  prebuilt-mortgage.us.1008          → Form 1008           │
│  prebuilt-mortgage.us.closingDisclosure                   │
│                                                            │
│  US TAX FORMS                                             │
│  prebuilt-tax.us.w2                 → W2                  │
│  prebuilt-tax.us.1098 / 1098E / 1098T                    │
│  prebuilt-tax.us.1040 (variations)  → 1040               │
│  prebuilt-tax.us.1099 (variations)  → 1099               │
│                                                            │
│  CUSTOM                                                   │
│  {customModelName}      → Your trained model              │
└────────────────────────────────────────────────────────────┘

Search Terms

ai-102 · azure · knowledge · mining · document · intelligence · ai · services · artificial · generative · search · custom · model · models · json · process · index · skill · pre-built · rest · sdk · skillset · api · attributes