AI-102: Knowledge Mining and Information Extraction

Target Exam: AI-102 – Designing and Implementing a Microsoft Azure AI Solution

Table of Contents

1. Module 1 — Azure AI Search
   • 1.1 Core Concepts and Provisioning
   • 1.2 Data Sources and Search Index
   • 1.3 Skills and Skillsets (Built-in and Custom)
   • 1.4 Indexers — Creation and Execution
   • 1.5 Queries — Sorting and Filtering
   • 1.6 Knowledge Store — Projections
   • 1.7 Semantic and Vector Search
2. Module 2 — Azure AI Document Intelligence
   • 2.1 Service Provisioning
   • 2.2 Pre-built Models
   • 2.3 Custom Models — Training and Publishing
   • 2.4 Composed Models
3. Module 3 — Azure AI Content Understanding
   • 3.1 OCR Pipeline
   • 3.2 Summarization, Classification and Attribute Detection
   • 3.3 Entity, Table and Image Extraction
   • 3.4 Media Processing — Video and Audio
4. References and Resources

Module 1 — Azure AI Search

1.1 Core Concepts and Provisioning

Knowledge mining is the process of transforming unstructured content (PDFs, images, database records) into searchable, actionable insights. Azure AI Search is the Azure service dedicated to this purpose.

General Architecture

flowchart LR
    DS["🗄️ Data Sources\n(Azure Blob, SQL, Cosmos DB…)"]
    IDX["⚙️ Indexer\n(AI processing)"]
    SK["🧠 Skillset\n(built-in & custom skills)"]
    INDEX["📋 Search Index"]
    CLIENT["💻 Client App\n(REST / SDK)"]
    LLM["🤖 LLM / RAG\n(Azure OpenAI)"]

    DS --> IDX
    IDX --> SK
    SK --> IDX
    IDX --> INDEX
    INDEX --> CLIENT
    INDEX --> LLM

Pricing Tiers

Billing unit:
Search Unit = Number of Replicas × Number of Partitions

┌─────────────────────────────────────────────┐
│             Search Unit (SU)                │
│                                             │
│   SU = Replicas  ×  Partitions              │
│                                             │
│   Replica  = service instance               │
│             (load balancing, high avail.)   │
│   Partition = storage + I/O                 │
│             (index capacity)                │
└─────────────────────────────────────────────┘

Authentication Options

flowchart TD
    A[Azure AI Search Authentication] --> B["API Key\n(Primary / Secondary)"]
    A --> C["Entra ID\n(Managed Identity + RBAC)"]
    C --> D[Recommended for production]

Demo — Service Provisioning

1. Azure Portal → All services → AI Search → Create
2. Choose the Resource Group, name the service (e.g., fabrikam-search)
3. Select the region: East US 2
4. Choose the pricing tier (Free for testing)
5. Retrieve the API key: Settings → Keys

🔑 The service URL looks like: https://fabrikam-search.search.windows.net

1.2 Data Sources and Search Index

Supported Data Sources

mindmap
  root((Azure AI Search\nData Sources))
    Azure Storage
      Blob Storage
      Data Lake Gen2
      Table Storage
    Databases
      Azure SQL Database
      SQL Server on Azure VMs
      Azure Cosmos DB
    Other
      Microsoft Fabric OneLake

Typical Knowledge Mining Scenario

┌──────────────┐    ┌──────────────┐
│ Blob Storage │    │  Azure SQL   │
│ (PDF, Word,  │    │ (Metadata)   │
│  TXT files)  │    │              │
└──────┬───────┘    └──────┬───────┘
       │                   │
       └─────────┬─────────┘
                 ▼
         ┌───────────────┐
         │  AI Search    │
         │   Indexer     │
         └───────┬───────┘
                 ▼
         ┌───────────────┐
         │ Search Index  │◄──── Client queries
         └───────────────┘

Index Structure

An index is comparable to a database table:

Field Attributes

flowchart LR
    F[Field] --> S["searchable\nFull-text & vector search"]
    F --> FL["filterable\nFilter queries"]
    F --> SO["sortable\nResult sorting"]
    F --> FA["facetable\nGrouping / counting"]
    F --> R["retrievable\nReturned in results"]

Demo — Data Import

1. Click Import data from the AI Search service
2. Choose Azure Blob Storage as the data source
3. Configure extraction: content and metadata
4. Name the index: docs-index
5. Configure fields (retrievable, sortable, etc.)
6. Create the indexer (one-time or scheduled)

1.3 Skills and Skillsets (Built-in and Custom)

AI Enrichment Concept

AI enrichment is the integration of Azure AI Search with Azure AI Foundry tools. It enriches content to make it more searchable and useful.

flowchart LR
    DS["🗄️ Data Source\n(raw content)"] --> IDX["⚙️ Indexer"]
    IDX --> SS["🧠 Skillset\n(built-in + custom skills)"]
    SS --> IDX
    IDX --> INDEX["📋 Search Index\n(enriched)"]

Skill Types

When to Use a Custom Skill?

• Implement a transformation unique to your content
• Use a model trained on your specific domain
• Any custom scenario not covered by built-in skills

Skillset — JSON Example (built-in: language detection + key phrase extraction + GenAI summary)

{
  "name": "search-fabrikam-docs-skillset",
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Text.LanguageDetectionSkill",
      "name": "languageDetectionSkill",
      "context": "/document",
      "inputs": [
        { "name": "text", "source": "/document/content" }
      ],
      "outputs": [
        { "name": "languageCode", "targetName": "language" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Text.KeyPhraseExtractionSkill",
      "name": "keyPhraseExtractionSkill",
      "context": "/document/content",
      "defaultLanguageCode": "en",
      "inputs": [
        { "name": "text", "source": "/document/content" },
        { "name": "languageCode", "source": "/document/language" }
      ],
      "outputs": [
        { "name": "keyPhrases", "targetName": "keyPhrases" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Custom.ChatCompletionSkill",
      "name": "Summarizer",
      "description": "Summarizes the document content.",
      "context": "/document",
      "uri": "https://fabrikam-foundry.cognitiveservices.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2025-01-01-preview",
      "httpMethod": "POST",
      "timeout": "PT30S",
      "batchSize": 1,
      "inputs": [
        { "name": "text",          "source": "/document/content" },
        { "name": "systemMessage", "source": "='You are a concise AI assistant.'" },
        { "name": "userMessage",   "source": "='Summarize the following text:'" }
      ],
      "outputs": [
        { "name": "response", "targetName": "summary" }
      ],
      "commonModelParameters": {
        "temperature": 0.3,
        "maxTokens": 0
      }
    }
  ]
}

Best Practices — Design and Governance

✅ Check built-in skills before developing a custom skill
✅ Ensure the custom endpoint is reliable, secure, and scalable
✅ Monitor performance and handle errors in the pipeline

1.4 Indexers — Creation and Execution

Indexer Role

sequenceDiagram
    participant DS as Data Source
    participant IDX as Indexer
    participant SS as Skillset (optional)
    participant INDEX as Search Index

    IDX->>DS: Read data (blob, SQL, etc.)
    DS-->>IDX: Raw documents
    IDX->>SS: Apply AI enrichment
    SS-->>IDX: Enriched content
    IDX->>INDEX: Write enriched results

Key Characteristics

• Can run on demand or on a schedule (hourly, daily…)
• Detects changes: only new or modified items are processed
• Maintains an execution history (successes, failures, durations)
• Reset required to reprocess all documents

Example — Indexer Configuration (JSON)

{
  "name": "search-fabrikam-docs-indexer",
  "dataSourceName": "search-fabrikam-docs-datasource",
  "skillsetName": "search-fabrikam-docs-skillset",
  "targetIndexName": "search-fabrikam-docs",
  "parameters": {
    "configuration": {
      "parsingMode": "default",
      "dataToExtract": "contentAndMetadata"
    }
  },
  "fieldMappings": [
    {
      "sourceFieldName": "metadata_storage_name",
      "targetFieldName": "title"
    },
    {
      "sourceFieldName": "metadata_storage_path",
      "targetFieldName": "id",
      "mappingFunction": { "name": "base64Encode" }
    }
  ],
  "outputFieldMappings": [
    { "sourceFieldName": "/document/content/keyPhrases", "targetFieldName": "keyPhrases" },
    { "sourceFieldName": "/document/summary",            "targetFieldName": "summary" },
    { "sourceFieldName": "/document/contentVector",      "targetFieldName": "contentVector" }
  ]
}

OCR Skillset (Vision + Merge)

{
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Vision.OcrSkill",
      "name": "ocrSkill",
      "context": "/document/normalized_images/*",
      "lineEnding": "Space",
      "defaultLanguageCode": "en",
      "detectOrientation": true,
      "inputs": [{ "name": "image", "source": "/document/normalized_images/*" }],
      "outputs": [
        { "name": "text",       "targetName": "text" },
        { "name": "layoutText", "targetName": "layoutText" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Text.MergeSkill",
      "name": "mergeSkill",
      "context": "/document",
      "inputs": [
        { "name": "text",          "source": "/document/content" },
        { "name": "itemsToInsert", "source": "/document/normalized_images/*/text" },
        { "name": "offsets",       "source": "/document/normalized_images/*/contentOffset" }
      ],
      "outputs": [
        { "name": "mergedText", "targetName": "merged_content" }
      ]
    }
  ]
}

💡 Real-world scenario: Globomantics processes legal documents. PDFs arrive every hour. An indexer is configured to run automatically and add new files to the index.

1.5 Queries — Sorting and Filtering

Supported Query Syntaxes

flowchart TD
    Q[Query Types] --> S["Simple Syntax\n(default)"]
    Q --> A["Advanced / Lucene Syntax\n(full)"]
    S --> S1["Common use cases\nForgiving — interpretive"]
    A --> A1["Wildcards, fuzzy, proximity\nRegex, term boosting"]
    S --> FILTER[Filters & sorting supported]
    A --> FILTER

Operators — Simple Syntax

Operators — Advanced Syntax (Lucene)

REST API Call — Example

POST https://fabrikam-search.search.windows.net/indexes/docs-index/docs/search?api-version=2024-07-01
Content-Type: application/json
api-key: <YOUR_API_KEY>

{
  "search": "*",
  "queryType": "simple",
  "top": 5
}

.NET Application — Querying the Index

using Azure;
using Azure.Search.Documents;
using Azure.Search.Documents.Models;

// Configuration
private const string SearchEndpoint = "https://fabrikam-search.search.windows.net";
private const string IndexName      = "search-fabrikam-docs";
private const string ApiKey         = "<YOUR_API_KEY>";

private static readonly string[] FieldsToReturn = ["id", "title", "summary"];

static async Task Main(string[] args)
{
    var credential = new AzureKeyCredential(ApiKey);
    var client = new SearchClient(new Uri(SearchEndpoint), IndexName, credential);

    Console.WriteLine("Azure AI Search (.NET 8) — Document Search");
    Console.Write("Query> ");
    var query = Console.ReadLine();

    var options = new SearchOptions
    {
        Size = 5,
        IncludeTotalCount = true,
        QueryType = SearchQueryType.Simple   // or SearchQueryType.Full
    };

    foreach (var f in FieldsToReturn)
        options.Select.Add(f);

    SearchResults<SearchDocument> response =
        await client.SearchAsync<SearchDocument>(query, options);

    Console.WriteLine($"\nTotal: {response.TotalCount}");

    await foreach (var result in response.GetResultsAsync())
    {
        Console.WriteLine($"\n--- Score: {result.Score} ---");
        foreach (var field in FieldsToReturn)
        {
            if (result.Document.TryGetValue(field, out var value))
                Console.WriteLine($"{field}: {value}");
        }
    }
}

🔍 Query examples:

• * → returns all documents
• title:*.pdf → only PDF documents
• "security group"~2 → proximity search (Lucene full)

1.6 Knowledge Store — Projections

What Is the Knowledge Store?

The knowledge store is an optional output of the enrichment pipeline. Unlike the index (optimized for search), it is optimized for exploration, analysis, and integration with other services.

┌──────────────────────────────────────────────────────┐
│                  Knowledge Store                     │
│                                                      │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │
│  │   File      │  │   Object    │  │   Table     │  │
│  │ Projection  │  │ Projection  │  │ Projection  │  │
│  │             │  │             │  │             │  │
│  │ JSON, IMG   │  │ Hierarchical│  │ Rows/cols.  │  │
│  │ OCR output  │  │ JSON        │  │ Power BI    │  │
│  │ AI summaries│  │ sentiment   │  │ Synapse     │  │
│  └─────────────┘  └─────────────┘  └─────────────┘  │
│          ↓               ↓               ↓           │
│     Azure Blob      Azure Blob     Azure Table       │
│     Container       Container      Storage           │
└──────────────────────────────────────────────────────┘

Projection Types

Skillset Example with Knowledge Store and Shaper Skill

{
  "skills": [
    {
      "@odata.type": "#Microsoft.Skills.Vision.ImageAnalysisSkill",
      "name": "imageAnalysisSkill",
      "context": "/document/normalized_images/*",
      "visualFeatures": ["tags", "objects", "brands", "faces"],
      "inputs": [{ "name": "image", "source": "/document/normalized_images/*" }],
      "outputs": [
        { "name": "tags",    "targetName": "tags" },
        { "name": "objects", "targetName": "objects" }
      ]
    },
    {
      "@odata.type": "#Microsoft.Skills.Util.ShaperSkill",
      "name": "ShaperSkill",
      "context": "/document",
      "inputs": [
        { "name": "DocTitle",   "source": "/document/title" },
        { "name": "DocContent", "source": "/document/merged_content" },
        { "name": "DocOcrText", "source": "/document/normalized_images/*/text" }
      ],
      "outputs": [
        { "name": "output", "targetName": "objectprojection" }
      ]
    }
  ],
  "knowledgeStore": {
    "storageConnectionString": "<STORAGE_CONNECTION_STRING>",
    "projections": [
      {
        "files": [
          {
            "storageContainer": "ocrimages",
            "generatedKeyName": "ocrimagesKey",
            "source": "/document/normalized_images/*"
          }
        ],
        "objects": [
          {
            "storageContainer": "contents",
            "generatedKeyName": "contentsKey",
            "source": "/document/objectprojection"
          }
        ],
        "tables": [
          {
            "tableName": "OcrTextByDocument",
            "generatedKeyName": "ocrDocKey",
            "source": "/document/objectprojection",
            "inputs": [
              { "name": "DocumentTitle", "source": "/document/objectprojection/DocTitle" },
              { "name": "DocOcrText",    "source": "/document/objectprojection/DocOcrText" }
            ]
          }
        ]
      }
    ]
  }
}

⚠️ Limitation: Two projections in the same projection group cannot share the same source. Use separate projection groups if needed.

Governance Considerations

✅ Map each projection type to a specific storage account and container
✅ Align retention, cost, and security policies with enterprise standards
✅ Store only necessary fields to control storage growth

1.7 Semantic and Vector Search

Search Type Comparison

flowchart TD
    Q[Search Types] --> KW["Keyword Search\n(full-text)"]
    Q --> SEM["Semantic Search\nLLM-based re-ranking"]
    Q --> VEC["Vector Search\nNumerical similarity"]
    Q --> HYB["Hybrid Search\nCombines all three"]

    SEM --> SEM1["How do I protect my network?\n→ returns docs on Azure Firewall, NSG"]
    VEC --> VEC1["Search 'dog'\n→ also returns docs with 'canine'"]
    VEC --> VEC2["Multilingual search\n'dog' → returns docs with 'hund' (DE)"]

Semantic Search

Semantic search improves retrieval by re-ranking results using language understanding models. It works with existing indexes (no re-ingestion required).

Configuring a semantic configuration:

• title field: document title
• content fields: used for semantic search
• keywords fields: key phrases

Vector Search

A vector is the numerical representation of content. Vector search enables:

• Finding similar documents
• Image search
• Powering chatbots / agents via RAG
• Multilingual search

Vector search pipeline:

sequenceDiagram
    participant APP as Application
    participant OPENAI as Azure OpenAI
    participant INDEX as AI Search Index

    APP->>OPENAI: Query text
    OPENAI-->>APP: Numerical vector [0.23, -0.14, ...]
    APP->>INDEX: POST /docs/search (vector query)
    INDEX-->>APP: Closest documents (cosine similarity)

Skillset with Azure OpenAI Embedding Skill

{
  "@odata.type": "#Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill",
  "name": "aoaiEmbedding",
  "description": "Generate embeddings for vector search",
  "context": "/document",
  "resourceUri": "https://fabrikam-openai.openai.azure.com",
  "deploymentId": "text-embedding-ada-002",
  "dimensions": 1536,
  "modelName": "text-embedding-ada-002",
  "inputs": [
    { "name": "text", "source": "/document/summary" }
  ],
  "outputs": [
    { "name": "embedding", "targetName": "contentVector" }
  ]
}

REST Call for Vector Search (Postman)

# Step 1 — Get the query vector
POST https://fabrikam-openai.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2024-02-01
Content-Type: application/json
api-key: <OPENAI_API_KEY>

{
  "input": "Need information about wall of fire!"
}

# Step 2 — Vector search in AI Search
POST https://fabrikam-search.search.windows.net/indexes/search-fabrikam-docs/docs/search?api-version=2024-07-01
Content-Type: application/json
api-key: <SEARCH_API_KEY>

{
  "vectorQueries": [
    {
      "kind": "vector",
      "vector": [0.023, -0.014, ...],
      "fields": "contentVector",
      "k": 5
    }
  ]
}

💡 Result: Searching for "wall of fire" returns documents about Azure Firewall — vector semantics understands the concept!

Hybrid Search

Azure AI Search also supports hybrid search, which combines:

• Full-text search
• Vector search
• Semantic scoring

Module 2 — Azure AI Document Intelligence

Azure AI Document Intelligence automates data extraction from forms, receipts, invoices, and custom documents using AI.

Advantages

• Eliminates manual data entry
• Improves accuracy for knowledge mining and automation workloads
• Part of the Foundry Tools family
• Integrates with Logic Apps, Power Automate, Azure AI Search
• SDK available for C#, Python, JavaScript, Java

Available Models

┌────────────────────────────────────────────────────────────┐
│                Document Intelligence Models                │
│                                                            │
│  PRE-BUILT MODELS                                         │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│  │ Invoice  │ │ Receipt  │ │  ID Doc  │ │Business Card │ │
│  └──────────┘ └──────────┘ └──────────┘ └──────────────┘ │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│  │ Layout   │ │  Check   │ │ Contract │ │  Credit Card │ │
│  └──────────┘ └──────────┘ └──────────┘ └──────────────┘ │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────────┐ │
│  │ US Tax   │ │Mortgage  │ │Health Ins│ │  Marriage    │ │
│  │  Forms   │ │   Doc    │ │   Card   │ │  Certificate │ │
│  └──────────┘ └──────────┘ └──────────┘ └──────────────┘ │
│                                                            │
│  CUSTOM MODELS                                            │
│  ┌──────────────────┐   ┌───────────────────────────────┐ │
│  │ Template Model   │   │       Neural Model            │ │
│  │ (fixed formats)  │   │ (variable layouts, deep learn)│ │
│  └──────────────────┘   └───────────────────────────────┘ │
│                                                            │
│  COMPOSED MODEL                                           │
│  ┌────────────────────────────────────────────────────┐   │
│  │  Classifier → Routes to the right custom model     │   │
│  └────────────────────────────────────────────────────┘   │
└────────────────────────────────────────────────────────────┘

2.1 Service Provisioning

Demo — Resource Creation

1. All services → Document Intelligence → Create
2. Choose the Resource Group and region (East US 2)
3. Name the resource (e.g., fabrikam-docintell)
4. Pricing tier:
   • Free: 500 pages/month, 20 calls/minute
   • Standard: unlimited (usage-based billing)
5. Configure the firewall (recommended: private endpoints in production)
6. Enable managed identity if needed
7. Retrieve keys and endpoint: Resource Management → Keys and Endpoints

🌐 Document Intelligence Studio: accessible from the resource page to test models without code.

2.2 Pre-built Models

Processing Pipeline

flowchart LR
    DOC["📄 Documents\n(scanned or text)"] --> DI["⚙️ Document Intelligence\nService"]
    DI --> JSON["📋 JSON Result\n• Extracted fields\n• Confidence score\n• Text coordinates"]
    JSON --> DB["🗄️ Database\n(further processing)"]

Fields Extracted by Model

C# Code — Pre-built Receipt Model

using Azure;
using Azure.AI.DocumentIntelligence;

string endpoint = "https://fabrikam-docintell.cognitiveservices.azure.com/";
string apiKey   = "<YOUR_API_KEY>";

AzureKeyCredential credential = new AzureKeyCredential(apiKey);
DocumentIntelligenceClient client =
    new DocumentIntelligenceClient(new Uri(endpoint), credential);

// Sample document (Contoso receipt)
Uri receiptUri = new Uri(
    "https://raw.githubusercontent.com/Azure/azure-sdk-for-python/main/" +
    "sdk/formrecognizer/azure-ai-formrecognizer/tests/sample_forms/receipt/contoso-receipt.png");

Operation<AnalyzeResult> operation =
    await client.AnalyzeDocumentAsync(WaitUntil.Completed, "prebuilt-receipt", receiptUri);

AnalyzeResult receipts = operation.Value;

foreach (AnalyzedDocument receipt in receipts.Documents)
{
    // Merchant Name
    if (receipt.Fields.TryGetValue("MerchantName", out DocumentField merchantNameField)
        && merchantNameField.FieldType == DocumentFieldType.String)
    {
        Console.WriteLine(
            $"Merchant Name: '{merchantNameField.ValueString}', " +
            $"confidence: {merchantNameField.Confidence}");
    }

    // Transaction Date
    if (receipt.Fields.TryGetValue("TransactionDate", out DocumentField transactionDateField)
        && transactionDateField.FieldType == DocumentFieldType.Date)
    {
        Console.WriteLine(
            $"Transaction Date: '{transactionDateField.ValueDate}', " +
            $"confidence: {transactionDateField.Confidence}");
    }

    // Items
    if (receipt.Fields.TryGetValue("Items", out DocumentField itemsField)
        && itemsField.FieldType == DocumentFieldType.List)
    {
        foreach (DocumentField itemField in itemsField.ValueList)
        {
            if (itemField.FieldType == DocumentFieldType.Dictionary)
            {
                var itemFields = itemField.ValueDictionary;

                if (itemFields.TryGetValue("Description", out var descField))
                    Console.WriteLine($"  Description: '{descField.ValueString}', " +
                                      $"confidence: {descField.Confidence}");

                if (itemFields.TryGetValue("TotalPrice", out var priceField)
                    && priceField.FieldType == DocumentFieldType.Currency)
                    Console.WriteLine($"  Price: {priceField.ValueCurrency.Amount}, " +
                                      $"confidence: {priceField.Confidence}");
            }
        }
    }

    // Total
    if (receipt.Fields.TryGetValue("Total", out DocumentField totalField)
        && totalField.FieldType == DocumentFieldType.Currency)
    {
        Console.WriteLine($"Total: '{totalField.ValueCurrency.Amount}', " +
                          $"confidence: '{totalField.Confidence}'");
    }
}

⚠️ Best practice: Never hard-code API keys in production code. Use Azure Key Vault to store credentials.

Return JSON Structure (simplified example)

{
  "status": "succeeded",
  "analyzeResult": {
    "documents": [
      {
        "docType": "receipt",
        "confidence": 0.99,
        "fields": {
          "MerchantName": { "type": "string", "content": "Contoso",  "confidence": 0.99 },
          "TransactionDate": { "type": "date", "valueDate": "2019-06-10", "confidence": 0.98 },
          "Total": { "type": "currency", "valueCurrency": { "amount": 2516.28 }, "confidence": 0.97 },
          "Items": {
            "type": "array",
            "valueArray": [
              {
                "valueObject": {
                  "Description": { "content": "Surface Pro 6", "confidence": 0.99 },
                  "TotalPrice":  { "valueCurrency": { "amount": 1998.00 }, "confidence": 0.98 }
                }
              }
            ]
          }
        }
      }
    ]
  }
}

2.3 Custom Models — Training and Publishing

Custom Model Creation Workflow

flowchart TD
    A["📁 Prepare documents\n(5+ consistent examples)"] --> B["☁️ Upload to\nAzure Blob Storage"]
    B --> C["🏷️ Label documents\nin Document Intelligence Studio"]
    C --> D["⚙️ Launch training\n(neural or template)"]
    D --> E["📊 Evaluate accuracy scores"]
    E --> F{Score sufficient?}
    F -- No --> C
    F -- Yes --> G["🚀 Deploy and integrate\n(Code / REST API)"]

Custom Model Types

C# Code — Custom Model

using Azure;
using Azure.AI.DocumentIntelligence;

string endpoint = "YOUR_FORM_RECOGNIZER_ENDPOINT";
string key      = "YOUR_FORM_RECOGNIZER_KEY";
string modelId  = "<YOUR_CUSTOM_MODEL_ID>";
Uri    fileUri  = new Uri("<YOUR_DOCUMENT_URI>");

AzureKeyCredential credential = new AzureKeyCredential(key);
DocumentIntelligenceClient client =
    new DocumentIntelligenceClient(new Uri(endpoint), credential);

Operation<AnalyzeResult> operation =
    await client.AnalyzeDocumentAsync(WaitUntil.Completed, modelId, fileUri);

AnalyzeResult result = operation.Value;
Console.WriteLine($"Model used: {result.ModelId}");

foreach (AnalyzedDocument document in result.Documents)
{
    Console.WriteLine($"Document type: {document.DocumentType}");

    foreach (var fieldKvp in document.Fields)
    {
        string        fieldName = fieldKvp.Key;
        DocumentField field     = fieldKvp.Value;
        Console.WriteLine($"  Field '{fieldName}': '{field.Content}' " +
                          $"(confidence: {field.Confidence})");
    }
}

// Iterate over pages
foreach (DocumentPage page in result.Pages)
{
    Console.WriteLine($"Page {page.PageNumber}:");
    foreach (var line in page.Lines)
        Console.WriteLine($"  Line: {line.Content}");

    foreach (var mark in page.SelectionMarks)
        Console.WriteLine($"  Checkbox: '{mark.State}', confidence: {mark.Confidence}");
}

// Iterate over tables
for (int i = 0; i < result.Tables.Count; i++)
{
    Console.WriteLine($"Table {i + 1}:");
    foreach (var cell in result.Tables[i].Cells)
        Console.WriteLine($"  [{cell.RowIndex}][{cell.ColumnIndex}]: '{cell.Content}'");
}

Python Code — Custom Model

from azure.core.credentials import AzureKeyCredential
from azure.ai.documentintelligence import DocumentIntelligenceClient
from azure.ai.documentintelligence.models import AnalyzeDocumentRequest

endpoint = "YOUR_FORM_RECOGNIZER_ENDPOINT"
key      = "YOUR_FORM_RECOGNIZER_KEY"
model_id = "YOUR_CUSTOM_BUILT_MODEL_ID"
formUrl  = "YOUR_DOCUMENT_URL"

client = DocumentIntelligenceClient(
    endpoint=endpoint,
    credential=AzureKeyCredential(key)
)

poller = client.begin_analyze_document(
    model_id,
    AnalyzeDocumentRequest(url_source=formUrl)
)
result = poller.result()

for idx, document in enumerate(result.documents):
    print(f"--- Document #{idx + 1} ---")
    print(f"  Type       : {document.doc_type}")
    print(f"  Confidence : {document.confidence}")
    print(f"  Model      : {result.model_id}")
    for name, field in document.fields.items():
        print(f"  Field '{name}': type={field.type}, "
              f"value='{field.content}', confidence={field.confidence}")

# Pages
for page in result.pages:
    print(f"\nPage {page.page_number}:")
    for line in page.lines:
        print(f"  Line: {line.content}")
    for word in page.words:
        print(f"  Word: '{word.content}', confidence: {word.confidence}")

# Tables
for i, table in enumerate(result.tables):
    print(f"\nTable {i + 1}:")
    for cell in table.cells:
        print(f"  [{cell.row_index}][{cell.column_index}]: '{cell.content}'")

💡 Globomantics demo: 5 IT tickets in PDF used to train a model. Extracted fields: username, department, timestamp, description. After ~45 minutes of training, the model correctly extracts fields from new documents.

2.4 Composed Models

Concept

A composed model combines multiple custom models into a single endpoint. A classifier automatically determines which child model should process the document.

flowchart TD
    DOC["📄 Incoming Document"] --> CM["🔀 Composed Model"]
    CM --> CLS["🏷️ Classifier\n(detects document type)"]
    CLS --> M1["📋 Custom Model 1\n(e.g., Globomantics ticket)"]
    CLS --> M2["📋 Custom Model 2\n(e.g., GloboTickets ticket)"]
    M1 --> JSON["📊 Unified JSON\n(same structure)"]
    M2 --> JSON

Advantages

• Single endpoint for multiple document types
• Automatic routing to the correct model
• Consistent output regardless of document variant
• Simplifies version management of child models

Best Practices

✅ Use clear naming conventions for child models
✅ Align field names across models for consistent output
✅ Avoid preview versions for production systems
✅ Set a confidence threshold (confidence threshold) — recommended: 0.5

Composed Model Creation Workflow

1. Train Custom Model A (e.g., globo-custom-ticket)
         ↓
2. Train Custom Model B (e.g., globotickets-custom-ticket)
         ↓
3. Create a Custom Classification Model
   → Label examples of each type
   → Train the classifier
         ↓
4. Compose models A + B with the classifier
   → Name the composed model
   → Define routing rules by document type
   → Define splitMode (none = entire document)
         ↓
5. Test → Integrate via SDK / REST API

Module 3 — Azure AI Content Understanding

Full service name: Azure Content Understanding in Foundry Tools

Comparison — Document Intelligence vs Content Understanding

📌 At the time of course recording, both services are included in the AI-102 exam.

Azure Services Using OCR

┌─────────────────────────────────────────────────────┐
│                    OCR in Azure                      │
│                                                     │
│  ┌─────────────────┐  ┌─────────────────────────┐   │
│  │  Azure AI Vision│  │  Document Intelligence  │   │
│  │  Foundry Tools  │  │  Foundry Tools          │   │
│  └─────────────────┘  └─────────────────────────┘   │
│                                                     │
│  ┌─────────────────────────────────────────────┐    │
│  │       Content Understanding                 │    │
│  │       Foundry Tools                         │    │
│  └─────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────┘

3.1 OCR Pipeline

OCR Capabilities

• Recognizes both printed and handwritten text
• Support for multiple languages
• Detects layout elements: paragraphs, lines, bounding boxes
• Supports both image and document inputs (PDF)
• Returns text positions and confidence scores

Generic OCR Pipeline

flowchart LR
    DOCS["📄 Documents\n(PDFs, scanned images\nhandwritten notes)"] --> OCR["⚙️ OCR Service\n(Content Understanding)"]
    OCR --> JSON["📋 JSON Result\n• Extracted text\n• Coordinates\n• Confidence score"]
    JSON --> DB["🗄️ Database"]
    DB --> DOWNSTREAM["🔄 Downstream services\n• Summarizer\n• Azure AI Search\n• Generative AI"]

REST API Call — OCR Analyzer (Postman)

# Step 1 — Submit document (asynchronous POST)
POST https://fabrikam-foundry.services.ai.azure.com/contentunderstanding/analyzers/prebuilt-read:analyze?api-version=2025-11-01
Ocp-Apim-Subscription-Key: <FOUNDRY_SUBSCRIPTION_KEY>
Content-Type: application/json

{
  "inputs": [
    { "url": "https://contentunderstanding.ai.azure.com/assets/prebuilt/read_resume.png" }
  ]
}

# Response: 202 Accepted
# Headers: Operation-Location: https://fabrikam-foundry.../operations/<operationId>

# Step 2 — Retrieve result (GET)
GET https://fabrikam-foundry.services.ai.azure.com/contentunderstanding/analyzers/prebuilt-read/operations/<operationId>?api-version=2025-11-01
Ocp-Apim-Subscription-Key: <FOUNDRY_SUBSCRIPTION_KEY>

ℹ️ Asynchronous pattern: Content Understanding uses a POST → GET pattern (202 Accepted + Operation-Location header). This pattern is the same for all analyzers; only the analyzerID in the URL changes.

Services That Can Implement This Pipeline

• Logic Apps
• Azure Functions
• Web applications

3.2 Summarization, Classification and Attribute Detection

Text Analysis Models

mindmap
  root((Content Understanding Text Analysis))
    Summarization
      Extractive
        Selects key sentences
      Abstractive
        Generates new phrasing\npowered by GenAI
    Classification
      By sentiment
      By topic
      Powered by AI Language
    Attribute Detection
      Person names
      Dates
      Organization names
      Amounts

Summary Types

Complete Content Understanding Pipeline

┌──────────────────────────────────────────────────────────┐
│            Content Understanding Pipeline                │
│                                                          │
│  Document →  OCR  →  Summarization  →  Classification   │
│                             ↓               ↓           │
│                     Quick document    Routing to         │
│                     summary           the right service  │
│                                          ↓               │
│                              Attribute Detection         │
│                                          ↓               │
│                              Metadata tagging            │
│                              (department, date, etc.)    │
└──────────────────────────────────────────────────────────┘

Demo — Content Understanding Studio

Summarization (via Microsoft Foundry Playgrounds):

1. Foundry Portal → Playgrounds → Summarize text
2. Choose language, type (extractive/abstractive), length
3. Paste the text and click Run
4. Retrieve the JSON or code (C#, Java, JavaScript, Python, REST)

Custom classifier (via Content Understanding Studio):

flowchart TD
    A["Create classifier project\n(e.g., globo-classifier)"] --> B["Upload documents\n(invoice.pdf, receipt.pdf)"]
    B --> C["Define categories\n• globo-invoice → prebuilt-invoice analyzer\n• globo-receipt → prebuilt-receipt analyzer"]
    C --> D["Save and test"]
    D --> E["Build analyzer\n(e.g., customclassifier)"]
    E --> F["Integrate via REST API / SDK"]

💡 Content Understanding uses generative AI in the background to determine which category a document belongs to by analyzing its fields.

3.3 Entity, Table and Image Extraction

Available Analyzers

Analyzer Comparison (same document)

┌────────────────────────────────────────────────────────────────┐
│  Document: Expense Report (John Doe, Globomantics)             │
│                                                                │
│  Analyzer: layout          Analyzer: documentFields           │
│  ┌─────────────────────┐   ┌─────────────────────────────┐    │
│  │ Extracted text      │   │ engineer_name: John Doe     │    │
│  │ Recognized tables   │   │ department: Engineering     │    │
│  │ Detected images     │   │ office_location: Seattle    │    │
│  │ No field names      │   │ report_date: 2024-01-15     │    │
│  └─────────────────────┘   │ Items: [battery, cable...]  │    │
│                             └─────────────────────────────┘    │
│                                                                │
│  Analyzer: procurement      Analyzer: documentSearch          │
│  ┌─────────────────────┐   ┌─────────────────────────────┐    │
│  │ Same fields +       │   │ Summary: "This document is  │    │
│  │ structured items    │   │ an expense report for John  │    │
│  │ No images           │   │ Doe, including items and    │    │
│  └─────────────────────┘   │ images of products."        │    │
│                             └─────────────────────────────┘    │
└────────────────────────────────────────────────────────────────┘

REST Call — documentFields Analyzer

# Step 1 — Submit the PDF
POST https://fabrikam-foundry.services.ai.azure.com/contentunderstanding/analyzers/prebuilt-documentFields:analyze?api-version=2025-11-01
Ocp-Apim-Subscription-Key: <FOUNDRY_KEY>
Content-Type: application/json

{
  "inputs": [
    {
      "url": "https://<storage>.blob.core.windows.net/<container>/expense-report.pdf?<SAS_TOKEN>"
    }
  ]
}

# Response: 202 Accepted + Operation-Location header

# Step 2 — Retrieve results
GET <Operation-Location>
Ocp-Apim-Subscription-Key: <FOUNDRY_KEY>

What Extraction Can Feed

flowchart LR
    EX["📊 Extracted data\n(entities, tables, images)"] --> AS["🔍 Azure AI Search\n(indexing)"]
    EX --> PBI["📊 Power BI\n(analytics)"]
    EX --> VIS["👁️ Azure AI Vision\n(deep image analysis)"]
    EX --> GENAI["🤖 GenAI Models\n(advanced processing)"]

📌 Content Understanding ↔ Document Intelligence relationship: According to Microsoft documentation, Content Understanding is an evolution of Document Intelligence. Eventually, Content Understanding may replace Document Intelligence.

3.4 Media Processing — Video and Audio

Supported Media Types

┌──────────────────────────────────────────────────────────────┐
│          Azure AI Content Understanding — Mixed Media        │
│                                                              │
│   📄 Documents    🖼️ Images    🎵 Audio    🎬 Video           │
│                                                              │
│   ← Module 2 ─────────────────────────────── Module 3 →    │
└──────────────────────────────────────────────────────────────┘

Audio and Video Analyzers

Mixed-Media Pipeline

flowchart LR
    MEDIA["🎬 Media files\n(docs, images, audio, video)"] --> ANALYZER["⚙️ Content Understanding\nAnalyzer"]
    ANALYZER --> JSON["📋 JSON\n• analyzerID\n• Transcription\n• Summary\n• Extracted fields"]
    JSON --> PARSE["🔄 Parse & Store"]
    PARSE --> SEARCH["🔍 Azure AI Search\n(make searchable)"]
    PARSE --> DB["🗄️ Database"]

REST Call — Audio Analyzer

# Step 1 — Submit audio file
POST https://fabrikam-foundry.services.ai.azure.com/contentunderstanding/analyzers/prebuilt-audioSearch:analyze?api-version=2025-11-01
Ocp-Apim-Subscription-Key: <FOUNDRY_KEY>
Content-Type: application/json

{
  "inputs": [
    {
      "url": "https://<storage>.blob.core.windows.net/<container>/audio-clip.mp3?<SAS_TOKEN>"
    }
  ]
}

# Response: 202 Accepted

# Step 2 — Retrieve results
GET <Operation-Location>
Ocp-Apim-Subscription-Key: <FOUNDRY_KEY>

Audio response structure:

{
  "status": "succeeded",
  "result": {
    "analyzerId": "prebuilt-audioSearch",
    "contents": [
      {
        "markdown": "...",
        "fields": {
          "summary": {
            "valueString": "The conversation is a brief explanation of an expense report by GloboTicket."
          },
          "transcript": {
            "valueString": "Hello, in this video, we will discuss the expense report..."
          }
        }
      }
    ]
  }
}

💡 Use case: If your company has a library of unstructured videos, Content Understanding can transcribe and summarize them, then AI Search can make them searchable.

Required GenAI Models

For audio and video analyzers, the model deployed in Microsoft Foundry must support these media types. The gpt-4.1 model supports both audio and video.

⚠️ Important cost note: Each processing operation in Azure Content Understanding sends input tokens to a GenAI model. These tokens are not free. Consult the Azure Content Understanding pricing page before going to production.

References and Resources

Module 1 — Azure AI Search

Module 2 — Document Intelligence

Module 3 — Content Understanding

Document generated from the AI-102 course — Implement Knowledge Mining and Information Extraction Solutions

Search Terms

ai-102 · knowledge · mining · information · extraction · azure · ai · services · artificial · intelligence · generative · search · custom · model · models · ocr · pipeline · call · content · rest · types · analyzer · creation · document