AZ-400: Processes and Communications

Course: Azure DevOps Engineer Processes and Communications

Module 1 – DevOps Workflow

DevOps Definition

• Collaboration between developers and operations to deliver software quickly, reliably, and with high quality.
• Infinite loop: Plan → Code → Build → Test → Release → Deploy → Operate → Monitor → (Plan…)

Source Control Basics

GitHub Flow

1. Create a branch from main
2. Make changes and commit
3. Push the branch
4. Create a Pull Request (PR)
5. Code review + discussions
6. Merge into main
7. Deploy from main

Continuous Feedback

GitHub Issues

• One issue = one problem or improvement at a time.
• Key components: title, description, screenshots, task checklists, assignees, labels, milestones.
• Labels: bug, enhancement, documentation, help wanted, question.

GitHub Notifications

• Configure at the repository level: All Activity / Participating and @mentions / Custom.
• Avoid notification fatigue: choose only relevant events.

Module 2 – Traceability

Traceability Concept

• Forward traceability: from business requirements to code → “Which feature implements this user story?”
• Backward traceability: from code to requirements → “Why was this code written?”

GitHub Projects

• Kanban view (Board) and List + timeline view (Roadmap).
• Link Issues and PRs to a project.
• Automations: move items between columns automatically.

Azure Boards

• Work Items: trackable unit of work.
• Backlogs: prioritized list of work items.
• Sprints: iterations with planned capacity.
• Dashboards: customizable visualization.

Agile vs Scrum Process in Azure Boards

Work Item Hierarchy

Epics (high-level business objectives)
  └── Features (deliverable capabilities)
        └── User Stories (user value)
              └── Tasks (technical subtasks)
              └── Bugs (defects to fix)

Daily Standup – 3 Questions

1. What did I do yesterday?
2. What will I do today?
3. Are there any blockers?

GitHub + Azure Boards: Integration

Syntax for Linking Commits/PRs to Work Items

git commit -m "Implement login feature AB#1234"
# AB# = Azure Boards prefix

Automatic Work Item Closure

Fixes AB#1234    # Closes the item when the PR is merged
Fixed AB#1234
Fix AB#1234

Delivery Plans

• Visualize work items on an iteration calendar.
• Multi-team and multi-project view.
• Display dependencies between work items.

Bug Traceability

Quality Traceability

• Documented test strategy.
• Azure Test Plans: traceable manual and automated tests.
• Tests linked to Work Items.

Module 3 – DevOps Dashboards

Custom Dashboards in Azure DevOps

• Pre-built widgets: Burndown, Burnup, CFD, Velocity, etc.
• Charts based on Work Item queries.
• Pipeline data: Build History, Release Pipeline Overview.

Azure DevOps Analytics Service

Integrated reporting service in Azure DevOps with specialized widgets.

Burndown Chart

Y-axis: Remaining work (points or hours)
X-axis: Time (sprint days)
Ideal curve = straight descending line
Actual curve = can rise if new items are added

CFD (Cumulative Flow Diagram)

• Bands = work item states (Backlog, In Progress, Testing, Done).
• Narrowing of a band = bottleneck in that state.

Module 4 – Development and Test Metrics

Code Metric Widgets

• Code Tile: code quality metrics.
• Pull Request widget: pending, approved PRs.
• Build History: history of build results.

Pipeline Reports

Test Results Trend

• Widget showing: passed / failed / total tests over time.
• Usable for build pipelines or release pipelines.
• Advanced version: real-time data.

Performance Testing

1. Establish performance baselines first.
2. Then perform load tests to measure under load.
3. Azure Load Testing: managed service to generate load from Azure.

Azure Monitor Insights

• VM Insights: CPU, memory, disk performance of VMs.
• Storage Insights: storage metrics.
• Workbooks: custom and interactive reports.

Module 5 – Security, Delivery, and Operations Metrics

Security Feedback Loop

Microsoft Defender for Cloud DevOps Security

• Central console for Azure DevOps, GitHub, GitLab.
• Features:
  • Code findings (vulnerabilities in code).
  • Secret scans (exposed credentials).
  • OSS dependency scanning.
  • IaC misconfigurations.
  • Cloud Security Explorer: query engine with pre-built templates.

Release Management Metrics

Application Insights Usage Analytics

Operations – KPIs

High-Level (visible to the business)

• Response time.
• Concurrent users.
• Network traffic.
• Transaction rates.
• Latency.

Low-Level (infrastructure)

• CPU utilization.
• Memory usage.
• Disk I/O.

SLO (Service Level Objectives)

SLO = SLI + Target + Timespan
Example: "99.9% of HTTP requests (SLI) must respond in < 200ms (target) over the last 30 days (timespan)"

• SLI (Service Level Indicator): actual measurement (e.g., % of successful requests).
• SLO: target objective for the SLI.
• SLA: contractual commitment with penalties.

Module 6 – Collaboration and Communication

Azure DevOps Wikis

Essential Markdown Syntax

# Heading 1
## Heading 2

**bold**   *italic*

- Unordered list
1. Ordered list

[Link text](https://url.com)

![Alt text](image.png)

`inline code`

```code block```

| Column 1 | Column 2 |
|---|---|
| Value | Value |

Mermaid in Azure DevOps Wikis

::: mermaid
graph TD
    A[Start] --> B{Decision}
    B -- Yes --> C[Do Something]
    B -- No --> D[Do Something Else]
:::

Supported Mermaid Diagram Types

• graph TD: top-down flowchart.
• sequenceDiagram: sequence diagram.
• gantt: Gantt chart.
• classDiagram: class diagram.

Release Notes and Semantic Versioning

• Format: MAJOR.MINOR.PATCH (e.g., 2.3.1).
  • MAJOR: breaking changes → consumers must update.
  • MINOR: new features, backward compatible.
  • PATCH: bug fixes.
• Release notes document changes between versions.

API Documentation

Essential elements of good API documentation:

• Endpoints (URLs).
• HTTP methods (GET, POST, PUT, DELETE).
• Required authentication.
• Request and response examples.
• Error codes and messages.
• API versioning.
• Examples by programming language.

Git History

git log                           # Full log
git log --oneline                 # Condensed log (one line per commit)
git log --pretty=format:"%h %an %s"  # Custom format
git log v1.0..v2.0               # Commits between two tags
git log --follow -- file.txt      # History of a specific file

Service Hooks vs Webhooks

• Webhooks: HTTP notifications to an external endpoint on each event.
• Service Hooks: broader concept including webhooks + native integrations (Teams, Slack, ServiceNow, etc.).

Microsoft Teams Integration

Available Apps

Teams Commands (Azure Boards)

@azure boards signin
@azure boards link https://dev.azure.com/org/project
@azure boards subscriptions
@azure boards addAreapath /MyTeamPath

Slack Integration

• Install apps: Azure Boards, Azure Pipelines, Azure Repos in Slack.
• Commands:

  /az boards signin
  /az repos signin
  /az boards link <project-url>
  /az boards subscribe --event workitem.created
  

Key Points for the Exam

• GitHub Flow: branch → commit → PR → review → merge → deploy.
• AB#1234 syntax: link commits/PRs to Azure Boards Work Items.
• Burndown = remaining work vs. time. Burnup = completed work.
• CFD = identify bottlenecks by state.
• Velocity = sprint throughput (estimated vs. actual).
• Lead Time = creation to delivery. Cycle Time = start of work to delivery.
• SLO = SLI + Target + Timespan.
• Provisioned Wiki = created in DevOps portal. Code Wiki = Markdown files in a Git repo.
• Semantic Versioning: MAJOR = breaking, MINOR = new features, PATCH = bug fixes.
• Forward traceability = requirements → code. Backward = code → requirements.
• Mermaid: ::: mermaid ... ::: for diagrams in Azure DevOps wikis.

Module 7 – DevOps Culture and Organizational Transformation

The Three Ways of DevOps

flowchart LR
    DEV[Dev Team] -->|"1st Way\nFlow"| OPS[Ops Team]
    OPS -->|"2nd Way\nFeedback"| DEV
    DEV -->|"3rd Way\nExperimentation"| LEARN[Continuous learning]
    LEARN --> DEV

Conway’s Law and Architecture

“Organizations which design systems… produce designs which are copies of the communication structures of these organizations.” — Melvin Conway

flowchart TD
    CONWAY["Organizational structure\nDetermines software architecture"]
    SILOS[Siloed teams\nDev / QA / Ops / Security] --> MONO[Monolithic\ncoupled architecture]
    CROSS[Cross-functional teams\naligned on products] --> MICRO[Microservices\ndecoupled architecture]
    CONWAY --> SILOS
    CONWAY --> CROSS

DORA Metrics – The 4 Key DevOps Performance Indicators

flowchart LR
    subgraph Speed
        DF[Deployment Frequency]
        LT[Lead Time for Changes]
    end
    subgraph Stability
        MTTR[Mean Time to Restore]
        CFR[Change Failure Rate]
    end
    DF --> ELITE[Elite DevOps Team]
    LT --> ELITE
    MTTR --> ELITE
    CFR --> ELITE

Lean and Value Stream Mapping

flowchart LR
    subgraph "Value Stream Map"
        REQ[Requirement\nreceived] -->|"Wait: 2 days"| DEV_START[Dev starts]
        DEV_START -->|"Work: 3 days"| CODE_DONE[Code complete]
        CODE_DONE -->|"Wait: 1 day"| QA_START[QA starts]
        QA_START -->|"Work: 2 days"| QA_DONE[QA validated]
        QA_DONE -->|"Wait: 3 days"| DEPLOY[Deployed]
    end
    TOTAL["Total Lead Time: 11 days\nValue-add time: 5 days (45%)\nWaste: 6 days (55%)"]

Goal: Identify and eliminate waste:

• Waiting: waits between steps.
• Defects: bugs, rework.
• Overprocessing: unnecessary procedures.
• Overproduction: unrequested features.

Module 8 – Advanced Azure Boards

Creating and Managing a Scrum Project

sequenceDiagram
    participant PO as Product Owner
    participant SM as Scrum Master
    participant TEAM as Dev Team
    participant ADO as Azure Boards

    PO->>ADO: Create User Stories in the Backlog
    PO->>ADO: Prioritize the Product Backlog
    
    Note over SM,TEAM: Sprint Planning (sprint start)
    SM->>ADO: Create a Sprint (2 weeks)
    PO->>TEAM: Present priority Stories
    TEAM->>ADO: Break down into Tasks
    TEAM->>ADO: Estimate in Story Points
    TEAM->>ADO: Move Stories into the Sprint
    
    loop Every day
        TEAM->>TEAM: Daily Standup (15 min)
        TEAM->>ADO: Update Task status
    end
    
    Note over PO,TEAM: Sprint Review (sprint end)
    TEAM->>PO: Demo of delivered features
    PO->>ADO: Close completed Stories
    
    Note over SM,TEAM: Retrospective
    TEAM->>TEAM: Continuous improvement

Work Item Types and Transitions

Scrum states: New → Approved → Committed → Done → Removed
Agile states: New → Active → Resolved → Closed → Removed

Azure Boards – Links Between Work Items

# Available link types
Child / Parent          # Direct hierarchy
Related                 # Weak link
Predecessor / Successor # Sequential dependencies
Duplicate / Duplicate Of # Duplicates
Affects / Affected By   # Software impact
Consumes From / Produces For # Cross-org dependencies
Remote Related          # Cross-org link

Azure Boards Queries

# Query: All unassigned critical bugs
[Work Item Type] = Bug 
AND [Severity] = 1 - Critical
AND [Assigned To] = <Empty>
AND [State] <> Done
ORDER BY [Created Date] Descending

# Query: Current sprint stories for my team
[Work Item Type] = User Story
AND [Iteration Path] = @CurrentIteration('[MyProject]\MyTeam')
AND [Area Path] = [MyProject]\MyTeam
AND [State] <> Removed

Module 9 – Advanced Metrics and Performance

Application Insights – Performance Analytics

flowchart TD
    APP[.NET Application] -->|"Application Insights SDK"| AI[Application Insights]

    subgraph AI
        REQ[Requests\nHTTP requests]
        DEP[Dependencies\nDatabase, external APIs]
        EXC[Exceptions\nUnhandled exceptions]
        PERF[Performance\nP50/P95/P99]
        USER[Users/Sessions\nUser behavior]
        FUNNEL[Funnels\nConversion paths]
    end

    AI --> KQL[Log Analytics\nKQL Queries]
    AI --> ALERT[Smart alerts\nSmart Detection]
    AI --> DASHBOARD[Dashboards]

Essential Application Insights Metrics

// Analyze slow requests P95
requests
| where timestamp > ago(24h)
| summarize P95=percentile(duration, 95), P99=percentile(duration, 99), 
            avgDuration=avg(duration), count=count()
  by name
| where P95 > 1000   // > 1 second
| order by P95 desc
| take 10

// Error rates per hour
requests
| where timestamp > ago(7d)
| summarize 
    successCount = countif(success == true),
    failedCount = countif(success == false),
    total = count()
  by bin(timestamp, 1h)
| extend errorRate = round(failedCount * 100.0 / total, 2)
| order by timestamp desc

// Most frequent exceptions
exceptions
| where timestamp > ago(24h)
| summarize Count = count(), 
            LastOccurrence = max(timestamp)
  by type, method
| order by Count desc
| take 20

// Feature usage (custom events)
customEvents
| where timestamp > ago(30d)
| where name startswith "Feature_"
| summarize UniqueUsers = dcount(user_Id), 
            TotalUsages = count()
  by name, bin(timestamp, 1d)
| order by timestamp desc

Azure Monitor – Alert Configuration

# Create an alert on the HTTP error rate
az monitor metrics alert create \
  --name "HighErrorRate" \
  --resource-group "monitoring-rg" \
  --scopes "/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/components/myapp" \
  --condition "avg requests/failed > 10" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 2 \
  --description "High HTTP error rate" \
  --action-groups "/subscriptions/{sub}/resourceGroups/{rg}/providers/microsoft.insights/actionGroups/ops-team"

# Create an availability alert (URL Ping Test)
az monitor app-insights web-test create \
  --name "homepage-availability-test" \
  --resource-group "monitoring-rg" \
  --app-insights "myapp" \
  --url "https://myapp.azurewebsites.net/health" \
  --frequency 300 \   # Every 5 minutes
  --timeout 30 \
  --locations "East US" "West Europe" "Southeast Asia"

SRE – Error Budgets and Risk Management

flowchart LR
    subgraph SLO Definition
        SLI["SLI = % of requests < 500ms"] 
        TARGET["Target = 99.9%"]
        WINDOW["Window = 30 days"]
    end

    SLI & TARGET & WINDOW --> SLO["SLO: 99.9% of requests\nrespond < 500ms\nover 30 days"]
    
    SLO --> BUDGET["Error Budget = 0.1%\n= 43.8 minutes/month of allowed downtime"]
    
    BUDGET -->|"Budget consumed < 50%"| RELEASE["🚀 Accelerated deployment possible"]
    BUDGET -->|"Budget consumed > 90%"| FREEZE["❄️ Feature freeze - stabilization"]
    BUDGET -->|"Budget exhausted"| INCIDENT["🚨 Post-mortem + mandatory remediation"]

Post-Incident Reviews (Post-mortems)

Core principle: Blameless. The objective is learning, not punishment.

Structure of a good post-mortem:

# Post-Mortem: [Incident Title]

**Incident date:** 2024-03-15  
**Severity:** P1 — Critical customer impact  
**Affected services:** Checkout API  

## Summary
[2-3 sentences describing what happened]

## Impact
- Affected users: ~15,000
- Lost transactions: ~200  
- Revenue impact: ~$8,000

## Timeline
| Time | Event |
|---|---|
| 14:32 | First alert triggered on error rate |
| 14:35 | On-call engineer notified |
| 14:42 | Deployment identified as probable cause |
| 15:00 | Rollback initiated |
| 15:17 | Service restored, all metrics normal |

## Root Cause Analysis (5 Whys)
- **Symptom**: Checkout API returning 500 errors
- **Why 1**: Timeout on database connection
- **Why 2**: Connection pool exhausted
- **Why 3**: Blocking schema migration deployed to production
- **Why 4**: Validation gate does not cover long migrations
- **Why 5**: No migration duration test in the pipeline

## Corrective Actions
| Action | Owner | Deadline | Work Item |
|---|---|---|---|
| Add migration duration test to pipeline | @john | 2024-03-22 | #2456 |
| Implement blue/green for migrations | @sarah | 2024-04-01 | #2457 |
| Revise connection pool alert thresholds | @ops | 2024-03-19 | #2458 |

## What Went Well
- Alert triggered quickly
- Automated rollback functional

## What Can Be Improved
- Rollback runbook documentation incomplete
- Customer communication delayed

Module 10 – Wikis, Documentation, and Knowledge Management

Azure DevOps Wiki – Recommended Structure

/ (Home)
├── Onboarding/
│   ├── Getting Started
│   ├── Development Environment Setup
│   └── Team Processes
├── Architecture/
│   ├── System Overview
│   ├── Database Schema
│   └── API Documentation
├── Runbooks/
│   ├── Deployment Procedures
│   ├── Incident Response
│   └── Database Maintenance
└── Retrospectives/
    ├── Sprint 42 Retro
    └── Sprint 43 Retro

Advanced Markdown for Wikis

## Warnings and Notes

> ⚠️ **Important**: Always back up before modifying.

> ℹ️ **Note**: This procedure applies only to the production environment.

> ✅ **Best practice**: Test in the staging environment first.

## Advanced Tables

| Service | SLA | Cost/month | Region |
|:--------|----:|----------:|:-------|
| App Service P1 | 99.95% | $150 | East US |
| Azure SQL S2 | 99.99% | $80 | East US |
| Azure Redis C1 | 99.9% | $55 | East US |

## Code with Syntax

```csharp
public async Task<IActionResult> GetOrders(
    [FromQuery] OrderFilter filter,
    CancellationToken cancellationToken)
{
    var orders = await _orderService.GetAsync(filter, cancellationToken);
    return Ok(orders);
}

Advanced Mermaid Diagrams

::: mermaid sequenceDiagram autonumber participant C as Client participant API as API Gateway participant AUTH as Auth Service participant DB as Database

C->>API: POST /api/orders (Bearer token)
API->>AUTH: Validate token
AUTH-->>API: Token valid (userId: 42)
API->>DB: INSERT order (userId: 42)
DB-->>API: Order created (id: 1234)
API-->>C: 201 Created {orderId: 1234}

:::

### API Documentation with Swagger/OpenAPI

```yaml
# openapi.yaml
openapi: 3.0.3
info:
  title: Orders API
  version: 1.0.0
  description: |
    Order management API.
    ## Authentication
    Uses Azure Entra ID (OAuth 2.0).
    ## Versioning
    The API is versioned. Current version is v1.

servers:
  - url: https://api.contoso.com/v1
    description: Production
  - url: https://api-staging.contoso.com/v1
    description: Staging

security:
  - BearerAuth: []

paths:
  /orders:
    get:
      summary: List orders
      operationId: listOrders
      parameters:
        - name: status
          in: query
          schema:
            type: string
            enum: [pending, processing, completed, cancelled]
        - name: limit
          in: query
          schema:
            type: integer
            minimum: 1
            maximum: 100
            default: 20
      responses:
        '200':
          description: List of orders
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/OrderList'
        '401':
          $ref: '#/components/responses/Unauthorized'
    
    post:
      summary: Create an order
      operationId: createOrder
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateOrderRequest'
            example:
              customerId: "cust-123"
              items:
                - productId: "prod-456"
                  quantity: 2
      responses:
        '201':
          description: Order created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Order'

components:
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT
  
  schemas:
    Order:
      type: object
      properties:
        id:
          type: string
          format: uuid
        status:
          type: string
          enum: [pending, processing, completed, cancelled]
        totalAmount:
          type: number
          format: decimal
        createdAt:
          type: string
          format: date-time

Automated Changelogs

# Automatically generate a changelog from Git commits
# Using Conventional Commits

# Conventional commit format
feat: new feature
fix: bug fix
docs: documentation only
style: formatting, no logic change
refactor: refactoring without new feature
test: add/modify tests
chore: maintenance, tooling
BREAKING CHANGE: incompatible change

# Examples
git commit -m "feat(auth): add Google OAuth login"
git commit -m "fix(orders): fix VAT calculation #1234"
git commit -m "feat!: change API from v1 to v2 (BREAKING CHANGE)"

# Generate CHANGELOG.md with conventional-changelog
npx conventional-changelog-cli -p angular -i CHANGELOG.md -s

# Result in CHANGELOG.md:
# ## [2.0.0] - 2024-03-15
# ### BREAKING CHANGES
# * change API from v1 to v2
# ### Features
# * **auth:** add Google OAuth login
# ### Bug Fixes
# * **orders:** fix VAT calculation ([#1234](link))

Module 11 – Service Hooks and Advanced Integrations

Service Hooks Architecture

flowchart LR
    subgraph "Azure DevOps Events"
        BUILD_COMPLETE[Build Completed]
        RELEASE_DEPLOY[Release Deployed]
        PR_CREATED[PR Created]
        WORK_ITEM[Work Item Changed]
    end

    subgraph "Service Hook Consumers"
        TEAMS[Microsoft Teams\nChat + Notifications]
        SLACK[Slack\nChat + Notifications]
        SERVICENOW[ServiceNow\nIT Service Management]
        JIRA[Jira\nWork Tracking]
        CUSTOM[Custom Webhook\nYour HTTPS endpoint]
    end

    BUILD_COMPLETE -->|service hook| TEAMS
    BUILD_COMPLETE -->|service hook| SLACK
    RELEASE_DEPLOY -->|service hook| SERVICENOW
    PR_CREATED -->|service hook| TEAMS
    WORK_ITEM -->|service hook| JIRA
    RELEASE_DEPLOY -->|webhook| CUSTOM

Configure an Azure DevOps Webhook

# Create a service hook webhook via Azure DevOps REST API
$pat = "your-pat-token"
$org = "MyOrganization"
$projectId = "your-project-id"
$encodedPat = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$pat"))

$body = @{
    publisherId = "tfs"
    eventType = "build.complete"
    resourceVersion = "1.0"
    consumerId = "webHooks"
    consumerActionId = "httpRequest"
    publisherInputs = @{
        buildStatus = "Failed"
        projectId = $projectId
    }
    consumerInputs = @{
        url = "https://my-endpoint.azurewebsites.net/api/webhook"
        httpHeaders = "X-Custom-Header: myvalue"
        messagesToSend = "payload"
    }
} | ConvertTo-Json -Depth 10

$response = Invoke-RestMethod `
    -Uri "https://dev.azure.com/$org/_apis/hooks/subscriptions?api-version=7.0" `
    -Method Post `
    -Headers @{ Authorization = "Basic $encodedPat"; "Content-Type" = "application/json" } `
    -Body $body

Write-Host "Service Hook created: $($response.id)"

Logic Apps Integration for Advanced Notifications

// Logic App: Send a Teams notification when a production deployment fails
{
  "definition": {
    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#",
    "triggers": {
      "manual": {
        "type": "Request",
        "kind": "Http",
        "inputs": {
          "schema": {
            "properties": {
              "buildId": { "type": "string" },
              "buildStatus": { "type": "string" },
              "buildUrl": { "type": "string" }
            }
          }
        }
      }
    },
    "actions": {
      "Condition_Check_Failed": {
        "type": "If",
        "expression": {
          "equals": ["@triggerBody()?['buildStatus']", "failed"]
        },
        "actions": {
          "Post_Teams_Message": {
            "type": "ApiConnection",
            "inputs": {
              "host": { "connection": { "name": "@parameters('$connections')['teams']" } },
              "method": "post",
              "path": "/v3/beta/teams/@{encodeURIComponent('team-id')}/channels/@{encodeURIComponent('channel-id')}/messages",
              "body": {
                "body": {
                  "contentType": "html",
                  "content": "🚨 <b>Deployment FAILED</b><br/>Build ID: @{triggerBody()?['buildId']}<br/><a href='@{triggerBody()?['buildUrl']}'>View details</a>"
                }
              }
            }
          }
        }
      }
    }
  }
}

Module 12 – Capacity Planning and Advanced SRE

Capacity Planning with Azure SQL

flowchart TD
    WORKLOAD[Define expected workload\ne.g.: 1000 req/min peak] --> SLO[Define SLOs\ne.g.: P99 < 500ms]
    SLO --> BASELINE[Measure current baselines]
    BASELINE --> LOAD_TEST[Load tests\nAzure Load Testing]
    LOAD_TEST --> ANALYSIS{Analyze results}
    ANALYSIS -->|"SLOs met"| OK[✅ Sufficient capacity]
    ANALYSIS -->|"SLOs not met"| SCALE{Scaling strategy}
    SCALE -->|"Predictable peak"| SCALE_OUT[Scale Out\n+ instances]
    SCALE -->|"Consistent performance"| SCALE_UP[Scale Up\nmore powerful]
    SCALE -->|"Cost optimization"| AUTO[Azure Autoscale\nmetric-based rules]

Azure Monitor – Operational Dashboard

flowchart TD
    subgraph "Metrics Sources"
        VM[VMs\nCPU/Memory/Disk]
        SQL[Azure SQL\nDTU/Connections/Blocking]
        APP[App Service\nRequests/Errors/Response time]
        FUNC[Functions\nExecutions/Errors/Duration]
    end

    subgraph "Azure Monitor"
        METRICS[Metrics Explorer]
        LOGS[Log Analytics\nKQL Queries]
        ALERTS[Alert Rules\nThresholds + actions]
    end

    subgraph "Visualization"
        DASHBOARD[Azure Dashboard\nReal-time view]
        WORKBOOK[Azure Workbook\nInteractive reports]
        TEAMS_NOTIF[Teams / Slack\nAlert notifications]
    end

    VM & SQL & APP & FUNC --> METRICS & LOGS
    METRICS & LOGS --> ALERTS
    METRICS --> DASHBOARD
    LOGS --> WORKBOOK
    ALERTS --> TEAMS_NOTIF

Review Questions – Processes and Communications

Q1 – GitHub Flow

Question: A developer needs to fix a critical bug in the production application. What is the correct first step according to GitHub Flow?

• A. Fix directly on the main branch
• B. Create a new branch from main ✅
• C. Clone a new repository
• D. Create a branch from the last release

Explanation: GitHub Flow always dictates creating a dedicated branch from main for each change, even for a hotfix. This isolates the change, lets you test it, review it via PR, and merge it cleanly. Fixing directly on main is risky and bypasses the review process.

Q2 – Azure Boards Traceability

Question: A developer wants to automatically link their GitHub commit to Azure Boards Work Item number 1567, and close this Work Item when the PR is merged. What syntax do they use in their commit message?

• A. [AB#1567]
• B. #1567
• C. Fixes AB#1567 ✅
• D. workitem:1567

Explanation: The syntax AB#1567 links the commit to the Work Item without closing it. Adding the keyword Fixes (or Fix, Fixed, Close, Closes, Closed, Resolve, Resolves, Resolved) before AB#1567 automatically closes the Work Item when the PR is merged into the default branch.

Q3 – Azure DevOps Analytics Metrics

Question: Your team notices that many User Stories remain blocked in the “In Code Review” state for several days before moving to “Testing”. Which Azure DevOps report do you use to visualize this bottleneck?

• A. Velocity Report
• B. Burndown Chart
• C. Cumulative Flow Diagram (CFD) ✅
• D. Sprint Burnup

Explanation: The CFD (Cumulative Flow Diagram) shows the width of each band corresponding to a workflow state over time. If the “In Code Review” band widens abnormally, it indicates items are accumulating in that state — a bottleneck. The Burndown shows total remaining work, not distribution by state.

Q4 – SLO, SLI, SLA

Question: Your company signed an SLA guaranteeing 99.9% monthly availability. The SRE team defined an internal SLO of 99.95%. In a given month, the system was available 99.92% of the time. What is the situation?

• A. The SLA is violated, the SLO is met
• B. The SLA is met, the SLO is violated, no customer impact
• C. The SLA is met (99.92% > 99.9%), the SLO is violated (99.92% < 99.95%) ✅
• D. Neither the SLA nor the SLO is violated

Explanation: 99.92% exceeds the contractual SLA of 99.9% → no penalty. But 99.92% is below the internal SLO of 99.95%, meaning the team has consumed part of its Error Budget. The SLO is intentionally stricter than the SLA to have a safety margin before impacting customers.

Q5 – Azure DevOps Wiki Types

Question: Your team wants technical documentation to be versioned with the code, reviewed via PR, and changes traceable in Git history. Which Wiki type do you choose?

• A. Provisioned Wiki (created in the Azure DevOps UI)
• B. Code Wiki (Markdown files in an existing Git repository) ✅
• C. SharePoint Wiki integrated with Azure DevOps
• D. GitHub Wiki linked to Azure DevOps

Explanation: A Code Wiki consists of Markdown files stored directly in an Azure Repos Git repository. They can therefore be versioned with the code, reviewed via the same PR/code review processes, and their history is complete in Git. A Provisioned Wiki is stored in a separate repository managed by Azure DevOps and modifications don’t go through the standard PR process.

Q6 – Service Hooks vs Webhooks

Question: You want your ITSM tool (ServiceNow) to be automatically notified when a production deployment completes in Azure Pipelines. Which Azure DevOps feature do you configure?

• A. Azure Monitor Alerts
• B. Pipeline condition always()
• C. Service Hooks with a ServiceNow subscriber ✅
• D. Azure Event Grid

Explanation: Azure DevOps Service Hooks allow subscribing to events (such as “Release Deployment Completed”) and sending notifications to external consumers like ServiceNow. Azure DevOps has native integrations with ServiceNow for automatic creation of change requests during deployments.

Q7 – Semantic Versioning

Question: A Python library is at version 3.5.2. You fix a calculation bug that was returning incorrect results in certain edge cases. The fix does not modify any public API. What version do you publish?

• A. 4.0.0
• B. 3.6.0
• C. 3.5.3 ✅
• D. 3.5.2-hotfix

Explanation: In Semantic Versioning, a backward-compatible bug fix = increment the PATCH (third number). MAJOR for breaking changes, MINOR for backward-compatible new features, PATCH for backward-compatible bug fixes. 3.5.2 → 3.5.3.

Q8 – Post-Mortem Culture

Question: After a production incident, your manager wants to identify “who” caused the incident to take disciplinary action. According to SRE/DevOps best practices, what approach do you recommend?

• A. Conduct a thorough investigation to identify the responsible person
• B. Ask each person involved to write an individual report
• C. Conduct a blameless post-mortem to identify systemic root causes and improve processes ✅
• D. Document the incident and move on

Explanation: Blameless post-mortem culture is fundamental in SRE/DevOps. Punishing individuals discourages incident reporting, hides systemic problems, and destroys trust. The goal is to identify systemic root causes (processes, tools, training) and continuously improve the system, not punish people who acted in good faith with available information.

Glossary – Processes and Communications

Appendix – Reference Scripts and Commands

Azure CLI – Azure DevOps

# Default configuration
az devops configure --defaults organization=https://dev.azure.com/MyOrg project=MyProject

# Create a work item
az boards work-item create \
  --title "Implement the login page" \
  --type "User Story" \
  --description "As a user, I want to log in with my email and password" \
  --area "MyProject\Frontend" \
  --iteration "MyProject\Sprint 5" \
  --assigned-to "john@company.com" \
  --fields "Microsoft.VSTS.Common.Priority=1" "Microsoft.VSTS.Common.Severity=2 - High"

# List work items for a sprint
az boards query \
  --wiql "SELECT [System.Id], [System.Title], [System.State], [System.AssignedTo] 
          FROM WorkItems 
          WHERE [System.TeamProject] = 'MyProject' 
          AND [System.IterationPath] = 'MyProject\Sprint 5'
          AND [System.WorkItemType] = 'User Story'
          ORDER BY [System.ChangedDate] DESC"

# Create a tag on a commit (release)
az repos tag create \
  --name "v2.3.1" \
  --object-id "abc123def456" \
  --repository "my-repo"

# Generate a burndown report (analytics)
az boards burndown \
  --team "Frontend Team" \
  --iteration "Sprint 5"

GitHub CLI – Issue and PR Management

# Install GitHub CLI
brew install gh    # macOS
winget install GitHub.cli    # Windows

# Authentication
gh auth login

# Create an issue
gh issue create \
  --title "Bug: Login fails on Safari mobile" \
  --body "Steps to reproduce: 1. Open app on Safari iOS 2. Click Login 3. Error appears" \
  --label "bug,mobile,high-priority" \
  --assignee "@john_smith" \
  --milestone "v2.4.0"

# List open issues
gh issue list \
  --state open \
  --label "bug" \
  --assignee "@me"

# Create a Pull Request
gh pr create \
  --title "feat: Add Google OAuth login" \
  --body "## Summary
  Implements Google OAuth 2.0 for user authentication.
  
  ## Testing
  - Unit tests added (coverage: 85%)
  - Manual tested on Chrome, Firefox, Safari
  
  Fixes #1234" \
  --reviewer "@sarah,@michael" \
  --draft

# Approve a PR
gh pr review 456 --approve --body "LGTM! Tests pass and code is clean."

# Merge a PR with squash
gh pr merge 456 --squash --delete-branch

# View PR check status
gh pr checks 456

PowerShell – Azure DevOps REST API

# Utility functions for the Azure DevOps API
function Get-ADOHeaders {
    param([string]$Pat)
    $encoded = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$Pat"))
    return @{
        "Authorization" = "Basic $encoded"
        "Content-Type" = "application/json"
    }
}

$org = "MyOrganization"
$project = "MyProject"
$pat = $env:AZURE_DEVOPS_PAT
$headers = Get-ADOHeaders -Pat $pat
$baseUrl = "https://dev.azure.com/$org/$project/_apis"

# Get active sprints
$sprints = Invoke-RestMethod `
    -Uri "$baseUrl/work/teamsettings/iterations?`$timeframe=current&api-version=7.0" `
    -Headers $headers

Write-Host "Current sprint: $($sprints.value[0].name)"
Write-Host "Dates: $($sprints.value[0].attributes.startDate) → $($sprints.value[0].attributes.finishDate)"

# Create a work item
$workItemBody = @(
    @{ op = "add"; path = "/fields/System.Title"; value = "Improve API performance" }
    @{ op = "add"; path = "/fields/System.IterationPath"; value = "MyProject\Sprint 5" }
    @{ op = "add"; path = "/fields/Microsoft.VSTS.Common.Priority"; value = 1 }
    @{ op = "add"; path = "/fields/Microsoft.VSTS.Scheduling.StoryPoints"; value = 5 }
) | ConvertTo-Json

$createdItem = Invoke-RestMethod `
    -Uri "$baseUrl/wit/workitems/`$User%20Story?api-version=7.0" `
    -Method Post `
    -Headers ($headers + @{ "Content-Type" = "application/json-patch+json" }) `
    -Body $workItemBody

Write-Host "Work Item created: #$($createdItem.id) - $($createdItem.fields.'System.Title')"

# Get build pipeline statistics
$pipeline = Invoke-RestMethod `
    -Uri "$baseUrl/pipelines/42/runs?`$top=30&api-version=7.0" `
    -Headers $headers

$runs = $pipeline.value
$successRate = ($runs | Where-Object { $_.result -eq "succeeded" }).Count / $runs.Count * 100
Write-Host "Success rate (last 30 runs): $([Math]::Round($successRate, 1))%"

Azure Monitor – Advanced KQL Queries for DevOps

// Pipeline health: success rate by pipeline over 7 days
AzureDiagnostics
| where TimeGenerated > ago(7d)
| where ResourceProvider == "MICROSOFT.DEVOPS"
| where OperationName has "pipeline"
| summarize 
    TotalRuns = count(),
    SuccessCount = countif(ResultType == "Success"),
    FailureCount = countif(ResultType == "Failed")
  by PipelineName_s
| extend SuccessRate = round(SuccessCount * 100.0 / TotalRuns, 1)
| order by SuccessRate asc

// Detect deployments causing errors
let deploymentTime = datatable(deployment:string, timestamp:datetime)
[
    "v2.3.1", datetime(2024-03-15 14:00),
    "v2.3.2", datetime(2024-03-16 10:30)
];
requests
| where timestamp > ago(48h)
| join kind=asof deploymentTime on $left.timestamp == $right.timestamp
| summarize ErrorRate = countif(success == false) * 100.0 / count()
  by deployment, bin(timestamp, 30m)
| order by timestamp

// SLO Dashboard: availability over the last 30 days
let sloTarget = 0.999;  // 99.9%
requests
| where timestamp > ago(30d)
| summarize 
    TotalRequests = count(),
    SuccessfulRequests = countif(resultCode !startswith "5")
  by bin(timestamp, 1d)
| extend AvailabilityDay = SuccessfulRequests * 1.0 / TotalRequests
| summarize 
    AvgAvailability = round(avg(AvailabilityDay) * 100, 3),
    MinAvailability = round(min(AvailabilityDay) * 100, 3),
    SLOMetTarget = countif(AvailabilityDay >= sloTarget),
    TotalDays = count()
| extend SLOComplianceRate = round(SLOMetTarget * 100.0 / TotalDays, 1)
| project AvgAvailability, MinAvailability, SLOComplianceRate,
          ErrorBudgetUsed = round((1 - AvgAvailability / 100) / (1 - sloTarget) * 100, 1)

Azure DevOps Dashboard Configuration via JSON

// Export/Import an Azure DevOps dashboard
{
  "name": "Production Health Dashboard",
  "description": "Operational metrics for the production environment",
  "refreshInterval": 300,
  "widgets": [
    {
      "name": "Build Health",
      "typeId": "Microsoft.VisualStudioServices.Widgets.BuildHistogram",
      "position": { "row": 1, "column": 1 },
      "size": { "rowSpan": 1, "columnSpan": 2 },
      "settings": {
        "pipelineDefinitionId": "42",
        "numberOfBuilds": 30
      }
    },
    {
      "name": "Sprint Burndown",
      "typeId": "Microsoft.VisualStudioServices.Widgets.Burndown",
      "position": { "row": 1, "column": 3 },
      "size": { "rowSpan": 2, "columnSpan": 3 },
      "settings": {
        "teamId": "my-team-id",
        "iterationId": "current"
      }
    },
    {
      "name": "Deployment Frequency",
      "typeId": "Microsoft.VisualStudioServices.Widgets.Chart",
      "position": { "row": 3, "column": 1 },
      "size": { "rowSpan": 2, "columnSpan": 4 },
      "settings": {
        "chartType": "trend",
        "pipelineId": "42",
        "metric": "deploymentFrequency"
      }
    }
  ]
}

Reference Tables for the Exam

Azure Boards – Process Comparison

DORA Metrics – Precise Definitions

GitHub vs Azure Boards Comparison

Search Terms

az-400 · processes · communications · azure · devops · iac · microsoft · github · traceability · boards · metrics · security · syntax · analytics · dashboards · delivery · feedback · flow · insights · item · operations · service · test