Understanding Azure Arc for Hybrid Environments

Course: Understanding Azure Arc for Hybrid Environments + Implementing Hybrid Management with Azure Arc Level: Intermediate / Advanced Last Updated: June 2026

Table of Contents

1. Introduction to Hybrid Cloud
2. Azure Arc Capabilities
3. Implementing Azure Arc – Servers
4. Implementing Azure Arc – Kubernetes
5. Implementing Azure Arc – SQL and Data Services
6. Governance and Security with Arc
7. Monitoring Arc Resources
8. GitOps with Arc Kubernetes
9. Troubleshooting and Advanced Architecture
10. Complete Code Examples
11. Comparative Tables
12. Glossary

1. Introduction to Hybrid Cloud

The Cloud Sprawl Challenge

Cloud adoption has brought unprecedented agility — but also a new problem: cloud sprawl. It is now as easy to deploy a new cloud service as it was difficult in the past to order and install a physical server. This ease leads to an uncontrolled proliferation of resources across multiple clouds and environments.

timeline
    title Evolution of IT Environments
    1990s : Physical servers
          : Weeks to deploy
          : "Pets" (unique, precious)
          : Over-provisioned resources
    2000s : Virtualization
          : Minutes to deploy
          : "Cattle" (interchangeable)
          : Better hardware utilization
          : VM explosion
    2010s : Public cloud
          : Seconds to deploy
          : Auto-scaling, PaaS, SaaS
          : Cloud Sprawl begins
          : Multi-cloud by default
    2020s : Hybrid and Multi-cloud
          : Maximum complexity
          : Need for unification
          : Azure Arc as the solution

Multi-Cloud and Hybrid Challenges

mindmap
  root((Hybrid Challenges))
    Inventory
      Unknown resources
      No unified view
      Multiple portals
    Governance
      Disparate policies
      Varied compliance
      Manual processes
    Security
      Different RBAC per platform
      Fragmented monitoring
      Undetected vulnerabilities
    Costs
      Multiple invoices
      Difficult optimization
      Orphaned resources
    Skills
      Specialized teams
      Organizational silos
      Multiple training needs

What Azure Offers Natively vs What Arc Adds

Summary: Azure Arc is not a migration service. It is an extension service that brings Azure management capabilities to resources that remain in other environments.

2. Azure Arc Capabilities

Azure Arc Global Architecture

graph TB
    subgraph AZURE["Azure Cloud"]
        ARM[Azure Resource\nManager API]
        POLICY[Azure Policy]
        RBAC[Azure RBAC\nEntra ID]
        MONITOR[Azure Monitor\nLog Analytics]
        DEFENDER[Microsoft Defender\nfor Cloud]
        PORTAL[Azure Portal]
    end
    
    subgraph ARC_PLANE["Azure Arc Control Plane"]
        HYBRID_RP[Microsoft.HybridCompute\nResource Provider]
        K8S_RP[Microsoft.Kubernetes\nResource Provider]
        DATA_RP[Microsoft.AzureArcData\nResource Provider]
    end
    
    subgraph ONPREM["On-Premises / Other Cloud"]
        SERVERS[Windows/Linux\nServers]
        K8S[Kubernetes\nClusters]
        SQL[SQL Server\nInstances]
        VMWARE[VMware\nvSphere VMs]
    end
    
    subgraph AGENTS["Arc Agents"]
        CMA["Connected Machine\nAgent (servers)"]
        HELM_AGENTS["Helm Agents\n(kubernetes)"]
        SQL_EXT[Azure Extension\nfor SQL Server]
        RESOURCE_BRIDGE[Azure Arc\nResource Bridge]
    end
    
    SERVERS --> CMA
    K8S --> HELM_AGENTS
    SQL --> SQL_EXT
    VMWARE --> RESOURCE_BRIDGE
    
    CMA -->|Outbound HTTPS| ARC_PLANE
    HELM_AGENTS -->|Outbound HTTPS| ARC_PLANE
    SQL_EXT -->|Outbound HTTPS| ARC_PLANE
    RESOURCE_BRIDGE -->|Outbound HTTPS| ARC_PLANE
    
    ARC_PLANE --> ARM
    ARM --> POLICY
    ARM --> RBAC
    ARM --> MONITOR
    ARM --> DEFENDER
    ARM --> PORTAL
    
    style AZURE fill:#1e3a5f,color:#fff
    style ARC_PLANE fill:#7c3aed,color:#fff
    style ONPREM fill:#0f4c75,color:#fff
    style AGENTS fill:#065f46,color:#fff

Azure Arc Technical Prerequisites

Network (main challenge):

sequenceDiagram
    participant SERVER as On-premises\nServer
    participant PROXY as Proxy/Firewall\n(optional)
    participant AZURE as Azure Endpoints\n(HTTPS/TLS)

    SERVER->>PROXY: Outbound HTTPS (443)
    PROXY->>AZURE: Heartbeat every 5 min
    AZURE-->>SERVER: Config, policies, extensions
    
    Note over SERVER,AZURE: All traffic is OUTBOUND\nNo inbound connections required!
    Note over PROXY: Support: HTTP/HTTPS proxy\nSupport: Azure Private Link

Required Azure Endpoints (partial list):

# Endpoints required for Azure Arc (outbound)
management.azure.com            # Azure Resource Manager
login.microsoftonline.com       # Microsoft Entra ID (auth)
*.his.arc.azure.com             # Hybrid Identity Service
*.guestconfiguration.azure.com  # Azure Policy Guest Config
*.blob.core.windows.net         # Download extensions
*.trafficmanager.net            # Global load balancing
gbl.his.arc.azure.com           # Heartbeat

Required Azure Permissions:

Supported Systems:

3. Implementing Azure Arc – Servers

Server Onboarding Methods

flowchart TD
    COUNT{Number of\nservers?}
    
    COUNT -->|< 10 servers| INTERACTIVE["Interactive methods\n(manual login on each server)"]
    COUNT -->|> 10 servers| ATSCALE["At-scale methods\n(automated, Service Principal)"]
    
    INTERACTIVE --> PORTAL_SCRIPT["Script generated from\nAzure Portal"]
    INTERACTIVE --> WIN_ADMIN["Windows Admin Center"]
    INTERACTIVE --> POWERSHELL["PowerShell module\naz.connectedmachine"]
    INTERACTIVE --> ARC_WIZARD["Azure Arc Setup Wizard\n(built into Windows Server)"]
    
    ATSCALE --> DEPLOYMENT_SCRIPT["Deployment script\n+ config management tool"]
    ATSCALE --> SCCM["Configuration Manager\n(SCCM)"]
    ATSCALE --> GPO["Group Policy\n(Windows AD)"]
    ATSCALE --> ANSIBLE["Ansible\n(multi-platform)"]
    ATSCALE --> CHEF_PUPPET["Chef / Puppet"]
    ATSCALE --> AZURE_MIGRATE["Azure Migrate"]
    ATSCALE --> DEFENDER["Microsoft Defender\nfor Cloud"]
    
    style INTERACTIVE fill:#3b82f6,color:#fff
    style ATSCALE fill:#ef4444,color:#fff

Interactive Server Onboarding

# ===== GENERATE ONBOARDING SCRIPT VIA CLI =====

# Prerequisites: Register resource providers
az provider register --namespace Microsoft.HybridCompute
az provider register --namespace Microsoft.GuestConfiguration
az provider register --namespace Microsoft.HybridConnectivity

# Create a service principal for at-scale onboarding
az ad sp create-for-rbac \
    --name "arc-onboarding-sp" \
    --role "Azure Connected Machine Onboarding" \
    --scopes "/subscriptions/{sub-id}/resourceGroups/arc-servers-rg"

# ===== ONBOARDING SCRIPT FOR LINUX =====

# On the Linux machine to onboard:

# Option 1: Script generated from Azure Portal
# (Download and run the generated script)
curl -L https://aka.ms/azcmagent -o /tmp/install_linux_azcmagent.sh
bash /tmp/install_linux_azcmagent.sh

# Connect to Azure
azcmagent connect \
    --resource-group "arc-servers-rg" \
    --resource-name "my-linux-server" \
    --subscription-id "sub-id" \
    --tenant-id "tenant-id" \
    --location "eastus" \
    --cloud "AzureCloud" \
    --correlation-id "$(uuidgen)"

# Verify the connection
azcmagent show

# ===== ONBOARDING SCRIPT FOR WINDOWS (PowerShell) =====

# Install the agent
Invoke-WebRequest -Uri "https://aka.ms/AzureConnectedMachineAgent" `
    -OutFile "$env:TEMP\AzureConnectedMachineAgent.msi"

msiexec /i "$env:TEMP\AzureConnectedMachineAgent.msi" /l*v "$env:TEMP\AzureConnectedMachineAgentInstall.log" /qn

# Connect to Azure (with Service Principal for automation)
& "$env:ProgramW6432\AzureConnectedMachineAgent\azcmagent.exe" connect `
    --service-principal-id "sp-client-id" `
    --service-principal-secret "sp-secret" `
    --resource-group "arc-servers-rg" `
    --resource-name $env:COMPUTERNAME `
    --subscription-id "sub-id" `
    --tenant-id "tenant-id" `
    --location "eastus"

# Verify
& "$env:ProgramW6432\AzureConnectedMachineAgent\azcmagent.exe" show

At-Scale Onboarding with Ansible

# arc-onboarding.yml - Ansible Playbook for at-scale onboarding

---
- name: Azure Arc Server Onboarding
  hosts: all
  become: yes
  vars:
    arc_tenant_id: "{{ lookup('env', 'AZURE_TENANT_ID') }}"
    arc_subscription_id: "{{ lookup('env', 'AZURE_SUBSCRIPTION_ID') }}"
    arc_resource_group: "arc-servers-rg"
    arc_location: "eastus"
    arc_sp_app_id: "{{ lookup('env', 'ARC_SP_APP_ID') }}"
    arc_sp_secret: "{{ lookup('env', 'ARC_SP_SECRET') }}"
    arc_tags:
      Environment: production
      Owner: ops-team
      ManagedBy: AzureArc
  
  tasks:
    - name: Install Azure Connected Machine agent (Linux)
      when: ansible_os_family != "Windows"
      block:
        - name: Download installation script
          get_url:
            url: https://aka.ms/azcmagent
            dest: /tmp/install_linux_azcmagent.sh
            mode: '0755'
        
        - name: Install agent
          command: bash /tmp/install_linux_azcmagent.sh
          register: install_result
        
        - name: Connect to Azure Arc
          command: >
            azcmagent connect
            --service-principal-id "{{ arc_sp_app_id }}"
            --service-principal-secret "{{ arc_sp_secret }}"
            --resource-group "{{ arc_resource_group }}"
            --resource-name "{{ inventory_hostname }}"
            --subscription-id "{{ arc_subscription_id }}"
            --tenant-id "{{ arc_tenant_id }}"
            --location "{{ arc_location }}"
          register: connect_result
        
        - name: Verify connection
          command: azcmagent show
          register: show_result
        
        - name: Display Arc status
          debug:
            msg: "{{ show_result.stdout }}"
    
    - name: Install Azure Connected Machine agent (Windows)
      when: ansible_os_family == "Windows"
      block:
        - name: Download agent MSI
          win_get_url:
            url: https://aka.ms/AzureConnectedMachineAgent
            dest: "C:\\Temp\\AzureConnectedMachineAgent.msi"
        
        - name: Install agent
          win_package:
            path: "C:\\Temp\\AzureConnectedMachineAgent.msi"
            state: present
        
        - name: Connect to Azure Arc
          win_command: >
            "C:\Program Files\AzureConnectedMachineAgent\azcmagent.exe" connect
            --service-principal-id "{{ arc_sp_app_id }}"
            --service-principal-secret "{{ arc_sp_secret }}"
            --resource-group "{{ arc_resource_group }}"
            --resource-name "{{ inventory_hostname }}"
            --subscription-id "{{ arc_subscription_id }}"
            --tenant-id "{{ arc_tenant_id }}"
            --location "{{ arc_location }}"

Windows Services Created by the Arc Agent

When onboarding a Windows server, three Windows services are created:

graph TD
    subgraph WIN_SERVICES["Windows Services Created by Azure Arc"]
        HIMDS["Azure Hybrid Instance\nMetadata Service (himds)\n\nSynchronizes metadata\nwith Azure\n\nHosts local REST API\nfor extensions and apps\n\nManages MI tokens\n(Managed Identity)"]
        
        GC_ARC["Guest Configuration\nArc Service\n\nAudit and enforcement\nAzure Policy Guest Config\n\nContinuous compliance"]
        
        GC_EXT["Guest Configuration\nExtension Service\n\nInstalls, updates\nand manages extensions\n\n(Azure Monitor, Defender,\nUpdate Management, etc.)"]
    end
    
    style WIN_SERVICES fill:#1e3a5f,color:#fff
    style HIMDS fill:#3b82f6,color:#fff
    style GC_ARC fill:#059669,color:#fff
    style GC_EXT fill:#7c3aed,color:#fff

Note: Group Policy can sometimes block these services (particularly the “Log on as a service” right for the HIMDS virtual account). Check GPOs if the agent is not working correctly.

4. Implementing Azure Arc – Kubernetes

Arc Kubernetes Architecture

graph TD
    subgraph CLUSTER["Kubernetes Cluster (on-premises / other cloud)"]
        subgraph NAMESPACE["Namespace: azure-arc"]
            AGENTS_HELM["Agents deployed via Helm"]
            RESOURCE_SYNC[resource-sync-agent\nSync metadata → Azure]
            CLUSTER_META[cluster-metadata-operator\nCollects metadata]
            CLUSTERCONN["clusterconnect (optional)\nReverse proxy API"]
            ENTRA_AUTH["Entra ID auth (optional)\nAAD Authentication"]
            AZURE_RBAC["Azure RBAC auth (optional)\nGranular authorization"]
        end
        
        subgraph WORKLOADS["Application Namespaces"]
            APPS[Applications]
            GITOPS_FLUX["Flux/Argo CD\n(deployed via GitOps config)"]
        end
    end
    
    subgraph AZURE_ARC["Azure Arc / Azure"]
        ARM2[Azure Resource Manager]
        POLICY2[Azure Policy]
        MONITOR2[Azure Monitor]
        GITOPS_CONF[GitOps Configuration]
    end
    
    RESOURCE_SYNC -->|Heartbeat + Metadata| ARM2
    ARM2 -->|Extensions, Policies| CLUSTER
    GITOPS_CONF -->|Pull config| GITOPS_FLUX
    GITOPS_FLUX -->|Deploy| WORKLOADS
    
    style CLUSTER fill:#0f172a,color:#fff
    style AZURE_ARC fill:#1e3a5f,color:#fff

Onboarding a Kubernetes Cluster

# ===== PREREQUISITES =====

# Register resource providers
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation

# Install the az connectedk8s extension
az extension add --name connectedk8s
az extension add --name k8s-configuration
az extension add --name k8s-extension

# ===== CONNECT THE CLUSTER =====

# The kubectl context must point to the cluster to onboard
kubectl config current-context

# Connect the cluster to Azure Arc
az connectedk8s connect \
    --name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --location "eastus" \
    --tags Environment=production Owner=ops-team

# Verify the connection
az connectedk8s show \
    --name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg"

# View agents deployed in the azure-arc namespace
kubectl get pods -n azure-arc

# ===== CONFIGURE CLUSTER CONNECT (API access without inbound rules) =====

az connectedk8s enable-features \
    --name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --features cluster-connect azure-rbac

# Access the cluster without inbound firewall rules
az connectedk8s proxy \
    --name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --port 8443

# In another terminal:
export KUBECONFIG="/tmp/config"
kubectl get nodes

# ===== CONFIGURE AZURE RBAC ON THE CLUSTER =====

# Role assignment for a user (via Entra ID)
az role assignment create \
    --role "Azure Arc Kubernetes Cluster Admin" \
    --assignee "user@company.com" \
    --scope "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Kubernetes/connectedClusters/my-k8s-arc-cluster"

Arc Kubernetes Extensions

# ===== INSTALL ARC KUBERNETES EXTENSIONS =====

# Azure Monitor Container Insights extension (logs)
az k8s-extension create \
    --name "azuremonitor-containers" \
    --extension-type Microsoft.AzureMonitor.Containers \
    --scope cluster \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters \
    --configuration-settings logAnalyticsWorkspaceResourceID="/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/my-workspace"

# Managed Prometheus extension (metrics)
az k8s-extension create \
    --name "azuremonitor-metrics" \
    --extension-type Microsoft.AzureMonitor.Containers.Metrics \
    --scope cluster \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters

# Microsoft Defender for Containers extension
az k8s-extension create \
    --name "mdc" \
    --extension-type Microsoft.AzureDefender.Kubernetes \
    --scope cluster \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters

# List installed extensions
az k8s-extension list \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters \
    --output table

5. Implementing Azure Arc – SQL and Data Services

SQL Server enabled by Azure Arc

graph LR
    subgraph SERVER["Arc-enabled Server\n(Windows/Linux)"]
        WINDOWS_SVC[Connected Machine\nAgent]
        SQL_EXT_LOCAL[Azure Extension\nfor SQL Server]
        SQL_INST["SQL Server Instance(s)\n(2012 - 2022)"]
    end
    
    subgraph AZURE2["Azure Arc / Azure"]
        ARC_SERVER[Arc-enabled Machine\nResource]
        ARC_SQL["Arc-enabled SQL Server\nResource(s)"]
        DEFENDER_SQL[Defender for SQL]
        ENTRA_AUTH2[Entra Authentication]
        PURVIEW[Microsoft Purview\nData Governance]
        BEST_PRACTICES[Best Practice\nAssessments]
        ESU["Extended Security\nUpdates (ESU)"]
    end
    
    WINDOWS_SVC -->|Heartbeat| ARC_SERVER
    SQL_EXT_LOCAL -->|Metadata| ARC_SQL
    SQL_INST --> SQL_EXT_LOCAL
    
    ARC_SQL --> DEFENDER_SQL
    ARC_SQL --> ENTRA_AUTH2
    ARC_SQL --> PURVIEW
    ARC_SQL --> BEST_PRACTICES
    ARC_SQL --> ESU
    
    style SERVER fill:#0f4c75,color:#fff
    style AZURE2 fill:#1e3a5f,color:#fff

# ===== SQL SERVER ONBOARDING VIA ARC =====

# Option 1: Enable SQL Server during server onboarding
# In the portal, when generating the onboarding script:
# → Enable "Automatically connect all SQL Server instances"

# Option 2: Enable SQL Server on an already-connected Arc server
az sql server-arc enable \
    --name "my-sql-server" \
    --resource-group "arc-servers-rg"

# ===== CONFIGURE BEST PRACTICE ASSESSMENTS =====

# Install the Azure Monitor extension (required for assessments)
az connectedmachine extension create \
    --name "MonitoringAgent" \
    --publisher "Microsoft.Azure.Monitor" \
    --type "AzureMonitorWindowsAgent" \
    --resource-group "arc-servers-rg" \
    --machine-name "my-windows-server" \
    --location "eastus"

# View assessment status
az sql server-arc show \
    --name "my-sql-server" \
    --resource-group "arc-servers-rg" \
    --query "properties.bestPracticesAssessment" \
    --output json

Arc-enabled Data Services (SQL Managed Instance on K8s)

flowchart TD
    PREREQ[Arc-enabled\nKubernetes Cluster]
    
    DATA_CTRL["Data Controller\n(deployed first)"]
    
    SQLMI[Arc-enabled\nSQL Managed Instance]
    
    MODES{Connectivity\nmode?}
    
    DIRECT[Directly Connected\n- Internet connection\n- Auto-updated policies\n- Direct Azure billing]
    
    INDIRECT["Indirectly Connected\n- Isolated network (air-gap)\n- Manual policy updates\n- Manually exported billing"]
    
    PREREQ --> DATA_CTRL
    DATA_CTRL --> SQLMI
    SQLMI --> MODES
    MODES --> DIRECT
    MODES --> INDIRECT
    
    style PREREQ fill:#3b82f6,color:#fff
    style DATA_CTRL fill:#7c3aed,color:#fff
    style SQLMI fill:#10b981,color:#fff
    style DIRECT fill:#059669,color:#fff
    style INDIRECT fill:#d97706,color:#fff

# ===== DEPLOY ARC DATA SERVICES =====

# Prerequisites: Arc-enabled Kubernetes cluster
# Register resource providers
az provider register --namespace Microsoft.AzureArcData
az provider register --namespace Microsoft.KubernetesConfiguration

# Install required CLI extensions
az extension add --name arcdata
az extension add --name k8s-extension

# 1. Create the Data Controller (directly connected mode)
az arcdata dc create \
    --name "arc-data-controller" \
    --resource-group "arc-kubernetes-rg" \
    --location "eastus" \
    --connectivity-mode direct \
    --profile-name "azure-arc-kubeadm" \
    --storage-class "default" \
    --k8s-namespace "arc-data"

# 2. Create a SQL Managed Instance
az sql mi-arc create \
    --name "my-sql-mi-arc" \
    --resource-group "arc-kubernetes-rg" \
    --cores-request "2" \
    --cores-limit "4" \
    --memory-request "2Gi" \
    --memory-limit "8Gi" \
    --storage-class-data "default" \
    --storage-class-logs "default" \
    --volume-size-data "5Gi" \
    --volume-size-logs "5Gi" \
    --tier GeneralPurpose \
    --k8s-namespace "arc-data"

# 3. Get connection endpoints
az sql mi-arc show \
    --name "my-sql-mi-arc" \
    --k8s-namespace "arc-data" \
    --use-k8s \
    --query "status.endpoints" \
    --output json

6. Governance and Security with Arc

Azure Policy for Arc Resources

graph TD
    subgraph POLICY_ASSIGNMENT["Azure Policy Assignments"]
        MG_POLICY[Policy on\nManagement Group]
        SUB_POLICY[Policy on\nSubscription]
        RG_POLICY[Policy on\nResource Group]
    end
    
    subgraph ARC_RESOURCES["Arc Resources"]
        SERVERS_ARC[Arc-enabled\nServers]
        K8S_ARC[Arc-enabled\nKubernetes]
    end
    
    MG_POLICY -->|Inheritance| SUB_POLICY
    SUB_POLICY -->|Inheritance| RG_POLICY
    RG_POLICY -->|Evaluates| SERVERS_ARC
    RG_POLICY -->|Evaluates| K8S_ARC
    
    subgraph EFFECTS["Effects on Arc Servers"]
        AUDIT_SERVER[Audit\nReport non-compliance]
        DEPLOY_SERVER[DeployIfNotExists\nInstall agent/extension]
        GC_SERVER[Guest Configuration\nAudit OS settings]
    end
    
    SERVERS_ARC --> EFFECTS
    
    style POLICY_ASSIGNMENT fill:#7c3aed,color:#fff
    style ARC_RESOURCES fill:#1e3a5f,color:#fff
    style EFFECTS fill:#059669,color:#fff

# ===== POLICY EXAMPLES FOR ARC =====

# Policy: Enforce Azure Monitor on all Arc servers (Linux)
az policy assignment create \
    --name "arc-linux-monitor" \
    --display-name "Install Azure Monitor on Arc Linux servers" \
    --scope "/subscriptions/{sub-id}/resourceGroups/arc-servers-rg" \
    --policy "/providers/Microsoft.Authorization/policyDefinitions/a4034bc6-ae50-406d-bf76-50f4ee5a7811" \
    --assign-identity \
    --location eastus \
    --params '{
        "logAnalyticsWorkspaceId": {
            "value": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/my-workspace"
        }
    }'

# Policy: Enforce Azure Monitor on all Arc servers (Windows)
az policy assignment create \
    --name "arc-windows-monitor" \
    --display-name "Install Azure Monitor on Arc Windows servers" \
    --scope "/subscriptions/{sub-id}/resourceGroups/arc-servers-rg" \
    --policy "/providers/Microsoft.Authorization/policyDefinitions/ca817e41-e85a-4783-bc7f-dc532d36235e" \
    --assign-identity \
    --location eastus \
    --params '{
        "logAnalyticsWorkspaceId": {
            "value": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/my-workspace"
        }
    }'

# Policy for Arc Kubernetes: Deploy Azure Policy Add-on
az policy assignment create \
    --name "arc-k8s-policy-addon" \
    --scope "/subscriptions/{sub-id}/resourceGroups/arc-kubernetes-rg" \
    --policy "/providers/Microsoft.Authorization/policyDefinitions/0adc5395-9169-4b9b-8687-af838d69410a"

# View compliance
az policy state summarize \
    --resource-group "arc-servers-rg" \
    --output table

RBAC and Entra ID with Arc

# ===== RBAC ON ARC SERVERS =====

# Azure Connected Machine Resource Administrator role
az role assignment create \
    --role "Azure Connected Machine Resource Administrator" \
    --assignee "ops-team@company.com" \
    --scope "/subscriptions/{sub}/resourceGroups/arc-servers-rg"

# ===== RBAC ON ARC K8S CLUSTERS =====

# Azure Arc Kubernetes Cluster Admin role (equivalent to cluster-admin)
az role assignment create \
    --role "Azure Arc Kubernetes Cluster Admin" \
    --assignee "admin@company.com" \
    --scope "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Kubernetes/connectedClusters/my-cluster"

# Azure Arc Kubernetes Viewer role (read-only equivalent)
az role assignment create \
    --role "Azure Arc Kubernetes Viewer" \
    --assignee "dev@company.com" \
    --scope "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Kubernetes/connectedClusters/my-cluster"

# ===== MICROSOFT DEFENDER FOR CLOUD =====

# Enable Defender for Servers on Arc-enabled servers
az security pricing create \
    --name Servers \
    --tier Standard

# View Arc security recommendations
az security assessment list \
    --resource-group "arc-servers-rg" \
    --query "[?properties.status.code == 'Unhealthy'].{Name:displayName, Severity:properties.metadata.severity}" \
    --output table

7. Monitoring Arc Resources

Configure Azure Monitor for Arc Servers

# ===== CONFIGURE DATA COLLECTION RULES =====

# Create a DCR to collect logs and metrics
az monitor data-collection rule create \
    --name "arc-servers-dcr" \
    --resource-group "arc-servers-rg" \
    --location "eastus" \
    --data-flows '[
        {
            "streams": ["Microsoft-Syslog", "Microsoft-Event"],
            "destinations": ["log-analytics-workspace"]
        },
        {
            "streams": ["Microsoft-InsightsMetrics"],
            "destinations": ["log-analytics-workspace"]
        }
    ]' \
    --destinations '{
        "logAnalytics": [
            {
                "workspaceResourceId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/my-workspace",
                "name": "log-analytics-workspace"
            }
        ]
    }' \
    --data-sources '{
        "syslog": [
            {
                "name": "syslog-source",
                "streams": ["Microsoft-Syslog"],
                "facilityNames": ["syslog"],
                "logLevels": ["Error", "Warning", "Info"]
            }
        ],
        "performanceCounters": [
            {
                "name": "performance-source",
                "streams": ["Microsoft-InsightsMetrics"],
                "samplingFrequencyInSeconds": 60,
                "counterSpecifiers": [
                    "\\Processor(_Total)\\% Processor Time",
                    "\\Memory\\Available MBytes"
                ]
            }
        ]
    }'

KQL Queries for Arc Servers

// ===== KQL QUERIES FOR ARC SERVERS =====

// Inventory of all Arc-enabled servers
AzureActivity
| where ResourceProviderValue == "MICROSOFT.HYBRIDCOMPUTE"
| where OperationNameValue == "MICROSOFT.HYBRIDCOMPUTE/MACHINES/WRITE"
| project TimeGenerated, ResourceGroup, Resource, Caller
| order by TimeGenerated desc

// Arc agent heartbeats (connection health)
Heartbeat
| where TimeGenerated > ago(1h)
| where Category == "Direct Agent"
| summarize LastHeartbeat = max(TimeGenerated), Count = count() by Computer
| where LastHeartbeat < ago(30m)  // Agents that haven't heartbeated in 30 min
| project Computer, LastHeartbeat, Count

// CPU and memory of Arc servers
InsightsMetrics
| where TimeGenerated > ago(1h)
| where Namespace == "Processor" and Name == "UtilizationPercentage"
| summarize AvgCPU = avg(Val), MaxCPU = max(Val) by Computer
| order by MaxCPU desc

// Errors in Syslog (Linux)
Syslog
| where TimeGenerated > ago(1h)
| where SeverityLevel in ("error", "critical", "alert")
| project TimeGenerated, Computer, Facility, SeverityLevel, SyslogMessage
| order by TimeGenerated desc

// Windows Events (critical errors)
Event
| where TimeGenerated > ago(1h)
| where EventLevelName in ("Error", "Critical")
| project TimeGenerated, Computer, EventID, EventLevelName, RenderedDescription
| order by TimeGenerated desc

// Azure Policy compliance on Arc servers
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.GUESTCONFIGURATION"
| where OperationName == "Microsoft.GuestConfiguration/guestConfigurationAssignments/write"
| project TimeGenerated, Resource, ResultDescription, complianceStatus_s
| summarize 
    Compliant = countif(complianceStatus_s == "Compliant"),
    NonCompliant = countif(complianceStatus_s == "NonCompliant")
    by Resource

8. GitOps with Arc Kubernetes

GitOps Principle with Azure Arc

sequenceDiagram
    participant DEV as Developer
    participant GIT as Git Repository\n(source of truth)
    participant FLUX as Flux CD\n(in the Arc cluster)
    participant K8S as Kubernetes Cluster\n(Arc-enabled)
    participant AZURE as Azure Arc\n(Control Plane)

    DEV->>GIT: Push K8s configuration\n(manifests, Helm charts)
    
    Note over AZURE: Defines the GitOps Configuration\n(source, frequency, namespace)
    AZURE->>FLUX: Deploys Flux in the cluster
    
    loop Reconciliation (every N minutes)
        FLUX->>GIT: Pull configuration
        GIT-->>FLUX: New manifests detected
        FLUX->>K8S: Apply manifests
        K8S-->>FLUX: Deployment status
        FLUX->>AZURE: Status report
    end
    
    Note over K8S: No inbound connections!\nThe cluster PULLS from Git

Configure GitOps with Azure Arc

# ===== CREATE A GITOPS CONFIGURATION =====

# Prerequisites: Install the CLI extension
az extension add --name k8s-configuration

# Basic GitOps configuration (Flux)
az k8s-configuration flux create \
    --name "my-app-config" \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters \
    --url "https://github.com/my-org/my-gitops-repo" \
    --branch "main" \
    --interval 1m \
    --sync-interval 1m \
    --scope cluster \
    --namespace "gitops-configs"

# GitOps configuration with authentication (private repo)
az k8s-configuration flux create \
    --name "my-app-private-config" \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters \
    --url "https://github.com/my-org/my-private-gitops-repo" \
    --branch "main" \
    --interval 1m \
    --https-user "my-user" \
    --https-key "$(cat ~/.ssh/github-pat)"

# View configuration status
az k8s-configuration flux show \
    --name "my-app-config" \
    --cluster-name "my-k8s-arc-cluster" \
    --resource-group "arc-kubernetes-rg" \
    --cluster-type connectedClusters \
    --output json

9. Troubleshooting and Advanced Architecture

Diagnosing Arc Agent Issues

# ===== ARC SERVER DIAGNOSTICS =====

# Check agent status
azcmagent show

# Agent logs (Linux)
cat /var/opt/azcmagent/log/azcmagent.log

# Agent logs (Windows)
Get-Content "C:\ProgramData\AzureConnectedMachineAgent\Log\azcmagent.log" -Tail 100

# Check network connectivity to Azure
azcmagent check --cloud AzureCloud --location eastus

# Test connectivity to required endpoints
# Linux
curl -I https://management.azure.com
curl -I https://login.microsoftonline.com
curl -I https://gbl.his.arc.azure.com

# Reinstall the agent if necessary
# Linux
azcmagent disconnect
bash /tmp/install_linux_azcmagent.sh
azcmagent connect [...]

# Windows (PowerShell)
& "$env:ProgramW6432\AzureConnectedMachineAgent\azcmagent.exe" disconnect
msiexec /i AzureConnectedMachineAgent.msi /qn
& "$env:ProgramW6432\AzureConnectedMachineAgent\azcmagent.exe" connect [...]

# ===== KUBERNETES ARC DIAGNOSTICS =====

# Check agents in the azure-arc namespace
kubectl get pods -n azure-arc
kubectl describe pod <pod-name> -n azure-arc

# Logs for a specific agent
kubectl logs <pod-name> -n azure-arc --previous
kubectl logs resource-sync-agent-xxx -n azure-arc

# Check Azure Arc events
kubectl get events -n azure-arc --sort-by=.metadata.creationTimestamp

# Check ConnectedCluster CRD conditions
kubectl get connectedcluster arc -n azure-arc -o json | jq '.status.conditions'

Azure Arc Resource Bridge

The Resource Bridge is a Microsoft-managed Kubernetes cluster deployed in your hypervisor. It acts as a relay between Azure and hypervisor platforms like VMware vSphere.

graph TD
    subgraph AZURE3["Azure Cloud"]
        ARM3[Azure Resource Manager]
        PORTAL3[Azure Portal]
    end
    
    subgraph VSPHERE["vSphere / SCVMM"]
        RESOURCE_BRIDGE2["Azure Arc Resource Bridge\n(managed K8s cluster)"]
        
        subgraph VMS["VMware VMs"]
            VM1[Windows VM\n→ Arc-enabled server]
            VM2[Linux VM\n→ Arc-enabled server]
            VM3[SQL VM\n→ Arc-enabled SQL]
        end
    end
    
    ARM3 -->|"Commands (create, start, stop VMs)"| RESOURCE_BRIDGE2
    RESOURCE_BRIDGE2 -->|vSphere API| VMS
    RESOURCE_BRIDGE2 -->|Heartbeat + metadata| ARM3
    PORTAL3 -->|VM management| ARM3
    
    style AZURE3 fill:#1e3a5f,color:#fff
    style VSPHERE fill:#0f4c75,color:#fff
    style RESOURCE_BRIDGE2 fill:#7c3aed,color:#fff

# ===== DEPLOY RESOURCE BRIDGE FOR VMWARE =====

# Prerequisites: 
# - vCenter 7.0+ with admin rights
# - Python 3.8+ on the installation machine
# - Azure CLI with arc, arcappliance extensions

az extension add --name arcappliance
az extension add --name connectedvmware

# Deploy the Resource Bridge
az arcappliance deploy vmware \
    --config-file "vmware-appliance.yaml" \
    --outfile "kubeconfig" \
    --address "vcenter-address" \
    --username "vcenter-admin@vsphere.local" \
    --password "$VCENTER_PASSWORD"

# Connect the Resource Bridge to Azure
az arcappliance connect vmware \
    --config-file "vmware-appliance.yaml" \
    --kubeconfig "kubeconfig"

# Create the Custom Location connection
az customlocation create \
    --name "my-vcenter-custom-location" \
    --resource-group "arc-vmware-rg" \
    --namespace "arc-vmware" \
    --host-resource-id "$(az arcappliance show ... --query id --output tsv)" \
    --cluster-extension-ids "$(az k8s-extension show ... --query id --output tsv)"

# Onboard the vCenter
az connectedvmware vcenter connect \
    --name "my-vcenter" \
    --resource-group "arc-vmware-rg" \
    --location "eastus" \
    --custom-location "my-vcenter-custom-location" \
    --address "vcenter-address" \
    --username "vcenter-admin@vsphere.local" \
    --password "$VCENTER_PASSWORD"

10. Complete Code Examples

Complete Arc Resource Inventory Script

#!/bin/bash
# arc-inventory-report.sh
# Generates an inventory report of all Arc resources

echo "=== AZURE ARC INVENTORY REPORT ==="
echo "Date: $(date)"
echo ""

# ===== ARC SERVERS =====
echo "--- Arc-enabled Servers ---"
az connectedmachine list \
    --query "[].{Name:name, RG:resourceGroup, OS:properties.osType, Version:properties.agentVersion, Status:properties.status, Location:location}" \
    --output table
echo ""

# ===== ARC KUBERNETES =====
echo "--- Arc-enabled Kubernetes Clusters ---"
az connectedk8s list \
    --query "[].{Name:name, RG:resourceGroup, Distribution:properties.distribution, K8sVersion:properties.kubernetesVersion, Connectivity:properties.connectivityStatus}" \
    --output table
echo ""

# ===== ARC SQL SERVER =====
echo "--- Arc-enabled SQL Servers ---"
az sql server-arc list \
    --query "[].{Name:name, RG:resourceGroup, Version:properties.version, Edition:properties.edition, Status:properties.status}" \
    --output table 2>/dev/null
echo ""

# ===== COUNT BY TYPE =====
SERVER_COUNT=$(az connectedmachine list --query "length(@)" --output tsv)
K8S_COUNT=$(az connectedk8s list --query "length(@)" --output tsv)

echo "=== SUMMARY ==="
echo "Arc-enabled Servers: $SERVER_COUNT"
echo "Arc-enabled K8s Clusters: $K8S_COUNT"
echo ""

# ===== OFFLINE AGENTS =====
echo "--- Offline agents (status != Connected) ---"
az connectedmachine list \
    --query "[?properties.status != 'Connected'].{Name:name, Status:properties.status, LastHeartbeat:properties.lastStatusChange}" \
    --output table

Deploy GitOps Configuration At-Scale via Azure Policy (Bicep)

// gitops-policy.bicep
// Deploys GitOps on all AKS and Arc clusters via Azure Policy

param gitRepoUrl string = 'https://github.com/my-org/baseline-gitops'
param gitRepoBranch string = 'main'
param syncInterval string = '5m'

// Policy Initiative for GitOps
resource gitopsPolicyInitiative 'Microsoft.Authorization/policySetDefinitions@2021-06-01' = {
  name: 'enforce-gitops-kubernetes'
  properties: {
    displayName: 'Enforce GitOps baseline on Kubernetes clusters'
    description: 'Deploys a baseline GitOps configuration on all AKS and Arc clusters'
    policyType: 'Custom'
    policies: [
      {
        policyDefinitionId: '/providers/Microsoft.Authorization/policyDefinitions/1e9e2d5c-4b62-4c0e-a8be-c06e75ef4e8f'  // Flux GitOps AKS
        parameters: {
          gitRepoUrl: {
            value: gitRepoUrl
          }
          gitRepoBranch: {
            value: gitRepoBranch
          }
        }
      }
    ]
  }
}

// Assignment on the subscription
resource gitopsPolicyAssignment 'Microsoft.Authorization/policyAssignments@2021-06-01' = {
  name: 'enforce-gitops-all-k8s'
  identity: {
    type: 'SystemAssigned'  // MI for DeployIfNotExists
  }
  location: 'eastus'
  properties: {
    displayName: 'Enforce GitOps on all Kubernetes clusters'
    policyDefinitionId: gitopsPolicyInitiative.id
    enforcementMode: 'Default'
  }
}

// Role assignment for the assignment's MI
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(gitopsPolicyAssignment.id, 'contributor')
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', 'b24988ac-6180-42a0-ab88-20f7382dd24c')  // Contributor
    principalId: gitopsPolicyAssignment.identity.principalId
    principalType: 'ServicePrincipal'
  }
}

output policyAssignmentId string = gitopsPolicyAssignment.id

11. Comparative Tables

Comparison of Arc Resource Types

Arc Network Connectivity: Mode Comparison

Recommended Scenarios for Azure Arc

12. Glossary

Search Terms

azure · arc · hybrid · environments · core · infrastructure · microsoft · kubernetes · gitops · onboarding · architecture · resource · server · servers · services · sql · agent · at-scale · cloud · comparison · configure · data · policy · resources