Azure ML Workspace Fundamentals

Course: Azure Machine Learning Workspace Fundamentals Level: Beginner Objective: Master the fundamentals of Azure ML infrastructure

Table of Contents

1. Azure ML Workspace Overview
2. Workspace Architecture and Dependencies
3. Creating and Configuring a Workspace
4. Compute – Compute Infrastructure
5. Data and Datasets
6. Environments – Reproducibility
7. Governance and Lifecycle
8. Workspace Security
9. Practical Implementation with the SDK
10. Cost Optimization
11. Summary and Key Points
12. Glossary

1. Azure ML Workspace Overview

1.1 What is an Azure ML Workspace?

Imagine opening a large restaurant. You need a kitchen, a pantry, recipes, and chefs. Instead of managing everything from separate buildings, you create one central restaurant where everything is organized together.

That is exactly what an Azure ML Workspace is: the central headquarters for all your machine learning projects.

flowchart TD
    WORKSPACE["🏢 Azure ML Workspace\n(ML Headquarters)"] --> COMPUTE["⚙️ Compute\n(CPU/GPU Clusters)"]
    WORKSPACE --> DATA["📊 Data Assets\n(Versioned Datasets)"]
    WORKSPACE --> ENV["📦 Environments\n(Python Dependencies)"]
    WORKSPACE --> MODELS["🧠 Models\n(Registry + Versions)"]
    WORKSPACE --> ENDPOINTS["🌐 Endpoints\n(ML APIs)"]
    WORKSPACE --> JOBS["▶️ Jobs/Experiments\n(Full History)"]
    WORKSPACE --> NOTEBOOKS["📓 Notebooks\n(Interactive IDE)"]
    WORKSPACE --> PIPELINES["🔄 Pipelines\n(Automated Workflows)"]

Why have a Workspace?

1.2 Workspace Use Cases

The Azure ML Workspace is used whenever you:

• Train an ML model (classification, regression, clustering…)
• Track experiments and compare runs
• Manage datasets and models
• Deploy models to production
• Collaborate with a Data Science team

2. Workspace Architecture and Dependencies

2.1 Automatically Created Services

When you create an Azure ML Workspace, Azure automatically deploys several services:

flowchart LR
    subgraph "Resource Group"
        WORKSPACE["🏢 Azure ML Workspace\n(Main Resource)"]
        
        subgraph "Required Dependencies"
            STORAGE["💾 Azure Storage Account\n• Datasets and models\n• Experiment logs\n• Job artifacts"]
            KV["🔑 Azure Key Vault\n• DB passwords\n• API keys\n• Secure connections"]
            AI["📈 Application Insights\n(Optional)\n• Endpoint monitoring\n• Alerts"]
        end
        
        subgraph "Optional Dependencies"
            ACR["🐳 Azure Container Registry\n• Custom Docker images\n• Custom environments"]
        end
    end
    
    WORKSPACE --> STORAGE
    WORKSPACE --> KV
    WORKSPACE -.-> AI
    WORKSPACE -.-> ACR

2.2 Role of Each Dependency

Hidden costs: These additional resources generate costs even when the workspace is not actively used. Be sure to include them in your budget estimate.

2.3 Datastores – Connections to Data Sources

A Datastore is a configured connection to an external data source:

flowchart TD
    WORKSPACE["Azure ML Workspace"] --> DS_DEFAULT["workspaceblobstore\n(Default Datastore)\n→ Associated Blob Storage"]
    WORKSPACE --> DS_CUSTOM["Custom Datastores"]
    
    DS_CUSTOM --> ADLS["Azure Data Lake\nStorage Gen2"]
    DS_CUSTOM --> SQL["Azure SQL Database"]
    DS_CUSTOM --> BLOB_EXT["Azure Blob Storage\n(other account)"]
    DS_CUSTOM --> FILESHARE["Azure File Share"]
    DS_CUSTOM --> DATABRICKS["Azure Databricks\nFilesystem"]

# Create and manage datastores
from azure.ai.ml.entities import AzureBlobDatastore, AzureDataLakeGen2Datastore
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
import os

ml_client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
    resource_group_name=os.environ["AZURE_RESOURCE_GROUP"],
    workspace_name=os.environ["AZURE_ML_WORKSPACE"]
)

# List existing datastores
print("=== Available Datastores ===")
for ds in ml_client.datastores.list():
    print(f"  {ds.name} ({ds.type}) - {'Default' if ds.is_default else 'Custom'}")

# Create a datastore pointing to an external Blob Storage
def create_blob_datastore(
    name: str,
    account_name: str,
    container_name: str,
    account_key: str = None
) -> AzureBlobDatastore:
    """Connects an external Azure Blob Storage to the workspace."""
    
    datastore = AzureBlobDatastore(
        name=name,
        description=f"Connection to Blob Storage {account_name}",
        account_name=account_name,
        container_name=container_name,
        credentials={
            "account_key": account_key
        } if account_key else None
    )
    
    created_ds = ml_client.datastores.create_or_update(datastore)
    print(f"✅ Datastore created: {created_ds.name}")
    return created_ds

# Create a datastore pointing to Azure Data Lake Gen2
def create_adls_datastore(
    name: str,
    account_name: str,
    filesystem: str
) -> AzureDataLakeGen2Datastore:
    """Connects an Azure Data Lake Storage Gen2 to the workspace."""
    
    datastore = AzureDataLakeGen2Datastore(
        name=name,
        description=f"Connection to ADLS {account_name}",
        account_name=account_name,
        filesystem=filesystem
    )
    
    return ml_client.datastores.create_or_update(datastore)

3. Creating and Configuring a Workspace

3.1 Via the Azure Portal (Step-by-Step Guide)

1. Azure Portal → Search for "Azure Machine Learning"
2. Click "+ Create"
3. Fill in the form:
   - Subscription: Your subscription
   - Resource Group: new or existing (e.g. rg-ml-project)
   - Workspace Name: GLOBALLY UNIQUE (e.g. ws-ml-bank-prod)
   - Region: East US or West Europe (VM availability)
   - Storage Account: Leave default or select existing
   - Key Vault: Leave default or select existing
   - Application Insights: Recommended for prod
   - Container Registry: Optional (created if custom environments)
4. "Networking" tab:
   - Public access: OK for dev/test
   - Private endpoint: Recommended for production
5. Review + Create → Create
6. After ~2-3 minutes → "Launch Studio"

3.2 Via the Python SDK

# Create an Azure ML workspace via the SDK
from azure.ai.ml.entities import Workspace
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential
import os

# Management client (without specific workspace)
mgmt_client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id=os.environ["AZURE_SUBSCRIPTION_ID"],
    resource_group_name=os.environ["AZURE_RESOURCE_GROUP"]
)

# Workspace configuration
workspace_config = Workspace(
    name="ws-ml-bank-prod",
    description="ML Workspace for the bank marketing project",
    location="eastus",
    
    # Existing resources (or let them be auto-created)
    storage_account="/subscriptions/.../storageAccounts/my-storage",
    key_vault="/subscriptions/.../vaults/my-keyvault",
    
    tags={
        "project": "bank-marketing",
        "environment": "production",
        "owner": "ml-team",
        "cost_center": "ML-001"
    }
)

# Create the workspace
workspace_created = mgmt_client.workspaces.begin_create(workspace_config).result()
print(f"✅ Workspace created: {workspace_created.name}")
print(f"   ID: {workspace_created.id}")
print(f"   Studio URL: https://ml.azure.com/?wsid={workspace_created.id}")

3.3 Via Infrastructure as Code (Bicep)

// workspace.bicep - Infrastructure as Code for Azure ML Workspace
@description('Azure ML workspace name')
param workspaceName string

@description('Azure region')
param location string = resourceGroup().location

@description('Governance tags')
param tags object = {
  project: 'ml-project'
  environment: 'production'
}

// Storage Account for the workspace
resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: '${toLower(workspaceName)}storage'
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'StorageV2'
  tags: tags
  properties: {
    accessTier: 'Hot'
    allowBlobPublicAccess: false
    minimumTlsVersion: 'TLS1_2'
  }
}

// Key Vault for secrets
resource keyVault 'Microsoft.KeyVault/vaults@2023-02-01' = {
  name: '${workspaceName}-kv'
  location: location
  tags: tags
  properties: {
    tenantId: subscription().tenantId
    sku: {
      family: 'A'
      name: 'standard'
    }
    accessPolicies: []
    enableSoftDelete: true
  }
}

// Azure ML Workspace
resource mlWorkspace 'Microsoft.MachineLearningServices/workspaces@2023-06-01-preview' = {
  name: workspaceName
  location: location
  tags: tags
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    storageAccount: storageAccount.id
    keyVault: keyVault.id
    description: 'ML Workspace deployed via Bicep'
  }
}

output workspaceId string = mlWorkspace.id
output workspaceName string = mlWorkspace.name

4. Compute – Compute Infrastructure

4.1 Decision Tree for Compute

flowchart TD
    Q["Which compute\nto choose?"] --> Q1{"Type of\nwork?"}
    
    Q1 -->|"Interactive\ndevelopment"| CI["Compute Instance\n\n• Single VM\n• Jupyter Notebooks\n• Step-by-step debug\n• Configurable auto-stop"]
    
    Q1 -->|"ML Training\n/ Pipeline"| CC["Compute Cluster\n\n• 0 to N nodes\n• Batch jobs\n• Auto-scale\n• Team-sharable"]
    
    Q1 -->|"Real-time\nDeployment"| MOE["Managed Online Endpoint\n\n• Always available\n• Auto-scaling\n• Guaranteed SLA"]
    
    Q1 -->|"Batch\nDeployment"| MBE["Managed Batch Endpoint\n\n• On demand\n• Large volumes\n• Cost-effective"]
    
    Q1 -->|"Big Data\nSpark"| SP["Serverless Spark\n\n• No provisioning\n• Pay-per-use\n• Native integration"]
    
    CI --> CI_USE["Use case: Exploration, initial\nexperimentation, debugging"]
    CC --> CC_USE["Use case: AutoML, pipelines,\nintensive training"]

4.2 Creating and Managing Compute

# Full Azure ML Compute management
from azure.ai.ml.entities import AmlCompute, ComputeInstance, IdentityConfiguration
from azure.ai.ml.constants import ManagedServiceIdentityType
import os

# Create an optimized Compute Cluster
def create_optimized_cluster(
    name: str,
    vm_type: str = "Standard_DS3_v2",
    min_instances: int = 0,
    max_instances: int = 4,
    tier: str = "Dedicated"
) -> AmlCompute:
    """
    Creates a Compute Cluster with optimized parameters.
    
    Args:
        name: Unique cluster name
        vm_type: VM size (impacts cost + performance)
        min_instances: Min nodes (0 = scale to zero)
        max_instances: Max nodes (automatic scaling)
        tier: "Dedicated" (guaranteed) or "LowPriority" (preemptible, ~70% cheaper)
    """
    cluster = AmlCompute(
        name=name,
        type="amlcompute",
        size=vm_type,
        min_instances=min_instances,
        max_instances=max_instances,
        idle_time_before_scale_down=120,  # Scale down after 2 min idle
        tier=tier,
        identity=IdentityConfiguration(
            type=ManagedServiceIdentityType.SYSTEM_ASSIGNED
        )
    )
    
    result = ml_client.compute.begin_create_or_update(cluster).result()
    
    print(f"✅ Cluster created: {result.name}")
    print(f"   VM: {vm_type}, Min: {min_instances}, Max: {max_instances}")
    print(f"   Tier: {tier}")
    
    return result

# Create a Compute Instance for development
def create_dev_instance(
    name: str,
    vm_type: str = "Standard_DS3_v2",
    idle_shutdown_minutes: int = 30
) -> ComputeInstance:
    """
    Creates a Compute Instance for interactive development.
    
    Args:
        name: Unique instance name
        vm_type: VM size
        idle_shutdown_minutes: Minutes before automatic shutdown
    """
    instance = ComputeInstance(
        name=name,
        size=vm_type,
        idle_time_before_shutdown=f"PT{idle_shutdown_minutes}M"
    )
    
    result = ml_client.compute.begin_create_or_update(instance).result()
    print(f"✅ Instance created: {result.name} ({vm_type})")
    print(f"   Auto-shutdown after {idle_shutdown_minutes} min idle")
    return result

# Compute usage and cost dashboard
def compute_report() -> dict:
    """Generates a report of compute usage."""
    report = {
        "instances": [],
        "clusters": []
    }
    
    for compute in ml_client.compute.list():
        info = {
            "name": compute.name,
            "type": compute.type,
            "vm_size": getattr(compute, 'size', 'N/A'),
            "state": compute.provisioning_state
        }
        
        if compute.type == "ComputeInstance":
            info["vm_status"] = getattr(compute, 'state', 'N/A')
            report["instances"].append(info)
        elif compute.type == "AmlCompute":
            info["min_instances"] = getattr(compute, 'min_instances', 0)
            info["max_instances"] = getattr(compute, 'max_instances', 0)
            report["clusters"].append(info)
    
    return report

# Create compute resources
cluster_cpu = create_optimized_cluster(
    name="cluster-cpu-standard",
    vm_type="Standard_DS3_v2",
    min_instances=0,
    max_instances=4
)

cluster_gpu = create_optimized_cluster(
    name="cluster-gpu-v100",
    vm_type="Standard_NC6s_v3",
    min_instances=0,
    max_instances=2,
    tier="LowPriority"  # Cost-effective for non-critical training
)

dev_instance = create_dev_instance(
    name="dev-instance-ds3",
    idle_shutdown_minutes=30
)

print("\n=== Compute Report ===")
import json
report = compute_report()
print(json.dumps(report, indent=2))

4.3 Selecting the VM Based on Need

# Azure VM selection guide
def recommend_vm(
    workload_type: str,
    data_size_gb: float,
    uses_deep_learning: bool = False,
    budget_per_hour_usd: float = None
) -> dict:
    """
    Recommends an Azure ML VM based on context.
    
    Args:
        workload_type: "dev", "training_ml", "training_dl", "inference"
        data_size_gb: Data size in GB
        uses_deep_learning: If True, recommends a GPU
        budget_per_hour_usd: Max budget in USD/hour
    
    Returns:
        Recommendation with justification
    """
    
    vms = {
        "dev": {
            "Standard_D1_v2": {"vcpu": 1, "ram_gb": 3.5, "gpu": False, "cost": 0.05},
            "Standard_DS2_v2": {"vcpu": 2, "ram_gb": 7, "gpu": False, "cost": 0.14},
            "Standard_DS3_v2": {"vcpu": 4, "ram_gb": 14, "gpu": False, "cost": 0.25}
        },
        "training_ml": {
            "Standard_DS3_v2": {"vcpu": 4, "ram_gb": 14, "gpu": False, "cost": 0.25},
            "Standard_DS4_v2": {"vcpu": 8, "ram_gb": 28, "gpu": False, "cost": 0.50},
            "Standard_DS5_v2": {"vcpu": 16, "ram_gb": 56, "gpu": False, "cost": 1.0},
            "Standard_E8s_v3": {"vcpu": 8, "ram_gb": 64, "gpu": False, "cost": 0.62}
        },
        "training_dl": {
            "Standard_NC6s_v3": {"vcpu": 6, "ram_gb": 112, "gpu": True, "cost": 3.0},
            "Standard_NC12s_v3": {"vcpu": 12, "ram_gb": 224, "gpu": True, "cost": 6.0},
            "Standard_ND40rs_v2": {"vcpu": 40, "ram_gb": 672, "gpu": True, "cost": 22.0}
        },
        "inference": {
            "Standard_F2s_v2": {"vcpu": 2, "ram_gb": 4, "gpu": False, "cost": 0.10},
            "Standard_F4s_v2": {"vcpu": 4, "ram_gb": 8, "gpu": False, "cost": 0.20},
            "Standard_F8s_v2": {"vcpu": 8, "ram_gb": 16, "gpu": False, "cost": 0.39}
        }
    }
    
    if uses_deep_learning:
        workload_type = "training_dl"
    
    pool = vms.get(workload_type, vms["training_ml"])
    
    # Filter by budget if specified
    if budget_per_hour_usd:
        pool = {k: v for k, v in pool.items() if v["cost"] <= budget_per_hour_usd}
    
    if not pool:
        return {"error": f"No VM available for budget {budget_per_hour_usd}/h"}
    
    # Recommend based on data size
    if data_size_gb < 10:
        vm_rec = min(pool, key=lambda k: pool[k]["cost"])  # Cheapest
    elif data_size_gb < 100:
        vm_rec = sorted(pool, key=lambda k: pool[k]["cost"])[len(pool)//2]  # Median
    else:
        vm_rec = max(pool, key=lambda k: pool[k]["ram_gb"])  # Most RAM
    
    return {
        "recommended_vm": vm_rec,
        "specifications": pool[vm_rec],
        "justification": f"Data: {data_size_gb}GB, Workload: {workload_type}",
        "cost_per_hour_usd": pool[vm_rec]["cost"],
        "estimated_monthly_cost_usd": round(pool[vm_rec]["cost"] * 8 * 22, 0)  # 8h/day, 22 days/month
    }

# Test scenarios
scenarios = [
    ("dev", 1, False, None),
    ("training_ml", 50, False, 0.6),
    ("training_dl", 200, True, None),
    ("inference", 0, False, 0.25)
]

for workload, data, dl, budget in scenarios:
    reco = recommend_vm(workload, data, dl, budget)
    print(f"\n{workload.upper()} ({data}GB):")
    print(f"  → {reco.get('recommended_vm', 'N/A')}: {reco.get('cost_per_hour_usd', 0):.2f}$/h")
    print(f"  Specs: {reco.get('specifications', {})}")

5. Data and Datasets

5.1 Data Lifecycle in Azure ML

flowchart LR
    RAW["📁 Raw Data\n(Local, S3, ADLS...)"] --> UPLOAD["Upload to\nAzure Storage"]
    UPLOAD --> REGISTER["📋 Register\nas Data Asset"]
    REGISTER --> VERSION["🔢 Version\n(v1, v2, v3...)"]
    VERSION --> USE["🔗 Use in\njobs/pipelines"]
    USE --> LINEAGE["🕵️ Traceability\n(which job → which dataset)"]

5.2 Creating and Managing Data Assets

# Complete Data Asset management
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes
import os
import pandas as pd
from pathlib import Path

def create_dataset_from_local_file(
    file_path: str,
    dataset_name: str,
    description: str,
    tags: dict = None
) -> Data:
    """
    Creates a Data Asset from a local CSV file.
    
    The file is automatically uploaded to the workspace Blob Storage.
    """
    if not os.path.exists(file_path):
        raise FileNotFoundError(f"File not found: {file_path}")
    
    # Validate the CSV file
    try:
        df_test = pd.read_csv(file_path, nrows=5)
        print(f"File preview: {df_test.shape} rows×columns (first 5 rows)")
    except Exception as e:
        raise ValueError(f"Invalid CSV file: {e}")
    
    data_asset = Data(
        path=file_path,
        type=AssetTypes.URI_FILE,
        name=dataset_name,
        description=description,
        tags=tags or {
            "source": "local_upload",
            "format": "csv"
        }
    )
    
    asset_created = ml_client.data.create_or_update(data_asset)
    
    print(f"✅ Data Asset created: {asset_created.name} v{asset_created.version}")
    print(f"   Azure Path: {asset_created.path}")
    
    return asset_created

def create_dataset_from_blob(
    blob_path: str,  # "azureml://datastores/my-datastore/paths/data/train.csv"
    dataset_name: str,
    description: str
) -> Data:
    """Creates a Data Asset from Azure Blob Storage."""
    
    data_asset = Data(
        path=blob_path,
        type=AssetTypes.URI_FILE,
        name=dataset_name,
        description=description
    )
    
    return ml_client.data.create_or_update(data_asset)

def create_folder_dataset(
    folder_path: str,  # Folder with multiple files
    dataset_name: str,
    description: str
) -> Data:
    """Creates a Data Asset from a full folder."""
    
    data_asset = Data(
        path=folder_path,
        type=AssetTypes.URI_FOLDER,
        name=dataset_name,
        description=description
    )
    
    return ml_client.data.create_or_update(data_asset)

def version_updated_dataset(
    dataset_name: str,
    new_path: str,
    change_description: str
) -> Data:
    """
    Creates a new version of an existing dataset.
    
    Useful for traceability: "Which version of data was this model trained on?"
    """
    # Get current version
    try:
        current_asset = ml_client.data.get(name=dataset_name, label="latest")
        current_version = int(current_asset.version)
        new_version = str(current_version + 1)
    except Exception:
        new_version = "1"
    
    data_asset = Data(
        path=new_path,
        type=AssetTypes.URI_FILE,
        name=dataset_name,
        version=new_version,
        description=f"v{new_version}: {change_description}"
    )
    
    asset_created = ml_client.data.create_or_update(data_asset)
    print(f"✅ New version: {dataset_name} v{new_version}")
    print(f"   Changes: {change_description}")
    
    return asset_created

def dataset_inventory() -> pd.DataFrame:
    """Generates an inventory of all Data Assets in the workspace."""
    assets = []
    
    for asset in ml_client.data.list():
        assets.append({
            "name": asset.name,
            "version": asset.version,
            "type": asset.type,
            "description": (asset.description or "")[:80],
            "tags": asset.tags or {}
        })
    
    df = pd.DataFrame(assets)
    
    print(f"\n=== Data Asset Inventory ({len(df)} assets) ===")
    print(df.to_string(index=False))
    
    return df

# Usage examples
create_dataset_from_local_file(
    file_path="./data/bank_marketing.csv",
    dataset_name="bank-marketing",
    description="Bank marketing dataset for binary classification",
    tags={"source": "UCI Repository", "task": "classification"}
)

inventory = dataset_inventory()

6. Environments – Reproducibility

6.1 Why Environments Are Critical

flowchart TD
    subgraph "Without Azure ML Environment ❌"
        E1["Run 1\nsklearn 1.2.0\npandas 1.5.0"]
        E2["Run 2\nsklearn 1.3.0\npandas 2.0.0"]
        E3["Run 3\nsklearn 1.1.0\npandas 1.5.0"]
        E1 --> RESULT1["Result X"]
        E2 --> RESULT2["Result Y ≠ X!"]
        E3 --> RESULT3["Result Z ≠ X!"]
    end
    
    subgraph "With Azure ML Environment ✅"
        ENV["Environment 'ml-env-v1'\nsklearn 1.3.0\npandas 2.0.0"]
        R1["Run 1"] --> ENV
        R2["Run 2"] --> ENV
        R3["Run 3"] --> ENV
        ENV --> STABLE["Reproducible results\n(always X)"]
    end

6.2 Creating Custom Environments

# Create Azure ML environments
from azure.ai.ml.entities import Environment, BuildContext
import os

# Option 1: From a Conda YAML file
def create_env_from_conda(name: str, extra_packages: list = None) -> Environment:
    """Creates a complete Python environment from conda."""
    
    packages = [
        "azure-ai-ml>=1.12.0",
        "scikit-learn>=1.3.0",
        "pandas>=2.0.0",
        "numpy>=1.26.0",
        "mlflow>=2.9.0",
        "joblib>=1.3.0"
    ] + (extra_packages or [])
    
    conda_content = f"""
name: {name}
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - pip:
{chr(10).join(f"    - {pkg}" for pkg in packages)}
"""
    
    conda_file = f"{name}_conda.yml"
    with open(conda_file, "w") as f:
        f.write(conda_content)
    
    env = Environment(
        name=name,
        description=f"ML environment {name} with scikit-learn and MLflow",
        conda_file=conda_file,
        image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest",
        tags={"python": "3.10", "type": "conda"}
    )
    
    env_created = ml_client.environments.create_or_update(env)
    print(f"✅ Environment created: {env_created.name}:{env_created.version}")
    
    # Clean up temp file
    os.remove(conda_file)
    
    return env_created

# Option 2: From a Dockerfile
def create_env_from_dockerfile(name: str) -> Environment:
    """Creates an environment from a custom Dockerfile."""
    
    dockerfile = """FROM mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest

# Install system dependencies (e.g. ODBC for SQL Server)
RUN apt-get update && apt-get install -y \\
    unixodbc-dev \\
    && rm -rf /var/lib/apt/lists/*

# Install Python packages
RUN pip install --no-cache-dir \\
    scikit-learn>=1.3.0 \\
    pandas>=2.0.0 \\
    pyodbc>=5.0.0 \\
    mlflow>=2.9.0 \\
    azure-ai-ml>=1.12.0

# Environment variables
ENV PYTHONUNBUFFERED=1
"""
    
    os.makedirs("./docker_env", exist_ok=True)
    with open("./docker_env/Dockerfile", "w") as f:
        f.write(dockerfile)
    
    env = Environment(
        name=name,
        description="Environment with ODBC drivers for SQL connection",
        build=BuildContext(path="./docker_env"),
        tags={"type": "docker", "has_odbc": "true"}
    )
    
    return ml_client.environments.create_or_update(env)

# Option 3: Microsoft curated environments (recommended for most use cases)
MICROSOFT_ENVS = {
    "sklearn": "AzureML-sklearn-1.0-ubuntu20.04-py38-cpu@latest",
    "pytorch": "AzureML-pytorch-2.0-ubuntu20.04-py38-cuda11.8-gpu@latest",
    "tensorflow": "AzureML-tensorflow-2.13-ubuntu20.04-py38-cuda11.8-gpu@latest",
    "spark": "AzureML-spark3.2-ubuntu18.04-py37-cpu@latest",
    "inference": "AzureML-minimal-ubuntu20.04-py38-inference@latest"
}

# Create common environments
env_ml = create_env_from_conda(
    name="ml-env-standard",
    extra_packages=["xgboost>=2.0.0", "lightgbm>=4.0.0", "shap>=0.44.0"]
)

env_nlp = create_env_from_conda(
    name="nlp-env",
    extra_packages=["transformers>=4.30.0", "torch>=2.0.0", "datasets>=2.0.0"]
)

7. Governance and Lifecycle

7.1 Tags and Metadata

# Tag policy for Azure ML governance
TAG_POLICY = {
    # Mandatory for all prod workspaces
    "owner": "owner-name",             # Who is responsible?
    "project": "project-name",         # Which project does this workspace belong to?
    "environment": "dev|staging|prod", # Which environment?
    "cost_center": "CC-001",           # Cost center for billing
    
    # Recommended
    "data_classification": "public|internal|confidential",
    "auto_shutdown": "true|false",
    "team": "team-name"
}

def validate_workspace_tags(workspace) -> list[str]:
    """Checks that mandatory tags are present."""
    current_tags = workspace.tags or {}
    missing_tags = []
    
    for required_tag in ["owner", "project", "environment", "cost_center"]:
        if required_tag not in current_tags:
            missing_tags.append(required_tag)
    
    if missing_tags:
        print(f"⚠️ Missing mandatory tags: {missing_tags}")
    else:
        print("✅ All mandatory tags are present")
    
    return missing_tags

7.2 Housekeeping – Periodic Cleanup

# Periodic workspace cleanup scripts
from datetime import datetime, timedelta

def audit_workspace_resources() -> dict:
    """
    Generates a workspace audit report with recommendations.
    """
    report = {
        "audit_date": datetime.now().isoformat(),
        "compute": {"stopped_instances": [], "unused_clusters": []},
        "models": {"stale_versions": []},
        "datasets": {"referenced_datasets": set(), "unused_datasets": []},
        "recommendations": []
    }
    
    # Check Compute instances stopped for a long time
    for compute in ml_client.compute.list():
        if compute.type == "ComputeInstance":
            if hasattr(compute, 'state') and compute.state == "Stopped":
                report["compute"]["stopped_instances"].append(compute.name)
                report["recommendations"].append(
                    f"💡 Instance '{compute.name}' is stopped - Consider deleting it"
                )
    
    # List models with more than 5 versions
    models_with_versions = {}
    for model in ml_client.models.list():
        if model.name not in models_with_versions:
            models_with_versions[model.name] = []
        models_with_versions[model.name].append(model.version)
    
    for name, versions in models_with_versions.items():
        if len(versions) > 5:
            report["recommendations"].append(
                f"💡 Model '{name}' has {len(versions)} versions - Archive old ones"
            )
    
    return report

def generate_estimated_cost_report() -> dict:
    """Estimates monthly workspace costs."""
    costs = {
        "compute_instances": 0.0,
        "active_compute_clusters": 0.0,
        "storage_gb": 0.0,
        "active_endpoints": 0
    }
    
    vm_prices = {
        "Standard_DS1_v2": 0.058,
        "Standard_DS2_v2": 0.141,
        "Standard_DS3_v2": 0.282,
        "Standard_DS4_v2": 0.564,
        "Standard_NC6s_v3": 3.06
    }
    
    for compute in ml_client.compute.list():
        if compute.type == "ComputeInstance":
            vm_size = getattr(compute, 'size', 'Standard_DS2_v2')
            price_per_hour = vm_prices.get(vm_size, 0.28)
            # Assume 8h/day, 22 days/month
            costs["compute_instances"] += price_per_hour * 8 * 22
    
    for endpoint in ml_client.online_endpoints.list():
        costs["active_endpoints"] += 1
    
    return {
        "estimated_monthly_costs": costs,
        "total_usd": sum(costs.values()),
        "note": "Estimate based on 8h/day instance usage"
    }

print("=== Workspace Audit ===")
audit = audit_workspace_resources()
import json
print(json.dumps(audit["recommendations"], indent=2))

print("\n=== Cost Estimate ===")
costs = generate_estimated_cost_report()
print(json.dumps(costs, indent=2))

8. Workspace Security

8.1 RBAC – Access Control

flowchart TD
    WORKSPACE["Azure ML Workspace"] --> ROLES["Azure ML Roles"]
    
    ROLES --> DS_ROLE["AzureML Data Scientist\n• Train models\n• Submit jobs\n• Read metrics\n• Deploy models"]
    
    ROLES --> OPS_ROLE["AzureML Compute Operator\n• Create/manage compute\n• No data access"]
    
    ROLES --> READER_ROLE["AzureML Reader\n• View only\n• No modifications"]
    
    ROLES --> ADMIN_ROLE["Contributor\n• Full access\n• For ML admins"]

8.2 Network Isolation (Production)

# Recommended configuration for a production workspace
from azure.ai.ml.entities import Workspace, ManagedNetwork, IsolationMode

# Workspace with complete network isolation
secure_workspace = Workspace(
    name="ws-ml-secure-prod",
    description="Secured ML Workspace for production",
    location="eastus",
    
    # Isolated network (no unauthorized outbound internet access)
    managed_network=ManagedNetwork(
        isolation_mode=IsolationMode.ALLOW_ONLY_APPROVED_OUTBOUND
    ),
    
    tags={
        "security_tier": "high",
        "data_classification": "confidential",
        "environment": "production"
    }
)

9. Practical Implementation with the SDK

9.1 Complete Workspace Setup Script

# setup_workspace.py - Full workspace initialization
# Usage: python setup_workspace.py --project my-project --env prod

import argparse
from azure.ai.ml import MLClient
from azure.ai.ml.entities import AmlCompute, Environment, Data
from azure.ai.ml.constants import AssetTypes
from azure.identity import DefaultAzureCredential
import os

def setup_complete_workspace(
    subscription_id: str,
    resource_group: str,
    workspace_name: str,
    project: str,
    environment: str
) -> dict:
    """
    Configures an Azure ML workspace end-to-end.
    
    Creates:
    - 2 Compute Clusters (CPU + GPU)
    - 1 Compute Instance for development
    - 3 Python Environments (ML, NLP, Inference)
    - Folder structure in Blob Storage
    
    Returns:
        Dict with created resources
    """
    
    print(f"\n{'='*60}")
    print(f"SETUP WORKSPACE: {workspace_name}")
    print(f"Project: {project} | Env: {environment}")
    print(f"{'='*60}\n")
    
    # Connection
    ml_client = MLClient(
        credential=DefaultAzureCredential(),
        subscription_id=subscription_id,
        resource_group_name=resource_group,
        workspace_name=workspace_name
    )
    
    created_resources = {}
    
    # === COMPUTE ===
    print("1️⃣ Creating Compute...")
    
    cluster_cpu = AmlCompute(
        name=f"cluster-cpu-{project[:8]}",
        type="amlcompute",
        size="Standard_DS3_v2",
        min_instances=0,
        max_instances=6,
        idle_time_before_scale_down=120
    )
    created_resources["cluster_cpu"] = ml_client.compute.begin_create_or_update(cluster_cpu).result().name
    print(f"  ✅ CPU Cluster: {created_resources['cluster_cpu']}")
    
    cluster_gpu = AmlCompute(
        name=f"cluster-gpu-{project[:8]}",
        type="amlcompute",
        size="Standard_NC6s_v3",
        min_instances=0,
        max_instances=2,
        idle_time_before_scale_down=60,
        tier="LowPriority"
    )
    created_resources["cluster_gpu"] = ml_client.compute.begin_create_or_update(cluster_gpu).result().name
    print(f"  ✅ GPU Cluster: {created_resources['cluster_gpu']}")
    
    # === ENVIRONMENTS ===
    print("\n2️⃣ Creating Environments...")
    
    envs = [
        {
            "name": f"env-ml-{project[:8]}",
            "packages": ["scikit-learn>=1.3.0", "xgboost>=2.0.0", "mlflow>=2.9.0"]
        },
        {
            "name": f"env-nlp-{project[:8]}",
            "packages": ["transformers>=4.30.0", "mlflow>=2.9.0"]
        }
    ]
    
    created_resources["environments"] = []
    for env_config in envs:
        conda_content = f"""
name: {env_config['name']}
channels:
  - conda-forge
dependencies:
  - python=3.10
  - pip:
{chr(10).join(f"    - {pkg}" for pkg in env_config['packages'])}
"""
        with open(f"/tmp/{env_config['name']}.yml", "w") as f:
            f.write(conda_content)
        
        env = Environment(
            name=env_config["name"],
            conda_file=f"/tmp/{env_config['name']}.yml",
            image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest"
        )
        ml_client.environments.create_or_update(env)
        created_resources["environments"].append(env_config["name"])
        print(f"  ✅ Environment: {env_config['name']}")
    
    print(f"\n✅ Setup complete! Workspace ready for: {project}")
    print(f"   Studio URL: https://ml.azure.com/?wsid=/subscriptions/{subscription_id}/...")
    
    return created_resources

# Entry point
if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Setup Azure ML Workspace")
    parser.add_argument("--subscription_id", type=str, required=True)
    parser.add_argument("--resource_group", type=str, required=True)
    parser.add_argument("--workspace", type=str, required=True)
    parser.add_argument("--project", type=str, required=True)
    parser.add_argument("--env", type=str, default="dev")
    args = parser.parse_args()
    
    setup_complete_workspace(
        subscription_id=args.subscription_id,
        resource_group=args.resource_group,
        workspace_name=args.workspace,
        project=args.project,
        environment=args.env
    )

10. Cost Optimization

10.1 Cost-Saving Strategies

flowchart TD
    COST["💰 Reduce\nAzure ML Costs"] --> SCALE["Scale to Zero\n(min_instances=0)\n→ No charge when\ncluster is idle"]
    COST --> LOW_PRIO["Low Priority VMs\n(60-80% savings)\n→ For non-critical jobs\n→ Acceptable if preemption OK"]
    COST --> SPOT["Auto-shutdown\nof instances\n(idle_time_before_shutdown)\n→ Avoid forgotten VMs"]
    COST --> RIGHTSIZE["Right-sizing\n(choose the right VM)\n→ No over-provisioning"]
    COST --> CACHE["Caching intermediate\nresults\n→ Reuse unchanged\npipeline steps"]

10.2 Cost Estimator

# Azure ML cost estimator
from dataclasses import dataclass

@dataclass
class CostScenario:
    """Azure ML cost calculation scenario."""
    
    project_name: str
    
    # Compute
    num_dev_instances: int = 1
    vm_dev: str = "Standard_DS3_v2"
    dev_hours_per_day: int = 6
    
    num_training_nodes: int = 4
    vm_training: str = "Standard_DS3_v2"
    training_hours_per_week: int = 20
    
    # Storage
    data_gb: float = 100
    models_gb: float = 10
    
    # Endpoints
    num_prod_endpoints: int = 1
    vm_endpoint: str = "Standard_DS2_v2"
    
    # USD/hour prices (approximate)
    VM_PRICES = {
        "Standard_DS1_v2": 0.058,
        "Standard_DS2_v2": 0.141,
        "Standard_DS3_v2": 0.282,
        "Standard_DS4_v2": 0.564,
        "Standard_NC6s_v3": 3.06
    }
    
    STORAGE_GB_PRICE = 0.024  # Per month
    
    def calculate(self) -> dict:
        """Calculates estimated monthly cost."""
        
        # Dev (22 days/month)
        dev_cost = (
            self.num_dev_instances
            * self.VM_PRICES.get(self.vm_dev, 0.28)
            * self.dev_hours_per_day
            * 22
        )
        
        # Training (4 weeks/month)
        training_cost = (
            self.num_training_nodes
            * self.VM_PRICES.get(self.vm_training, 0.28)
            * self.training_hours_per_week
            * 4
        )
        
        # Storage
        storage_cost = (self.data_gb + self.models_gb) * self.STORAGE_GB_PRICE
        
        # Endpoints
        endpoint_cost = (
            self.num_prod_endpoints
            * self.VM_PRICES.get(self.vm_endpoint, 0.14)
            * 24  # 24h/24 for production endpoint
            * 30
        )
        
        total = dev_cost + training_cost + storage_cost + endpoint_cost
        
        return {
            "project": self.project_name,
            "monthly_costs_usd": {
                "development": round(dev_cost, 2),
                "training": round(training_cost, 2),
                "storage": round(storage_cost, 2),
                "endpoints": round(endpoint_cost, 2),
                "total": round(total, 2)
            },
            "annual_costs_usd": round(total * 12, 2)
        }

# Examples
scenarios = [
    CostScenario("startup-ml", num_dev_instances=1, num_training_nodes=2, data_gb=10),
    CostScenario("ml-team-10", num_dev_instances=5, num_training_nodes=8, data_gb=500, num_prod_endpoints=3),
]

for scenario in scenarios:
    cost = scenario.calculate()
    print(f"\n=== {cost['project']} ===")
    for item, value in cost["monthly_costs_usd"].items():
        if item != "total":
            print(f"  {item:20}: ${value:.2f}/month")
    print(f"  {'TOTAL':20}: ${cost['monthly_costs_usd']['total']:.2f}/month")

11. Summary and Key Points

11.1 Reference Architecture

flowchart TB
    subgraph "Azure Subscription"
        subgraph "Resource Group: rg-ml-project"
            WS["🏢 Azure ML Workspace"]
            STORAGE["💾 Storage Account\n(Datasets, Models, Logs)"]
            KV["🔑 Key Vault\n(Secrets)"]
            ACR["🐳 Container Registry\n(Custom Images)"]
            AI["📊 Application Insights\n(Monitoring)"]
            
            subgraph "Compute Resources"
                CC["⚙️ Compute Clusters\n(CPU + GPU)"]
                CI["💻 Compute Instances\n(Dev)"]
                EP["🌐 Online Endpoints\n(Production)"]
            end
        end
    end
    
    WS --> STORAGE
    WS --> KV
    WS --> ACR
    WS --> AI
    WS --> CC
    WS --> CI
    WS --> EP

11.2 Resource Summary Table

12. Glossary

Additional Resources:

Table of Contents

1. Azure ML Workspace Overview
   • What is an Azure ML Workspace?
   • Architecture and Dependencies
   • Create a Workspace
2. Compute – Compute Infrastructure
   • Compute Types
   • Compute Instance vs Compute Cluster
3. Data and Datasets
   • Create a Data Asset
   • Dataset Types
4. Environments – Reproducible Environments
   • What is an Environment?
   • Create a Custom Environment
5. Governance and Lifecycle
6. Summary and Key Points

1. Azure ML Workspace Overview

1.1 What is an Azure ML Workspace?

The Azure ML Workspace is the central hub for all machine learning activities in Azure. It is the object that groups and organizes all ML resources:

Azure ML Workspace
├── Compute (instances, clusters, inference)
├── Data Assets (versioned datasets)
├── Environments (reproducible dependencies)
├── Experiments & Jobs (training runs)
├── Models (versioned model registry)
├── Endpoints (REST deployments)
└── Pipelines (automated workflows)

Why centralize?

• Avoid dispersing ML resources across different Azure accounts
• Facilitate collaboration between Data Science teams
• Track all experiments, metrics, and artifacts
• Manage costs through a single resource

1.2 Architecture and Dependencies

When creating an Azure ML workspace, 4 Azure resources are created automatically:

graph TD
    W[Azure ML Workspace] --> SA[Azure Storage Account]
    W --> KV[Azure Key Vault]
    W --> ACR[Azure Container Registry]
    W --> AI[Application Insights]

    SA -->|Stores data, models, artifacts| W
    KV -->|Stores secrets, credentials| W
    ACR -->|Stores Docker images for environments| W
    AI -->|Monitoring, telemetry, logs| W

1.3 Create a Workspace

Via Azure Portal:

Portal → Machine Learning → Create
→ Subscription → Resource Group → Workspace Name → Region
→ Review + Create

Via Azure CLI:

az ml workspace create \
  --name my-ml-workspace \
  --resource-group my-rg \
  --location eastus

Via Python SDK:

from azure.ai.ml import MLClient
from azure.ai.ml.entities import Workspace
from azure.identity import DefaultAzureCredential

ml_client = MLClient(DefaultAzureCredential())

workspace = Workspace(
    name="my-ml-workspace",
    location="eastus",
    resource_group="my-rg"
)

workspace = ml_client.workspaces.begin_create(workspace).result()
print(f"Workspace created: {workspace.name}")

2. Compute – Compute Infrastructure

2.1 Compute Types

Azure ML offers 3 types of compute for different use cases:

2.2 Compute Instance vs Compute Cluster

Compute Instance:

Studio → Compute → Compute Instances → + New

Compute Cluster:

Studio → Compute → Compute Clusters → + New

CLI Example:

az ml compute create \
  --name my-cluster \
  --type AmlCompute \
  --min-instances 0 \
  --max-instances 4 \
  --size Standard_DS3_v2

3. Data and Datasets

3.1 Create a Data Asset

Via Studio:

Studio → Data → Create
→ Name → Type (File or Tabular) → Source → Version

Supported sources:

• Local files (direct upload)
• Azure Blob Storage
• Azure Data Lake Storage
• Web URL
• Azure SQL Database
• Azure Open Datasets

Via SDK:

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes

data_asset = Data(
    name="bank-marketing-dataset",
    path="./data/bank_marketing.csv",
    type=AssetTypes.URI_FILE,
    description="Dataset for bank marketing classification",
    version="1"
)

ml_client.data.create_or_update(data_asset)

3.2 Dataset Types

Versioning:

# Version 1 (base)
data_v1 = Data(name="my-dataset", version="1", path="data_v1.csv")

# Version 2 (enriched)
data_v2 = Data(name="my-dataset", version="2", path="data_v2.csv")

# Reference a specific version
dataset_ref = Input(type="uri_file", path="azureml:my-dataset:1")

4. Environments – Reproducible Environments

4.1 What is an Environment?

An Environment in Azure ML defines the software dependencies (Python packages, Docker image) needed to run code reproducibly.

Problem solved:

• Guarantee that the same code produces the same results over time
• Avoid dependency conflicts between teams
• Enable experiment reproducibility

Two types of environments:

1. Curated environments: provided by Microsoft (AzureML-sklearn, AzureML-pytorch, etc.)
2. Custom environments: defined by the user (Conda YAML or Dockerfile)

4.2 Create a Custom Environment

Via Conda YAML:

# conda.yml
name: sklearn-env
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.9
  - scikit-learn=1.0.2
  - pandas=1.4.0
  - numpy=1.22.0
  - pip:
    - mlflow
    - azureml-mlflow

Register via SDK:

from azure.ai.ml.entities import Environment

custom_env = Environment(
    name="sklearn-custom-env",
    conda_file="./conda.yml",
    image="mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04:latest",
    version="1"
)

ml_client.environments.create_or_update(custom_env)

Via Dockerfile:

custom_env = Environment(
    name="docker-custom-env",
    build=BuildContext(path="./docker-context"),
    version="1"
)

Via Studio:

Studio → Environments → Custom environments → + Create
→ Name → Image (Conda YAML or Dockerfile)

5. Governance and Lifecycle

Governance best practices:

1. CREATION          2. USAGE               3. CLEANUP
──────────           ────────────────        ──────────
• ARM templates      • Experiment freely     • Quarterly cleanup
• SDK export         • Version datasets      • Delete compute
• Naming conventions • Stable environments   • Archive old models
• Azure tags         • Log all runs          • Clean up Storage

Export configuration via ARM Template:

Portal → Workspace → Export template

Clean up unused resources:

# Stop a Compute Instance
ml_client.compute.begin_stop("my-instance").result()

# Delete a Compute Cluster
ml_client.compute.begin_delete("my-old-cluster").result()

6. Summary and Key Points

Key takeaways:

• The Workspace is the central hub for all ML resources
• Always use Scale to Zero on clusters to avoid unnecessary costs
• Environments guarantee reproducibility across runs
• Data Assets with versioning enable full experiment traceability
• Use RBAC to control team access
• In production: network isolation + Key Vault for secrets

Search Terms

azure · ml · workspace · fundamentals · platforms · deployment · machine · data · science · compute · environments · architecture · infrastructure · lifecycle · via · cost · custom · datasets · dependencies · environment · governance · managing · points · sdk