Data Science with Microsoft Fabric

4 modules

Table of Contents

1. Module 1 — Designing Data Science Solutions in Microsoft Fabric

   • 1.1 Microsoft Fabric Architecture and Components
   • 1.2 Benefits and Challenges of Fabric for Data Science
   • 1.3 Microsoft Fabric Capabilities
   • 1.4 Key Steps of the Data Science Lifecycle with Fabric
   • 1.5 Comparison with Other Data Science Platforms
   • 1.6 Ideal Use Cases for Microsoft Fabric
   • 1.7 Demo — MS Fabric Environment Setup

2. Module 2 — Building Data Science Workflows in Microsoft Fabric

   • 2.1 Data Preparation and Transformation for Machine Learning
   • 2.2 Model Training: Manual and Automated Approaches (AutoML)
   • 2.3 Experiment Tracking — Monitoring and Managing Models
   • 2.4 Batch Predictions with Fabric’s PREDICT Function
   • 2.5 Distributed Computing for Machine Learning in Fabric

3. Module 3 — Combining AI with Business Intelligence in Microsoft Fabric

   • 3.1 Semantic Links — Connecting Data Sources
   • 3.2 Integrating Azure AI Tools into Fabric Workflows
   • 3.3 Deriving Insights from AI-Based Techniques
   • 3.4 NLP and Text Analysis with Fabric
   • 3.5 AI Skills in Microsoft Fabric

4. Module 4 — Data Science Governance and Management in Microsoft Fabric

   • 4.1 Lineage Tracking — Reproducibility and Transparency
   • 4.2 Disaster Recovery Plan for ML Projects
   • 4.3 Security Management and Access Control for ML Assets
   • 4.4 Compliance and Security Best Practices in Fabric
   • 4.5 Demo — Roles, Permissions and Lineage Tracking

Module 1

Module 1 — Designing Data Science Solutions in Microsoft Fabric

Microsoft Fabric is an all-in-one platform for data science. It handles everything from storage to artificial intelligence. Think of a kitchen where the refrigerator, oven, and pantry are perfectly organized for any recipe:

• Fabric’s data lake = the refrigerator holding all your raw ingredients
• The integration layer = your sous-chef who prepares everything
• The Machine Learning layer = the chef creating the magic
• BI tools (like Power BI) = the service presenting the meal to guests

1.1 Microsoft Fabric Architecture and Components

Microsoft Fabric is a unified data platform bringing together all the tools needed for modern data science.

graph TB
    subgraph OneLake["OneLake — Unified Foundation"]
        WA["Workspace A"]
        WB["Workspace B"]
    end

    subgraph Engines["Serverless Engines"]
        TSQL["T-SQL Engine"]
        SPARK["Spark Engine<br/>(Python, R, Scala)"]
        KQL["KQL Engine"]
        AS["Analysis Services"]
    end

    subgraph Services["Fabric Services"]
        DF["Data Factory<br/>(Dataflow, Pipeline)"]
        DW["Data Warehouse<br/>(T-SQL)"]
        DE["Data Engineering<br/>(Lakehouse, Spark)"]
        DS["Data Science<br/>(Notebooks, Experiments, ML Models)"]
        RI["Real-time Intelligence<br/>(Eventstream, KQL)"]
        PBI["Power BI<br/>(Semantic Model, Reports)"]
        DA["Data Activator"]
    end

    subgraph Security["Security and Governance"]
        SEC["Security Layer<br/>Microsoft Purview"]
    end

    subgraph Sources["Data Sources"]
        AZ["Azure"]
        AWS["AWS / Google Cloud"]
        DV["Dataverse"]
        OTHER["Other sources"]
    end

    Sources --> OneLake
    OneLake --> Services
    Engines --> Services
    SEC --> OneLake

Core Components

Unified Platform Architecture

┌─────────────────────────────────────────────────────────────────┐
│                    Microsoft Fabric                              │
├──────────┬──────────┬──────────┬──────────┬──────────┬──────────┤
│ Data     │ Data     │ Data     │ Data     │Real-time │ Power BI │
│ Factory  │ Warehouse│ Engineer │ Science  │Intellig. │          │
├──────────┴──────────┴──────────┴──────────┴──────────┴──────────┤
│          Serverless Compute (T-SQL / Spark / KQL)               │
├─────────────────────────────────────────────────────────────────┤
│                   OneLake (Delta Parquet)                        │
├─────────────────────────────────────────────────────────────────┤
│                Security Layer + Microsoft Purview                │
└─────────────────────────────────────────────────────────────────┘
         ↑                    ↑                    ↑
       Azure            AWS / GCP             Dataverse

1.2 Benefits and Challenges of Fabric for Data Science

Benefits

mindmap
  root((Microsoft Fabric))
    Unified Integration
      Storage, engineering, ML, analytics
      No need to switch tools
    Cloud-native Scalability
      Handles large volumes
      Scale up/down as needed
    Integrated AI Tools
      Azure Cognitive Services
      Rapid ML model deployment
      NLP, predictions, automation
    Collaboration
      Shared roles in workspaces
      Data engineers, scientists, analysts
      Increased productivity

Challenges

1.3 Microsoft Fabric Capabilities

1.4 Key Steps of the Data Science Lifecycle with Fabric

flowchart LR
    A["1️⃣ Business\nUnderstanding\n──────\nDefine the problem\nSet objectives\nMeasurable KPIs"] -->
    B["2️⃣ Data\nAcquisition\n──────\nIngest from\nDBs, APIs,\nexternal sources"] -->
    C["3️⃣ Exploration &\nCleaning\n──────\nTrends, patterns\nOutliers, missing\nvalues"] -->
    D["4️⃣ Feature\nEngineering\n──────\nTransformation\nStructuring\nEnrichment"] -->
    E["5️⃣ Model Training\n& Evaluation\n──────\nML Algorithms\nMetrics:\naccuracy, recall"] -->
    F["6️⃣ Model\nDeployment\n──────\nReal-time\nPredictions\nDecisions"] -->
    G["7️⃣ Operation-\nalization\n──────\nMonitoring\nUpdates\nContinuous Insights"]

    style A fill:#4472C4,color:#fff
    style B fill:#ED7D31,color:#fff
    style C fill:#A9D18E,color:#000
    style D fill:#FFD966,color:#000
    style E fill:#FF6B6B,color:#fff
    style F fill:#70AD47,color:#fff
    style G fill:#7030A0,color:#fff

Simplified End-to-End Scenario in Microsoft Fabric

flowchart TD
    A["External Data Source"] -->|Ingestion| B["Data Warehouse\n& Lakehouse"]
    B -->|Exploration and cleaning| C["Notebooks &\nDataflows & Pipelines"]
    C -->|Training| D["ML Models\nExperiments"]
    D -->|Batch scoring| E["Lakehouse\n(Predictions)"]
    E -->|Visualization| F["Power BI\nDashboard"]

1.5 Comparison with Other Data Science Platforms

Key Fabric advantage: Simplicity, integration, and unified approach — ideal for organizations looking to streamline their data workflows.

Ideal Scenarios for Microsoft Fabric

graph LR
    A["✅ Microsoft 365\n+ Azure Organizations"] --> D["Microsoft\nFabric"]
    B["✅ ML + BI\nintegration essential"] --> D
    C["✅ Shared teams\nData Eng + DS + Reporting"] --> D
    style D fill:#0078D4,color:#fff

1.6 Ideal Use Cases for Microsoft Fabric

Microsoft Fabric excels across various industries thanks to its versatility:

graph TD
    FABRIC["Microsoft Fabric\n🏭 Universal Platform"]

    subgraph Retail["🛒 Retail"]
        R1["Predicting consumer\ntrends"]
        R2["Inventory optimization"]
        R3["Targeted marketing\ncampaigns"]
    end

    subgraph Healthcare["🏥 Healthcare"]
        H1["Predicting patient\noutcomes"]
        H2["Real-time analytics"]
        H3["Hospital resource\noptimization"]
    end

    subgraph Finance["💰 Finance"]
        F1["Fraud detection\n(scalable ML pipelines)"]
        F2["Predictive risk\nanalysis"]
        F3["Regulatory reporting"]
    end

    FABRIC --> Retail
    FABRIC --> Healthcare
    FABRIC --> Finance

1.7 Demo — MS Fabric Environment Setup

Activating the Free Fabric Trial

1. Go to the Power BI workspaces page (Power BI service)
2. Click Settings → Start a free trial (60 days)
3. Available licenses:
   • Trial (FT1 — Free Tier 1)
   • Pro
   • Premium (per user or per capacity)
   • Embedded
   • Fabric capacity

Creating a Workspace

1. Click "New workspace"
2. Name: PS-Fabric-Demo-Workspace
3. Description: Demo workspace for MS Fabric
4. Click "Apply"
→ A diamond icon ◆ appears = "Fabric content"

License Options (Workspace settings)

Module 2

Module 2 — Building Data Science Workflows in Microsoft Fabric

This module dives into the technical implementation of workflows, focusing on model training and distributed computing. You will see how Fabric’s infrastructure efficiently handles large datasets and complex models.

2.1 Data Preparation and Transformation for Machine Learning

Data Preparation Pipeline in Fabric

flowchart LR
    subgraph Sources["Data Sources"]
        S1["Azure Data Lake"]
        S2["SQL Databases"]
        S3["APIs"]
        S4["Files"]
        S5["200+ Data Factory\nconnectors"]
    end

    subgraph Fabric["Microsoft Fabric Workspace"]
        direction TB
        A["Ingestion\n(Data Factory)"] -->
        B["Exploration\n(Notebooks)"] -->
        C["Cleaning\n(Spark / Dataflows)"] -->
        D["Feature Engineering\n(Transformations)"] -->
        E["Validation\n(Quality checks)"] -->
        F["Storage\n(Lakehouse)"]
    end

    subgraph Output["Results"]
        G["ML Models"]
        H["BI Insights"]
        I["Predictions"]
    end

    Sources --> Fabric
    F --> Output

Why Data Preparation is Critical

“Garbage in, garbage out” — Data quality determines model quality.

Preparation Tools in Fabric

Example — Cleaning and Transformation with Spark (Python)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when, avg

# Start a Spark session in a Fabric notebook
spark = SparkSession.builder.appName("FabricDataPrep").getOrCreate()

# Load data from the Lakehouse
df = spark.read.format("delta").load("abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/source_records")

# Drop rows with null values in key columns
df_clean = df.dropna(subset=["customer_id", "order_total"])

# Impute missing values with the mean
avg_total = df_clean.select(avg("order_total")).first()[0]
df_clean = df_clean.fillna({"order_total": avg_total})

# Feature Engineering — create a new variable
df_clean = df_clean.withColumn(
    "premium_customer",
    when(col("order_total") > 1000, 1).otherwise(0)
)

# Save the prepared dataset to the Lakehouse
df_clean.write.format("delta").mode("overwrite").save(
    "abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/cleaned_data"
)

print(f"Prepared data: {df_clean.count()} records")

2.2 Model Training: Manual and Automated Approaches (AutoML)

graph LR
    subgraph Manual["🔧 Manual Training"]
        M1["Full control over\nalgorithms + parameters"]
        M2["Libraries: scikit-learn\nPyTorch, TensorFlow"]
        M3["Ideal for specialized\nprojects with expertise"]
        M4["Slower development\nbut more flexible"]
    end

    subgraph Auto["🤖 AutoML (Automated)"]
        A1["Automatically finds\nthe best model"]
        A2["Standard tasks:\nclassification, regression"]
        A3["Accelerated development\nminimal effort"]
        A4["Ideal for predicting\nsales trends"]
    end

    CHOICE{"Which approach?"}

    COMPLEX["Complex project\nwith domain expertise"] --> Manual
    SPEED["Speed is priority\nor quick start"] --> Auto

Example — Manual Training with LightGBM in a Fabric Notebook

import mlflow
import mlflow.lightgbm
import lightgbm as lgb
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load prepared data
df = pd.read_parquet("abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Files/cleaned_data.parquet")

X = df.drop("target", axis=1)
y = df["target"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Start MLflow tracking
mlflow.set_experiment("customer_retention_model")

with mlflow.start_run():
    # Define hyperparameters
    params = {
        "objective": "binary",
        "metric": "binary_logloss",
        "num_leaves": 31,
        "learning_rate": 0.05,
        "n_estimators": 100
    }

    # Train the model
    model = lgb.LGBMClassifier(**params)
    model.fit(X_train, y_train)

    # Evaluate
    predictions = model.predict(X_test)
    accuracy = accuracy_score(y_test, predictions)

    # Log metrics and parameters
    mlflow.log_params(params)
    mlflow.log_metric("accuracy", accuracy)

    # Save model with signature
    mlflow.lightgbm.log_model(model, "lgbm_retention_model")

    print(f"Accuracy: {accuracy:.4f}")

2.3 Experiment Tracking — Monitoring and Managing Models

Experiment tracking in Fabric is like having a personal assistant for your data science projects. It records every detail of your training runs.

graph TD
    subgraph Experiment["MLflow Experiment in Fabric"]
        RUN1["Run 1\nLightGBM\nacc=0.85"] 
        RUN2["Run 2\nRandom Forest\nacc=0.82"]
        RUN3["Run 3\nXGBoost\nacc=0.88"]
    end

    subgraph Tracking["Tracked Elements"]
        P["Parameters\n(hyperparameters)"]
        M["Metrics\n(accuracy, F1, AUC)"]
        A["Artifacts\n(models, charts)"]
        D["Datasets\n(versions used)"]
    end

    subgraph Registry["Model Registry"]
        REG["Registered Model\n(MLflow format)"]
        STAGE["Staging → Production"]
    end

    Experiment --> Tracking
    Tracking --> Registry

Key Experiment Tracking Features

Prerequisites

• Power BI Premium subscription
• Power BI Workspace with assigned Premium capacity

Example — Creating an Experiment and Comparing Runs

import mlflow

# Configure the experiment
mlflow.set_experiment("experiment_baseline")

# Run 1 — LightGBM
with mlflow.start_run(run_name="LightGBM_v1"):
    mlflow.log_param("model_type", "LightGBM")
    mlflow.log_param("n_estimators", 100)
    mlflow.log_param("learning_rate", 0.05)
    mlflow.log_metric("accuracy", 0.885)
    mlflow.log_metric("f1_score", 0.872)
    mlflow.log_metric("auc", 0.921)

# Run 2 — XGBoost
with mlflow.start_run(run_name="XGBoost_v1"):
    mlflow.log_param("model_type", "XGBoost")
    mlflow.log_param("n_estimators", 150)
    mlflow.log_param("max_depth", 6)
    mlflow.log_metric("accuracy", 0.891)
    mlflow.log_metric("f1_score", 0.878)
    mlflow.log_metric("auc", 0.934)

# Retrieve and compare runs
runs = mlflow.search_runs(experiment_names=["experiment_baseline"])
best_run = runs.loc[runs["metrics.auc"].idxmax()]
print(f"Best model: {best_run['params.model_type']}")
print(f"Best AUC  : {best_run['metrics.auc']:.4f}")

2.4 Batch Predictions with Fabric’s PREDICT Function

Fabric’s PREDICT function is a key tool for operationalizing ML models at scale.

flowchart LR
    ML["ML Model\n(MLflow format)"] -->|Registration| REG["Model Registry\nFabric"]
    REG --> PRED["PREDICT\nFunction"]
    DATA["Large Dataset\n(Lakehouse / DW)"] --> PRED
    PRED --> OUT["Batch\nPredictions"]
    OUT --> PBI["Power BI\nDashboard"]
    OUT --> LH["Lakehouse\n(stored results)"]

PREDICT Features

ML Models Supported by PREDICT

✅ CatBoost    ✅ Keras      ✅ LightGBM    ✅ ONNX
✅ Prophet     ✅ PyTorch    ✅ Sklearn     ✅ Spark
✅ Statsmodels ✅ TensorFlow ✅ XGBoost

PREDICT Limitations

⚠️ Current limitations:

• Requires models saved in MLflow format with signatures
• Does not support models with multi-tensor inputs/outputs
• Limited set of supported ML frameworks

Common Use Cases

• Churn prediction — Identify customers likely to leave the service
• Sales forecasting — Predict next quarter’s sales
• Fraud detection — Identify abnormal transactions in batch

Example — Batch Scoring with the PREDICT Function

import mlflow.lightgbm
from synapse.ml.predict import MLFlowTransformer
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("BatchScoringPipeline").getOrCreate()

# Load the model from the Fabric registry
model = MLFlowTransformer(
    inputCols=["feature1", "feature2", "feature3"],
    outputCol="prediction",
    modelName="lgbm_retention_model",
    modelVersion=1
)

# Load data to score
df_new = spark.read.format("delta").load(
    "abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/prospects_data"
)

# Apply PREDICT
df_predictions = model.transform(df_new)

# Save results to the Lakehouse
df_predictions.write.format("delta").mode("overwrite").save(
    "abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/churn_risk_scores"
)

print(f"Predictions generated: {df_predictions.count()} records")

2.5 Distributed Computing for Machine Learning in Fabric

Distributed computing in Fabric is like assembling a team of experts where each member handles part of the task to go faster and better.

graph TD
    subgraph Single["❌ Single Machine"]
        CPU1["Single CPU\n(Slow, limited)"]
    end

    subgraph Distributed["✅ Distributed Computing with Spark"]
        direction LR
        DRIVER["Driver Node\n(Coordination)"]
        W1["Worker 1\n(Partition A)"]
        W2["Worker 2\n(Partition B)"]
        W3["Worker 3\n(Partition C)"]
        WN["Worker N\n(Partition N)"]
        DRIVER --> W1
        DRIVER --> W2
        DRIVER --> W3
        DRIVER --> WN
    end

    subgraph Benefits["Benefits"]
        B1["⚡ Parallel processing"]
        B2["📈 Scalable (up/down)"]
        B3["💰 Pay-per-use"]
        B4["🔄 Real-time (streaming)"]
    end

    Distributed --> Benefits

Distributed Computing Characteristics in Fabric

Example — Configuring a Spark Job for Distributed ML

from pyspark.sql import SparkSession
from pyspark.ml.classification import GBTClassifier
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.evaluation import BinaryClassificationEvaluator

# Spark is already configured in Fabric Notebooks
spark = SparkSession.builder.appName("ParallelMLTraining").getOrCreate()

# Configure the number of partitions for parallelism
spark.conf.set("spark.sql.shuffle.partitions", "200")

# Load a large dataset
df = spark.read.format("delta").load(
    "abfss://workspace@onelake.dfs.fabric.microsoft.com/lakehouse.Lakehouse/Tables/training_dataset"
)

# Assemble features
feature_cols = ["age", "income", "tenure", "usage_frequency"]
assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
df_assembled = assembler.transform(df)

# Train a distributed model (Gradient Boosted Trees)
gbt = GBTClassifier(
    featuresCol="features",
    labelCol="label",
    maxIter=100,
    maxDepth=5
)

# Training runs in parallel across all workers
model = gbt.fit(df_assembled)

# Evaluate
predictions = model.transform(df_assembled)
evaluator = BinaryClassificationEvaluator(labelCol="label")
auc = evaluator.evaluate(predictions)
print(f"AUC (distributed training): {auc:.4f}")

Module 3

Module 3 — Combining AI with Business Intelligence in Microsoft Fabric

This module explores how to combine the power of AI with Business Intelligence in Microsoft Fabric — transforming data insights into intelligent, dynamic tools that guide decision-making.

3.1 Semantic Links — Connecting Data Sources

Semantic Links are like a translator for your data: they help different datasets communicate with each other by defining meaningful relationships.

graph LR
    subgraph PowerBI["Power BI Datasets"]
        PBI1["Sales Dataset"]
        PBI2["Customers Dataset"]
        PBI3["Campaigns Dataset"]
    end

    subgraph SemanticLinks["Semantic Links\n(Link Layer)"]
        SL["🔗 Semantic\nRelations"]
    end

    subgraph FabricNotebook["Fabric Notebooks (Python)"]
        FDF["FabricDataFrame\n(Pandas/Spark wrapper)"]
        ANALYSIS["Python Analysis\nscikit-learn, statsmodels"]
    end

    PowerBI --> SemanticLinks
    SemanticLinks --> FabricNotebook
    FDF --> ANALYSIS

Benefits of Semantic Links

How Semantic Links Work

Semantic Links serve as a bridge between Power BI datasets and Fabric notebooks, allowing Python users to access Power BI data as DataFrames.

Example — Using Semantic Links in a Fabric Notebook

import sempy.fabric as fabric
from sempy.fabric import FabricDataFrame

# List available Power BI datasets
datasets = fabric.list_datasets()
print(datasets)

# Read a Power BI table into a FabricDataFrame
df_sales = fabric.read_table(
    dataset="SalesDataset",
    table="Sales"
)

# FabricDataFrame is a wrapper on Pandas/Spark
# Compatible with standard Python libraries
print(df_sales.head())
print(f"Columns: {df_sales.columns.tolist()}")

# Evaluate DAX metrics from Python
dax_query = "EVALUATE SUMMARIZECOLUMNS('Date'[Year], 'Sales'[TotalRevenue])"
df_dax = fabric.evaluate_dax(
    dataset="SalesDataset",
    dax_string=dax_query
)
print(df_dax)

3.2 Integrating Azure AI Tools into Fabric Workflows

Azure AI brings powerful capabilities directly into Fabric: text analysis, image analysis, speech, and much more.

mindmap
  root((Azure AI in Fabric))
    Content Analysis
      Text & NLP
      Images & Computer Vision
      Speech Processing
    ML Models
      Deploying pre-built models
      Training custom models
      ML Studio in your Fabric project
    Insights Generation
      GPT Models
      Actionable insights from raw data
    Azure Cognitive Services
      Sentiment Analysis
      Language Detection
      Computer vision

Why Integrate Azure AI into Workflows?

3.3 Deriving Insights from AI-Based Techniques

Fabric helps you unlock hidden patterns and trends in your data using advanced AI techniques.

graph TD
    RAW["Raw Data"] --> TECHNIQUES

    subgraph TECHNIQUES["AI Techniques in Fabric"]
        T1["📊 Automated\nAnalysis"]
        T2["🎯 Clustering\n(grouping)"]
        T3["🚨 Anomaly\nDetection"]
        T4["📈 Trend\nIdentification"]
        T5["🔮 Predictive\nModeling"]
        T6["💬 Sentiment\nAnalysis"]
        T7["✨ Data\nAugmentation"]
    end

    TECHNIQUES --> INSIGHTS["Actionable Insights\nStrategic Decisions"]

Example — Trend Identification with statsmodels (Time Series Decomposition)

import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.seasonal import seasonal_decompose

# Load time series data
data = pd.read_csv("time_series_data.csv", index_col="Date", parse_dates=True)

# Decompose the time series into trend, seasonality, and residual
decomposition = seasonal_decompose(data['Value'], model='additive', period=12)

# Visualize the components
fig, (ax1, ax2, ax3, ax4) = plt.subplots(4, 1, figsize=(12, 10))
decomposition.observed.plot(ax=ax1, title="Observed Data")
decomposition.trend.plot(ax=ax2, title="Trend")
decomposition.seasonal.plot(ax=ax3, title="Seasonality")
decomposition.resid.plot(ax=ax4, title="Residual")

plt.tight_layout()
plt.title("Trend Analysis")
plt.show()

print("Trend component (last values):")
print(decomposition.trend.dropna().tail(5))

Example — Predictive Modeling with HyperDrive (Azure ML)

from azureml.core import Workspace, Experiment
from azureml.train.hyperdrive import HyperDriveConfig, RandomParameterSampling
from azureml.train.hyperdrive.parameter_expressions import uniform, choice

# Connect to Azure ML from Fabric
ws = Workspace.from_config()
experiment = Experiment(workspace=ws, name="predictive_modeling")

# Define the parameter space for HyperDrive
param_sampling = RandomParameterSampling({
    "learning_rate": uniform(0.01, 0.1),
    "n_estimators": choice(50, 100, 200),
    "max_depth": choice(3, 5, 7)
})

# Configure HyperDrive for automated hyperparameter tuning
hyperdrive_config = HyperDriveConfig(
    run_config=run_config,
    estimator=estimator,
    hyperparameter_sampling=param_sampling,
    primary_metric_name="accuracy",
    primary_metric_goal="maximize",
    max_total_runs=20
)

# Launch the experiment with HyperDrive
run = experiment.submit(hyperdrive_config)
run.wait_for_completion(show_output=True)

# Retrieve the best model
best_run = run.get_best_run_by_primary_metric()
best_model = best_run.register_model(
    model_name="best_predictive_model",
    model_path="outputs/model"
)
print(f"Best model: {best_model.name} v{best_model.version}")

3.4 NLP and Text Analysis with Fabric

NLP (Natural Language Processing) enables computers to understand human language — text summarization, question answering, entity recognition.

graph LR
    subgraph Sources["Unstructured Text Sources"]
        E["📧 Emails"]
        SM["📱 Social Media"]
        CR["⭐ Customer Reviews"]
        DOC["📄 Documents"]
    end

    subgraph NLP["NLP in Fabric"]
        SA["Sentiment\nAnalysis\n(+/−/neutral)"]
        LT["Language\nTranslation"]
        DC["Document\nClassification"]
        NER["Entity\nRecognition"]
        SUM["Automatic\nSummarization"]
    end

    subgraph Output["Structured Data"]
        TABLE["Analyzed table\nwith confidence\nscores"]
    end

    Sources --> NLP
    NLP --> Output

Sentiment Analysis in Fabric

Sentiment analysis identifies:

• Labels: positive, neutral, negative (at the document AND sentence level)
• Confidence scores: between 0 and 1 for each category

Example — Sentiment Analysis via the Azure AI REST API

import requests
from pprint import pprint
import uuid

# Azure AI Language API configuration
service_url = "https://<your-endpoint>.cognitiveservices.azure.com/language/:analyze-text?api-version=2023-04-01"
post_headers = {
    "Content-Type": "application/json",
    "Ocp-Apim-Subscription-Key": "<your-api-key>"
}

# Request body for sentiment analysis
post_body = {
    "kind": "SentimentAnalysis",
    "parameters": {
        "modelVersion": "latest",
        "opinionMining": "True"
    },
    "analysisInput": {
        "documents": [
            {
                "id": "1",
                "language": "en",
                "text": "The food and service were unacceptable. The concierge was nice, however."
            },
            {
                "id": "2",
                "language": "en",
                "text": "The product is excellent but delivery was very slow."
            }
        ]
    }
}

# Add the unique workload identifier
post_headers["x-ms-workload-resource-moniker"] = str(uuid.uuid1())

# API call
response = requests.post(service_url, json=post_body, headers=post_headers)

# Display results
if response.status_code == 200:
    results = response.json()
    for doc in results["results"]["documents"]:
        print(f"\nDocument {doc['id']}:")
        print(f"  Overall sentiment: {doc['sentiment']}")
        print(f"  Scores: {doc['confidenceScores']}")
        for sentence in doc['sentences']:
            print(f"  Sentence: '{sentence['text']}'")
            print(f"    Sentiment: {sentence['sentiment']}")
else:
    print(f"Error: {response.status_code} - {response.text}")

Reference: https://learn.microsoft.com/en-us/fabric/data-science/ai-services/how-to-use-text-analytics?tabs=rest

3.5 AI Skills in Microsoft Fabric

AI Skills in Microsoft Fabric bridge the gap between complex data and everyday users, leveraging generative AI.

⚠️ Feature in Preview at the time of writing.

How AI Skills Work

sequenceDiagram
    participant U as Non-technical User
    participant AI as AI Skill (LLM)
    participant DB as Warehouse / Lakehouse

    U->>AI: "What were last month's sales by region?"
    AI->>DB: Understands the database schema
    AI->>AI: Generates the appropriate T-SQL query
    AI->>DB: SELECT region, SUM(sales) FROM sales WHERE...
    DB->>AI: Results
    AI->>U: Natural language response + data

Key AI Skills Capabilities

AI Skills Limitations

Module 4

Module 4 — Data Science Governance and Management in Microsoft Fabric

This module focuses on governance and sustainability of data science solutions: lineage tracking, disaster recovery, security, and compliance.

4.1 Lineage Tracking — Reproducibility and Transparency

Lineage Tracking is like a family tree for your data: it traces every step your data and models have taken.

flowchart LR
    subgraph Sources["Sources"]
        S1["Source A\n(CSV)"]
        S2["Source B\n(SQL)"]
        S3["External API"]
    end

    subgraph Transformations["Transformations"]
        T1["ETL Pipeline\nData Factory"]
        T2["Cleaning\nNotebook"]
        T3["Feature Eng.\nSpark Job"]
    end

    subgraph Models["Models"]
        M1["Experiment v1\n(Run 1, 2, 3)"]
        M2["Final registered\nmodel"]
    end

    subgraph Outputs["Outputs"]
        O1["Batch\nPredictions"]
        O2["Power BI\nDashboard"]
        O3["Production\nAPI"]
    end

    S1 --> T1
    S2 --> T1
    S3 --> T1
    T1 --> T2 --> T3 --> M1 --> M2
    M2 --> O1 --> O2
    M2 --> O3

Why Track Experiment Lineage?

Lineage Tracking Tools in Microsoft Fabric

Accessing the Lineage View in Fabric

1. From the workspace → Click "Lineage view"
2. Or from the Lakehouse → Lineage view
3. Or from the items page

→ Displays relationships between all workspace elements
  + external data sources (1 level upstream)

⚠️ Note: At the time of this training, lineage in Fabric is in preview and not all connections are fully supported yet.

4.2 Disaster Recovery Plan for ML Projects

A good disaster recovery plan is just as important as having the plan itself — like a fire drill.

flowchart TD
    subgraph Step1["1️⃣ Identify critical assets"]
        A1["Data"]
        A2["ML Models"]
        A3["Notebooks"]
        A4["Configurations"]
    end

    subgraph Step2["2️⃣ Implement backups + replication"]
        B1["Automated backups\n(data + models)"]
        B2["Multi-region replication"]
        B3["Version control\n(Git)"]
    end

    subgraph Step3["3️⃣ High Availability"]
        C1["Data copies in\nmultiple regions"]
        C2["Fabric components\nhighly available"]
    end

    subgraph Step4["4️⃣ Failover Procedure"]
        D1["Failover to\nbackup system"]
        D2["Defined RTO / RPO"]
    end

    subgraph Step5["5️⃣ Testing and Validation"]
        E1["Regular drills"]
        E2["Validate\neffectiveness"]
    end

    Step1 --> Step2 --> Step3 --> Step4 --> Step5
    Step5 -->|"Continuous improvement"| Step1

Disaster Recovery Strategies in Fabric

Best Practices

✅ Prioritize critical assets (data, models, notebooks, configs)
✅ Regular and automated backups
✅ Multi-region redundancy for critical assets
✅ Test the DR plan regularly (drills)
✅ Define RTO (Recovery Time Objective) and RPO (Recovery Point Objective)
✅ Document recovery procedures

4.3 Security Management and Access Control for ML Assets

Role-Based Access Control (RBAC) in Fabric

graph TD
    subgraph Roles["Fabric Workspace Roles"]
        ADMIN["🔑 Admin\n───────────\nFull access\nAdd/remove admins\nManage workspace"]
        MEMBER["👥 Member\n───────────\nAdd lower-level members\nReshare items\nDB Mirroring\nCANNOT: manage admins"]
        CONTRIB["✏️ Contributor\n───────────\nRead + Write\nCANNOT: manage access"]
        VIEWER["👁️ Viewer\n───────────\nRead only"]
    end

    ADMIN --> MEMBER --> CONTRIB --> VIEWER

    subgraph MLPerms["ML Permissions"]
        ML_RW["Experiments & Models\n— Admin, Member, Contributor:\n  Read + Write\n— Viewer:\n  Read only"]
    end

Permissions by Role

Principle of Least Privilege

“Least Privilege” — Grant only the permissions absolutely necessary.

❌ DO NOT: give Admin to everyone (open doors)
✅ DO: assign the minimum role required for the task
✅ DO: log all activities for audits
✅ DO: regularly review access

4.4 Compliance and Security Best Practices in Fabric

Key Security Features

mindmap
  root((Fabric Security))
    Encryption
      Data at rest
      Data in transit
      Standard encryption technologies
    Multi-factor Authentication MFA
      Additional security layer
      Account protection
    Activity Logging
      Tracking user actions
      Complete audit trails
    Data Masking
      Mask sensitive information
      During analysis

Compliance Features

Best Practices — Summary Table

graph TD
    subgraph Best["Security Best Practices"]
        B1["🔍 Regular access\ncontrol reviews"]
        B2["🔒 Principle of\nleast privilege"]
        B3["📚 Team training\n(security + compliance)"]
        B4["🔥 Regular DR tests\n(recovery drills)"]
        B5["👁️ Continuous monitoring\n(threats, vulnerabilities)"]
        B6["💻 Secure coding\npractices"]
    end

4.5 Demo — Roles, Permissions and Lineage Tracking

Assigning Roles in a Fabric Workspace

1. Go to the Fabric workspace
2. Click "Manage access"
3. Click "Add people or groups"
4. Search for the user (e.g.: "Raghav Kumar")
5. Select the user → Click "Add"
   → Default role: Viewer
6. Modify the role via the dropdown menu
   (e.g.: change to "Contributor" or "Member")

Groups Supported for Role Assignment

• Individual users
• Security groups
• Microsoft 365 groups
• Distribution lists

Accessing the Lineage View

Option 1: From the workspace → "Lineage view" icon (top right)
Option 2: From the Lakehouse → Lineage view
Option 3: From the items page

→ Visually displays:
   • All workspace elements
   • External data sources (1 level upstream)
   • Dependencies between elements

General Summary

graph TD
    subgraph M1["Module 1 — Foundations"]
        A["Fabric\nArchitecture"] --> B["Benefits\n& Challenges"]
        B --> C["Data Science\nLifecycle"]
        C --> D["Platform\nComparison"]
    end

    subgraph M2["Module 2 — Workflows"]
        E["Data\nPreparation"] --> F["ML\nTraining"]
        F --> G["Experiment\nTracking"]
        G --> H["PREDICT\n(Batch)"]
        H --> I["Distributed\nComputing"]
    end

    subgraph M3["Module 3 — AI + BI"]
        J["Semantic\nLinks"] --> K["Azure AI\nIntegration"]
        K --> L["AI Insights\n(Trends)"]
        L --> M["NLP &\nText Analytics"]
        M --> N["AI Skills"]
    end

    subgraph M4["Module 4 — Governance"]
        O["Lineage\nTracking"] --> P["Disaster\nRecovery"]
        P --> Q["RBAC &\nSecurity"]
        Q --> R["Compliance\n(GDPR, HIPAA)"]
    end

    M1 --> M2 --> M3 --> M4

Resources and References

Search Terms

data · science · microsoft · fabric · fundamentals · platforms · databases · sql · lineage · predict · tracking · azure · experiment · features · links · preparation · semantic · skills · analysis · benefits · distributed · security · tools · workspace