Azure Data Science with Microsoft Fabric

Table of Contents

1. Introduction and Overview
2. Microsoft Fabric Architecture
3. Advantages and Challenges of Fabric for Data Science
4. Microsoft Fabric Capabilities
5. Data Science Lifecycle with Fabric
6. Comparison with Other Platforms
7. Ideal Use Cases
8. Demo: Configuring the MS Fabric Environment
9. Data Preparation and Transformation
10. Model Training: Manual vs AutoML
11. Experiment Tracking
12. Batch Predictions with the PREDICT Function
13. Distributed Computing for Machine Learning
14. Semantic Links: Connecting Data and BI
15. Azure AI Tools Integration
16. Deriving Insights with AI Techniques
17. NLP and Text Analytics with Fabric
18. AI Skills in Microsoft Fabric
19. Lineage Tracking: Reproducibility and Transparency
20. Disaster Recovery Plan for ML Projects
21. Security and Access Control for ML Assets
22. Compliance and Security Best Practices

1. Introduction and Overview

Microsoft Fabric is an end-to-end unified platform for data science. It brings together all the necessary tools, from storage to artificial intelligence, including integration, machine learning, and BI.

Course structure in 4 modules:

Module 1 → Foundations and Fabric architecture
Module 2 → Technical implementation of workflows
Module 3 → Advanced AI + BI integration
Module 4 → Governance, security, and sustainability

2. Microsoft Fabric Architecture

Platform Overview

graph TB
    subgraph Sources["Data Sources"]
        AZ[Azure]
        AWS[AWS / Google Cloud]
        DV[Dataverse]
        EXT[External Sources]
    end

    subgraph OneLake["OneLake — Unified Data Lake (Delta Parquet)"]
        WA[Workspace A]
        WB[Workspace B]
    end

    subgraph Services["Microsoft Fabric Services"]
        DF[Data Factory\nIngestion + transformation]
        DW[Data Warehouse\nT-SQL + scalability]
        DE[Data Engineering\nLakehouse + Spark]
        DS[Data Science\nNotebooks + ML Models]
        RTI[Real-time Intelligence\nEventstream + KQL]
        PBI[Power BI\nSemantic Model + Reports]
        DA[Data Activator\nNo-code observability]
    end

    subgraph Compute["Serverless Compute Engines"]
        TSQL[T-SQL Engine]
        SPARK[Spark Engine\nPython · R · Scala]
        AS[Analysis Services]
        KQL[KQL Engine]
    end

    subgraph Security["Security and Governance"]
        SEC[Security Layer\nMicrosoft Purview]
    end

    Sources --> OneLake
    OneLake --> Services
    Services --> Compute
    Compute --> Security

Detailed Components

Hierarchical Structure of a Fabric Tenant

graph TD
    T[Microsoft Fabric Tenant] --> OL[OneLake — Single instance]
    OL --> WS1[Workspace A]
    OL --> WS2[Workspace B]
    OL --> WSN[Workspace N...]
    WS1 --> LH[Lakehouse]
    WS1 --> DWH[Data Warehouse]
    WS1 --> DS[Data Science Items]
    LH --> FILES[Files / Folders]
    LH --> TABLES[Delta Tables]
    DS --> EXP[MLflow Experiments]
    DS --> MOD[ML Models]
    WS1 -.->|Requires| CAP[Assigned compute capacity]

Important: Every workspace must have an assigned capacity (F2, F64, etc.) to function. Without capacity, the workspace cannot execute workloads.

3. Advantages and Challenges of Fabric for Data Science

graph LR
    subgraph AVA["✅ Advantages"]
        A1[Unified platform\nno tool switching]
        A2[Cloud-native scalability\nadaptable to needs]
        A3[Integrated AI tools\nAzure Cognitive Services]
        A4[Simplified collaboration\nshared workspaces]
        A5[Microsoft 365 integration\nTeams, Excel, Power BI]
    end
    subgraph DEF["⚠️ Challenges"]
        D1[Learning curve\nfor new users]
        D2[High costs\nfor intensive workloads]
        D3[Complex models\nmay require dedicated Azure ML]
    end

4. Microsoft Fabric Capabilities

mindmap
  root((Microsoft Fabric))
    Role-specific Workloads
      Data Scientists
      Data Engineers
      Analysts
    OneLake
      Centralized storage
      Simplified discovery
    Copilot Support
      Smart suggestions
      Task automation
    Microsoft 365 Integration
      Teams
      Excel
      Power BI
    Azure AI Foundry
      Model building
      ML deployment
    Unified Data Management
      Centralized governance
      Access control

5. Data Science Lifecycle with Fabric

Lifecycle Diagram

flowchart LR
    B[1️⃣ Business\nUnderstanding\nDefine objectives\nand metrics] --> DA
    DA[2️⃣ Data\nAcquisition\nSources: APIs,\nDBs, files] --> EX
    EX[3️⃣ Exploration\nand cleaning\nTrends,\noutliers, quality] --> MB
    MB[4️⃣ Model\nBuilding and\nEvaluation\nAlgorithms,\nmetrics] --> DEP
    DEP[5️⃣ Model\nDeployment\nReal-time\npredictions] --> OP
    OP[6️⃣ Operationalization\nMonitoring,\nupdates] --> B

    style B fill:#4472C4,color:#fff
    style DA fill:#4472C4,color:#fff
    style EX fill:#4472C4,color:#fff
    style MB fill:#4472C4,color:#fff
    style DEP fill:#4472C4,color:#fff
    style OP fill:#4472C4,color:#fff

Simplified End-to-End Scenario in Fabric

flowchart TD
    SRC[(External\ndata source)] -->|Ingestion| DW[Data Warehouse\n+ Lakehouse]
    DW -->|Exploration and cleaning| NB[Notebooks\nDataflows\nData Pipelines]
    NB -->|Training| ML[ML Models\n+ MLflow Experiments]
    ML -->|Evaluation and comparison| EXP[Experiment Tracking]
    EXP -->|Batch scoring| PRED[Predictions\nsaved in Lakehouse]
    PRED -->|Visualization| PBI[Power BI\nDashboards]

Description of Each Stage

1. Business Understanding — Define the problem, measurable objectives, and KPIs. E.g.: “Predict customer churn for a bank.”
2. Data Acquisition — Ingest data from APIs, databases, CSV files via Data Factory.
3. Exploration, Cleaning, and Visualization — Explore trends, fix missing values, remove duplicates, perform feature engineering.
4. Model Building and Evaluation — Train ML algorithms, evaluate via metrics (accuracy, AUC-ROC, etc.).
5. Model Deployment — Deploy the model to production systems to provide real-time predictions.
6. Operationalization — Monitor performance, update the model if necessary, maintain insight quality.

6. Comparison with Other Platforms

When to Use Microsoft Fabric?

graph TD
    Q1{Already in\nthe Microsoft/Azure\necosystem?} -->|Yes| FAB[✅ Microsoft Fabric\nis ideal]
    Q1 -->|No| Q2{Need tight\nBI integration?}
    Q2 -->|Yes| FAB
    Q2 -->|No| Q3{Complex big data\nwith advanced Spark?}
    Q3 -->|Yes| DBR[Databricks]
    Q3 -->|No| Q4{Native AWS?}
    Q4 -->|Yes| SM[AWS SageMaker]
    Q4 -->|No| GCP[Google Vertex AI]

Ideal scenarios for Fabric:

• Large organizations already using Microsoft 365 and Azure
• Indispensable integration between ML models and BI
• Teams needing shared workspaces (engineering + science + reporting)

7. Ideal Use Cases

graph TD
    subgraph Retail["🛍️ Retail"]
        R1[Customer trend prediction]
        R2[Inventory optimization]
        R3[Targeted marketing campaigns]
    end
    subgraph Healthcare["🏥 Healthcare"]
        H1[Patient outcome prediction]
        H2[Hospital resource optimization]
        H3[Real-time care analytics]
    end
    subgraph Finance["💳 Finance"]
        F1[Fraud detection]
        F2[Scalable ML pipelines for anomalies]
        F3[Real-time risk management]
    end

8. Demo: Configuring the MS Fabric Environment

Activation Steps

sequenceDiagram
    participant U as User
    participant PBI as Power BI Service
    participant AZ as Azure Portal
    participant WS as Fabric Workspace

    U->>PBI: Access Settings
    U->>PBI: Click "Start a free trial" (60 days)
    PBI-->>U: Trial activated (SKU: FT1)

    U->>PBI: Create new workspace
    U->>WS: Name the workspace (e.g.: PS-Fabric-Demo-Workspace)
    WS-->>U: Diamond icon ◆ appears = "Fabric content"

    U->>WS: Workspace Settings > License info
    WS-->>U: Current license: Trial / Pro / Premium

    U->>AZ: Search "Microsoft Fabric"
    U->>AZ: Create Fabric Capacity
    AZ-->>U: Select size (F2 to F64+)
    AZ-->>U: Capacity created

    U->>WS: Assign the capacity to the workspace

Available License Types

Note: F2 size = 2 capacity units. F64 is significantly more expensive. For demos, F2 is sufficient.

9. Data Preparation and Transformation

Importance of Data Preparation

"Garbage in, garbage out" — model quality directly depends
on the quality of the training data.

Available Tools in Fabric

Step 1 — Ingesting Data into the Lakehouse

# Configuration variables
IS_CUSTOM_DATA = False
DATA_ROOT = "/lakehouse/default"
DATA_FOLDER = "Files/churn"
DATA_FILE = "churn.csv"

import os, requests

# URL of the churn dataset (10,000 bank customers)
remote_url = "https://synapseaisolutionsa.blob.core.windows.net/public/bankcustomerchurn"
file_name = "churn.csv"

# Download file to the Lakehouse
os.makedirs(f"{DATA_ROOT}/{DATA_FOLDER}/raw", exist_ok=True)

response = requests.get(f"{remote_url}/{file_name}")
with open(f"{DATA_ROOT}/{DATA_FOLDER}/raw/{file_name}", "wb") as f:
    f.write(response.content)

print("Downloaded demo files into lakehouse")

Step 2 — Loading Data with Spark

# Read CSV from Lakehouse into a Spark DataFrame
df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(f"Files/churn/raw/churn.csv")

# Convert to Pandas DataFrame for visual exploration
df_pandas = df.toPandas()

# Display data with descriptive statistics
display(df)

Step 3 — Exploring and Visualizing Data

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Descriptive statistics
print(df_pandas.describe())
print(f"\nShape: {df_pandas.shape}")
print(f"\nMissing values:\n{df_pandas.isnull().sum()}")

# Distribution of numeric variables
fig, axes = plt.subplots(2, 3, figsize=(15, 8))
numerical_cols = ['CreditScore', 'Age', 'Tenure', 'Balance', 'EstimatedSalary', 'NumOfProducts']
for i, col in enumerate(numerical_cols):
    axes[i//3][i%3].hist(df_pandas[col], bins=30, color='steelblue', edgecolor='white')
    axes[i//3][i%3].set_title(f'Distribution: {col}')
plt.tight_layout()
plt.show()

Step 4 — Cleaning with Data Wrangler (auto-generated code)

# Code generated by Data Wrangler

def clean_data(df):
    # Step 1: Remove duplicate rows (based on RowNumber and CustomerId)
    df = df.drop_duplicates(subset=['RowNumber', 'CustomerId'])

    # Step 2: Remove rows with missing values (all columns)
    df = df.dropna()

    # Step 3: Drop columns not relevant for churn prediction
    cols_to_drop = ['RowNumber', 'CustomerId', 'Surname']
    df = df.drop(columns=cols_to_drop)

    return df

df_clean = clean_data(df_pandas.copy())
print(df_clean.head(5))

Step 5 — Feature Engineering

# Feature engineering to improve the model
def feature_engineering(df):
    # Encode categorical variables
    df = pd.get_dummies(df, columns=['Geography', 'Gender'], drop_first=True)

    # Create a new feature: balance per product
    df['BalancePerProduct'] = df['Balance'] / (df['NumOfProducts'] + 1)

    # Indicator for active customer with positive balance
    df['ActiveWithBalance'] = ((df['IsActiveMember'] == 1) & (df['Balance'] > 0)).astype(int)

    return df

df_features = feature_engineering(df_clean.copy())
print(f"Number of features after engineering: {df_features.shape[1]}")

Step 6 — Saving as Delta Table in the Lakehouse

# Convert back to Spark DataFrame and save in Delta format
df_spark = spark.createDataFrame(df_features)

# Save as Delta table in the Lakehouse
df_spark.write.mode("overwrite").format("delta").save("Tables/churn_data_clean")

print("Delta table created: Tables/churn_data_clean")

Data Preparation Flow (Summary)

flowchart LR
    RAW[(Raw CSV\nchurn.csv\n10,000 rows)] --> ING[Ingestion\nvia Notebook]
    ING --> EXP[Exploration\nStatistics\nVisualizations]
    EXP --> DW[Data Wrangler\nVisual\ncleaning]
    DW --> CODE[Auto-generated code\nby Data Wrangler]
    CODE --> FE[Feature Engineering\nEncoding + new features]
    FE --> DELTA[(Delta Table\nLakehouse\nchurn_data_clean)]
    DELTA --> ML[Ready for\ntraining]

10. Model Training: Manual vs AutoML

Approach Comparison

graph LR
    subgraph MAN["🧑‍🍳 Manual Training"]
        M1[Full control\nover parameters]
        M2[Libraries: scikit-learn,\nPyTorch, TensorFlow]
        M3[Ideal for\ncustom models]
        M4[Requires domain\nexpertise]
        M5[Slower\ndevelopment]
    end
    subgraph AUTO["🤖 AutoML (FLAML)"]
        A1[Automatically finds\nthe best model]
        A2[Accelerated\ndevelopment]
        A3[Ideal for\nstandard tasks]
        A4[Classification\nand regression]
        A5[Minimal manual\neffort required]
    end

Manual Training — LightGBM Model with MLflow

import logging
import mlflow
import mlflow.sklearn
from lightgbm import LGBMClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score, classification_report
from pyspark.ml.feature import VectorAssembler

# Configure logging level
logging.basicConfig(level=logging.WARN)

# Load cleaned data from the Lakehouse
df_ml = spark.read.format("delta").load("Tables/churn_data_clean").toPandas()

# Separate features and label
X = df_ml.drop(columns=['Exited'])
y = df_ml['Exited']

# 80/20 train/test split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# ────────────────────────────────────────────────
# MLflow configuration for experiment tracking
# ────────────────────────────────────────────────
experiment_name = "ps_automl_demo"
mlflow.set_experiment(experiment_name)

with mlflow.start_run(run_name="baseline_lgbm"):
    # Train LightGBM model
    model = LGBMClassifier(
        objective='binary',
        is_unbalance=True,
        n_estimators=300,
        learning_rate=0.05,
        random_state=42
    )
    model.fit(X_train, y_train)

    # Predictions and evaluation
    y_pred_proba = model.predict_proba(X_test)[:, 1]
    roc_auc = roc_auc_score(y_test, y_pred_proba)

    # Log metrics and parameters to MLflow
    mlflow.log_param("n_estimators", 300)
    mlflow.log_param("learning_rate", 0.05)
    mlflow.log_metric("roc_auc", roc_auc)

    # Register the model
    mlflow.sklearn.log_model(model, "lightgbm_churn_model")

    print(f"ROC AUC Score: {roc_auc:.4f}")
    print(classification_report(y_test, model.predict(X_test)))

AutoML with FLAML

from flaml import AutoML
from flaml.automl.spark.utils import to_pandas_on_spark
import mlflow

# Create an AutoML instance
automl = AutoML()

# Define AutoML experiment parameters
automl_settings = {
    "time_budget": 120,           # Time budget in seconds
    "metric": "roc_auc",          # Metric to optimize
    "task": "classification",     # Task type
    "log_file_name": "flaml_churn.log",
    "seed": 42,
    "n_concurrent_trials": 3,
    "estimator_list": ["lgbm", "rf", "xgboost", "extra_tree"],
    "eval_method": "cv",
    "n_splits": 5
}

# Convert to Pandas-on-Spark for Spark parallelization
X_train_ps = to_pandas_on_spark(spark.createDataFrame(
    pd.concat([X_train, y_train], axis=1)
))

# Run AutoML trial with MLflow tracking
with mlflow.start_run(run_name="automl_flaml"):
    automl.fit(
        X_train=X_train,
        y_train=y_train,
        **automl_settings
    )

    # Display the best hyperparameters found
    print(f"Best estimator: {automl.best_estimator}")
    print(f"Best configuration: {automl.best_config}")
    print(f"Best ROC AUC value: {automl.best_loss:.4f}")

    # Log to MLflow
    mlflow.log_params(automl.best_config)
    mlflow.log_metric("best_roc_auc", 1 - automl.best_loss)

Parallelizing AutoML with Apache Spark

# For datasets that fit in a single node's memory,
# multiple AutoML trials can be parallelized simultaneously with Spark

import ray
from flaml import AutoML

# Convert dataset to standard Pandas for parallelization
X_train_pd = X_train  # Already a Pandas DataFrame

# Configure parallelization
automl_parallel = AutoML()
settings_parallel = {
    "time_budget": 120,
    "metric": "roc_auc",
    "task": "classification",
    "n_concurrent_trials": 4,    # 4 parallel trials
    "use_spark": True,           # Enable Spark parallelization
    "seed": 42
}

automl_parallel.fit(X_train_pd, y_train, **settings_parallel)

print(f"Best model (parallel): {automl_parallel.best_estimator}")
print(f"ROC AUC Score: {1 - automl_parallel.best_loss:.4f}")

11. Experiment Tracking

Why Use Experiment Tracking?

graph TD
    ET[Experiment Tracking\nMLflow in Fabric] --> R[Reproducibility\nRecreate past results]
    ET --> C[Comparison\nCompare metrics across runs]
    ET --> V[Versioning\nData and model versions]
    ET --> LM[Lifecycle Management\nTraining → Deployment]
    ET --> COL[Collaboration\nDocument and share processes]

Prerequisites

• Power BI Premium subscription
• Power BI workspace with assigned Premium capacity

Creating and Managing MLflow Experiments

import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient

# ─────────────────────────────────────────────────
# Create or retrieve an MLflow experiment
# ─────────────────────────────────────────────────
experiment_name = "ps_automl_demo"
mlflow.set_experiment(experiment_name)

# View runs of an experiment
client = MlflowClient()
experiment = client.get_experiment_by_name(experiment_name)
runs = client.search_runs(
    experiment_ids=[experiment.experiment_id],
    order_by=["metrics.roc_auc DESC"]
)

print("Performance of registered runs:")
for run in runs:
    print(f"  Run ID: {run.info.run_id[:8]}... | "
          f"ROC AUC: {run.data.metrics.get('roc_auc', 'N/A'):.4f} | "
          f"Model: {run.data.params.get('estimator', 'N/A')} | "
          f"Duration: {(run.info.end_time - run.info.start_time)/1000:.1f}s")

Registering a Model in the Model Registry

# Register the best model in the MLflow registry
best_run = runs[0]
model_uri = f"runs:/{best_run.info.run_id}/lightgbm_churn_model"

# Register in Fabric's Model Registry
registered_model = mlflow.register_model(
    model_uri=model_uri,
    name="churn_prediction_model"
)

print(f"Model registered: {registered_model.name} v{registered_model.version}")

# Promote model to production
client.transition_model_version_stage(
    name="churn_prediction_model",
    version=registered_model.version,
    stage="Production"
)

Experiment Cycle Overview

sequenceDiagram
    participant DS as Data Scientist
    participant NB as Fabric Notebook
    participant MLF as MLflow Tracking
    participant MR as Model Registry
    participant PBI as Power BI

    DS->>NB: Write training code
    NB->>MLF: mlflow.start_run()
    NB->>MLF: log_params(), log_metrics()
    NB->>MLF: log_model()
    MLF-->>DS: Run registered with unique ID
    DS->>MLF: Compare runs (run list)
    DS->>MR: Register the best model
    MR-->>DS: Versioned model (v1, v2...)
    MR->>PBI: Model available for batch scoring

12. Batch Predictions with the PREDICT Function

PREDICT Overview

graph LR
    MOD[Registered MLflow\nModel] --> PRED[Fabric PREDICT\nFunction]
    DS_IN[(Test\nDataset)] --> PRED
    PRED --> OUT[Batch\nPredictions]
    OUT --> LH[(Lakehouse\nDelta Table)]
    OUT --> PBI[Power BI\nDashboards]

ML Frameworks Supported by PREDICT

Important Limitations

• Models must be saved in MLflow format with their signature
• No support for models with multi-tensor inputs/outputs
• Limited framework list (see table above)

Method 1 — Prediction via MLFlowTransformer (Transformer API)

from synapse.ml.predict import MLFlowTransformer
import pandas as pd

# Load test data from the Lakehouse
df_test = spark.read.format("delta").load("Tables/churn_data_clean") \
    .filter("split = 'test'")

# Create the MLFlowTransformer object
# This MLflow wrapper generates batch predictions on a DataFrame
model = MLFlowTransformer(
    inputCols=df_test.columns,           # Input columns
    outputCol="predictions",             # Output column name
    modelName="churn_prediction_model",  # Model name in registry
    modelVersion="1"                     # Model version
)

# Generate batch predictions
df_predictions = model.transform(df_test)

# Display results
display(df_predictions.select("CustomerId", "predictions", "Exited"))

Method 2 — Prediction via Spark SQL API

# Register the model as a temporary SQL function
spark.udf.register(
    "churn_predict",
    mlflow.pyfunc.spark_udf(spark, "models:/churn_prediction_model/1")
)

# Create a temporary view of the test dataset
df_test.createOrReplaceTempView("churn_test_data")

# Use PREDICT in a Spark SQL query
predictions_sql = spark.sql("""
    SELECT
        *,
        churn_predict(
            CreditScore, Geography_France, Geography_Germany,
            Gender_Male, Age, Tenure, Balance, NumOfProducts,
            HasCrCard, IsActiveMember, EstimatedSalary,
            BalancePerProduct, ActiveWithBalance
        ) AS predicted_churn
    FROM churn_test_data
""")

display(predictions_sql)

Method 3 — Prediction via User-Defined Function (UDF)

from pyspark.sql.functions import struct, col
import mlflow.pyfunc

# Load the model as a Spark UDF
predict_udf = mlflow.pyfunc.spark_udf(
    spark,
    model_uri="models:/churn_prediction_model/1",
    result_type="double"
)

# Apply the UDF to the DataFrame
feature_cols = [c for c in df_test.columns if c not in ['Exited', 'split']]
df_with_predictions = df_test.withColumn(
    "predicted_churn_proba",
    predict_udf(struct([col(c) for c in feature_cols]))
)

display(df_with_predictions)

Saving Predictions to the Lakehouse

# Save results to a Delta table in the Lakehouse
table_name = "customer_churn_test_predictions"

df_with_predictions.write \
    .mode("overwrite") \
    .format("delta") \
    .save(f"Tables/{table_name}")

print(f"Predictions saved to Delta table: Tables/{table_name}")

# Verification
df_saved = spark.read.format("delta").load(f"Tables/{table_name}")
print(f"Number of predictions saved: {df_saved.count()}")
display(df_saved.limit(5))

13. Distributed Computing for Machine Learning

Why Distributed Computing?

graph TD
    subgraph Problem["❌ Without Distributed Computing"]
        P1[Single machine]
        P2[Sequential processing]
        P3[Memory bottleneck]
        P4[Slow on large datasets]
    end
    subgraph Solution["✅ With Distributed Computing (Spark)"]
        S1[Cluster of machines]
        S2[Parallel processing]
        S3[Horizontal scalability]
        S4[Pay-as-you-go]
    end
    Problem --> SPARK[Apache Spark\nin Microsoft Fabric]
    SPARK --> Solution

Spark Characteristics in Fabric

Distributed Processing Example with Spark

from pyspark.sql import functions as F
from pyspark.ml.feature import VectorAssembler, StandardScaler
from pyspark.ml.classification import LightGBMClassifier as SparkLGBM
from pyspark.ml import Pipeline

# Load data (distributed across the cluster)
df_spark = spark.read.format("delta").load("Tables/churn_data_clean")

# Show cluster statistics
print(f"DataFrame partitions: {df_spark.rdd.getNumPartitions()}")
print(f"Row count: {df_spark.count()}")

# Assemble features into a vector (distributed operation)
feature_cols = [c for c in df_spark.columns if c != 'Exited']
assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")

# Normalize features
scaler = StandardScaler(inputCol="features", outputCol="scaled_features")

# Distributed ML pipeline
pipeline = Pipeline(stages=[assembler, scaler])
pipeline_model = pipeline.fit(df_spark)
df_prepared = pipeline_model.transform(df_spark)

print("Distributed pipeline executed successfully!")

14. Semantic Links: Connecting Data and BI

What is a Semantic Link?

graph LR
    PBI[Semantic Model\nPower BI\n(Tables + Measures + Relations)] <-->|Semantic Link| NB[Fabric\nNotebooks\n(Python / PySpark)]
    NB --> FDF[Fabric DataFrame\n= Pandas DF + semantic metadata\n(relations, measures, categories)]
    FDF --> ANA[Advanced analysis\nML + Python]

Semantic Link advantages:

• Read Power BI tables directly into a Fabric DataFrame
• Access metadata: relations, measures, data categories
• Data lineage tracking
• No data duplication
• Performance management without bottlenecks

Installation and Usage

# Install the semantic-link package
%pip install semantic-link

# Import the library
import sempy.fabric as fabric

# ─────────────────────────────────────────────────
# Use case 1: Read a Power BI table into a Fabric DataFrame
# ─────────────────────────────────────────────────
dataset_name = "Bank Churn Predictions"
table_name = "customer_churn_test_predictions"

# Load the table into a Fabric DataFrame
fabric_df = fabric.read_table(dataset_name, table_name)

# Display the first 5 rows
print(fabric_df.head())

# Check the object type (FabricDataFrame, not a standard Pandas DF)
print(type(fabric_df))
# Output: <class 'sempy.fabric.FabricDataFrame'>

# DataFrame dimensions
print(f"Dimensions: {fabric_df.shape}")
# E.g.: (2000, 19) → 2000 rows, 19 columns

Accessing Semantic Information from the Power BI Model

# ─────────────────────────────────────────────────
# Use case 2: Explore semantic metadata
# ─────────────────────────────────────────────────

# List relationships between data model tables
relationships = fabric.list_relationships(dataset_name)
print("Relationships in the model:")
print(relationships)

# List available measures (e.g.: Churn Rate, Average Revenue, etc.)
measures = fabric.list_measures(dataset_name)
print(f"\nAvailable measures ({len(measures)}):")
print(measures)

# ─────────────────────────────────────────────────
# Use case 3: Evaluate a measure directly from the notebook
# ─────────────────────────────────────────────────
churn_rate = fabric.evaluate_measure(
    dataset=dataset_name,
    measure="Churn Rate"
)
print(f"\nChurn rate (from notebook): {churn_rate:.2%}")
# Expected output: 0.17 (corresponds to the 17% visible in Power BI)

Semantic Link Flow

sequenceDiagram
    participant PBI as Power BI Semantic Model
    participant SL as Semantic Link (sempy)
    participant NB as Fabric Notebook
    participant ML as ML Model

    NB->>SL: fabric.read_table("Bank Churn Predictions", "customer_churn_test_predictions")
    SL->>PBI: Query the dataset
    PBI-->>SL: Data + metadata (measures, relations)
    SL-->>NB: FabricDataFrame (with lineage)
    NB->>SL: fabric.evaluate_measure("Churn Rate")
    PBI-->>NB: 0.17 (17%)
    NB->>ML: Use enriched data for ML

15. Azure AI Tools Integration

Azure AI Capabilities Available in Fabric

mindmap
  root((Azure AI\nin Fabric))
    Text analysis
      Sentiment analysis
      Language detection
      Entity extraction
    Computer vision
      Image analysis
      OCR
      Object detection
    Audio and speech
      Speech-to-text
      Text-to-speech
    GPT Models
      Insight generation
      Document summarization
      Q&A on data
    Azure AI Foundry
      Custom model building
      Deployment and management
    Machine Learning Studio
      AutoML
      Designer

Why Integrate AI into Workflows?

16. Deriving Insights with AI Techniques

Available Analysis Types

graph TD
    DATA[(Raw data)] --> AUT[Automated analysis\nlarge volumes]
    DATA --> CLUST[Clustering\nGroup similar data points]
    DATA --> ANOM[Anomaly detection\nFraud, outliers]
    DATA --> TREND[Trend identification\nStrategic decisions]
    DATA --> PRED[Predictive modeling\nFuture forecasts]
    DATA --> SENT[Sentiment analysis\nCustomer feedback]
    DATA --> AUG[Data augmentation\nAI-generated new features]

Code — Trend Identification (Time Series)

import pandas as pd
import matplotlib.pyplot as plt
from statsmodels.tsa.seasonal import seasonal_decompose

# Load time series data
data = pd.read_csv("time_series_data.csv", index_col="Date", parse_dates=True)

# Decompose the time series into trend, seasonality, and residual
decomposition = seasonal_decompose(data['Value'], model='additive', period=12)

# Visualize components
fig, (ax1, ax2, ax3, ax4) = plt.subplots(4, 1, figsize=(12, 10))
decomposition.observed.plot(ax=ax1, title='Observed data')
decomposition.trend.plot(ax=ax2, title='Trend')
decomposition.seasonal.plot(ax=ax3, title='Seasonality')
decomposition.resid.plot(ax=ax4, title='Residual')
plt.tight_layout()
plt.show()

Code — Predictive Modeling with HyperDrive (Azure ML)

from azureml.core.train import HyperDriveConfig
from azureml.train.hyperdrive import (
    RandomParameterSampling,
    choice, uniform,
    PrimaryMetricGoal
)

# Define the hyperparameter search space
hyperparameter_sampling = RandomParameterSampling({
    "learning_rate": uniform(0.001, 0.1),
    "n_estimators": choice(100, 200, 300, 500),
    "max_depth": choice(3, 5, 7, 10),
    "subsample": uniform(0.6, 1.0)
})

# Configure HyperDrive for automatic tuning
hyperdrive_config = HyperDriveConfig(
    run_config=run_config,
    estimator=estimator,
    hyperparameter_sampling=hyperparameter_sampling,
    primary_metric_name="accuracy",
    primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,
    max_total_runs=20,
    max_concurrent_runs=4
)

# Launch training with HyperDrive
run = experiment.submit(hyperdrive_config)
run.wait_for_completion(show_output=True)

# Retrieve the best model
best_run = run.get_best_run_by_primary_metric()
best_model = best_run.register_model(model_name="best_churn_model")
print(f"Best model: {best_run.get_metrics()}")

17. NLP and Text Analytics with Fabric

Available NLP Features

graph LR
    subgraph NLP["NLP in Fabric"]
        SA[Sentiment analysis\nPositive / Neutral / Negative]
        LD[Language detection]
        NER[Named entity recognition\nPersons, Locations, Organizations]
        DC[Document classification]
        TR[Automatic translation]
        OM[Opinion Mining\nAspect-based sentiment]
    end
    TEXT[Unstructured text\nEmails, Reviews, Social media] --> NLP
    NLP --> ACT[Structured\nactionable data]

Sentiment Analysis — Confidence Scores

Entire document → Overall sentiment (Positive / Negative / Mixed / Neutral)
    ↓
Each sentence → Individual sentiment with score [0.0 — 1.0]
    ↓
Opinion Mining → Targeted entities + sentiment per entity

Code — Sentiment Analysis via Azure Cognitive Services REST API

import requests
from pprint import pprint
import uuid

# ─────────────────────────────────────────────────
# Service configuration (retrieve from Azure Portal)
# ─────────────────────────────────────────────────
# service_url = "https://<your-endpoint>.cognitiveservices.azure.com/..."
# post_headers = {"Ocp-Apim-Subscription-Key": "<your-key>"}

# Sentiment analysis request body
post_body = {
    "kind": "SentimentAnalysis",
    "parameters": {
        "modelVersion": "latest",
        "opinionMining": "True"        # Enables Opinion Mining (aspect-based)
    },
    "analysisInput": {
        "documents": [
            {
                "id": "1",
                "language": "en",
                "text": "The food and service were unacceptable. The concierge was nice, however."
            }
        ]
    }
}

# Unique request identifier
post_headers["x-ms-workload-resource-moniker"] = str(uuid.uuid1())

# Send the request
response = requests.post(service_url, json=post_body, headers=post_headers)

# ─────────────────────────────────────────────────
# Expected results:
# ─────────────────────────────────────────────────
# Document overall  → Sentiment: "mixed"
# Sentence 1: "The food and service were unacceptable."
#            → Sentiment: "negative"
#            → Opinion Mining: food=negative, service=negative
# Sentence 2: "The concierge was nice, however."
#            → Sentiment: "positive"
#            → Opinion Mining: concierge=positive
# ─────────────────────────────────────────────────
pprint(response.json())

Interpreting Sentiment Results

Example response for:
"The food and service were unacceptable. The concierge was nice, however."

┌─────────────────────────────────────────────────────┐
│  Document sentiment: MIXED                          │
│  ┌──────────────────────────────────────────────┐   │
│  │  Sentence 1: "food and service unacceptable" │   │
│  │  → Sentiment: NEGATIVE (confidence: 0.97)    │   │
│  │  → food        : negative                    │   │
│  │  → service     : negative                    │   │
│  ├──────────────────────────────────────────────┤   │
│  │  Sentence 2: "concierge was nice"            │   │
│  │  → Sentiment: POSITIVE (confidence: 0.94)    │   │
│  │  → concierge   : positive                    │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

References

• Microsoft Documentation: Text Analytics in Fabric

18. AI Skills in Microsoft Fabric

⚠️ This feature is in Preview at the time of writing.

Overview

AI Skills in Fabric allow non-technical users to ask questions in natural language about their data and get precise answers via T-SQL queries automatically generated by a Large Language Model (LLM).

graph LR
    USER[Non-technical user] -->|Question in English\n"What is the average balance\nof churned customers?"| SKILL[AI Skill\nFabric]
    SKILL -->|Automatic T-SQL generation| SCHEMA[Data schema\nunderstanding]
    SCHEMA -->|Executed SQL query| DB[(Warehouse /\nLakehouse)]
    DB -->|Results| USER

Key Capabilities

Current Limitations

19. Lineage Tracking: Reproducibility and Transparency

What is Data Lineage?

graph LR
    SRC[(Data\nSource)] -->|Transformation 1| T1[Cleaning\n+ Feature Eng.]
    T1 -->|Transformation 2| T2[Model\nTraining]
    T2 --> M1[Model v1]
    T2 --> M2[Model v2]
    M1 -->|Comparison| EXP[Experiments]
    M1 -->|Deployment| API[Prediction API]
    API --> PBI[Power BI Dashboard]

    style SRC fill:#4472C4,color:#fff
    style T1 fill:#70AD47,color:#fff
    style T2 fill:#70AD47,color:#fff
    style M1 fill:#FFC000,color:#000
    style M2 fill:#FFC000,color:#000
    style API fill:#FF0000,color:#fff

Lineage = the “genetic map” of your ML pipeline.
It traces every step, from raw data all the way to insights presented in Power BI.

Why Track Experiment Lineage?

Lineage Tracking Tools in Fabric

graph TD
    subgraph Tools["Tracking tools in Microsoft Fabric"]
        AML[Azure ML Integration\nAutomatic tracking of sources,\ntransformations and outputs]
        VWT[Visual Workflow Tracking\nVisual dependency map]
        DML[Dataset & Model Logging\nVersion history]
        AMA[Activity Monitoring & Alerts\nMonitoring and notifications]
        COL[Collaboration features\nTeam sharing and transparency]
    end

Accessing the Lineage View in Fabric

To access the lineage view:

1. Go to your workspace
2. Click the Lineage view icon (list view)
3. See all relationships between workspace items and external sources

⚠️ Note: The Lineage view in Fabric is in Preview at the time of writing. Not all connections between items are fully supported yet.

Lineage View — Structure

Fabric Workspace
│
├── External source (CSV, API, DB)
│       ↓
├── Data Pipeline (Data Factory)
│       ↓
├── Lakehouse
│   ├── Files/churn/raw/churn.csv
│   └── Tables/churn_data_clean (Delta)
│           ↓
├── Notebook (preparation + feature engineering)
│           ↓
├── MLflow Experiment (ps_automl_demo)
│   ├── Run: baseline_lgbm (LightGBM)
│   └── Run: automl_flaml (FLAML)
│           ↓
├── ML Model (churn_prediction_model v1)
│           ↓
├── Delta Table (customer_churn_test_predictions)
│           ↓
└── Power BI Semantic Model (Bank Churn Predictions)
            ↓
        Power BI Dashboard (PS-Demo-Report)

20. Disaster Recovery Plan for ML Projects

Recovery Strategies in Fabric

graph TD
    DR[Disaster recovery\nplan] --> B[Automated backups\nData + Models + Configs]
    DR --> VC[Versioning\nReproduce previous states]
    DR --> MRR[Multi-region redundancy\nCritical assets protected]
    DR --> FO[Failover procedures\nSwitch to backup system]
    DR --> TEST[Regular testing\nValidation drills]

Steps to Build a DR Plan

flowchart LR
    S1[1. Identify\ncritical assets\nData, Models,\nNotebooks, Configs] --> S2
    S2[2. Implement\nbackups and\nreplication] --> S3
    S3[3. Configure\nredundancy\nFabric multi-region] --> S4
    S4[4. Define\nfailover\nprocedures] --> S5
    S5[5. Test\nand iterate\nthe DR plan]

Critical Assets to Back Up

DR Best Practices

• Perform regular and automated backups (data, models, notebooks)
• Implement multi-region redundancy for the most critical assets
• Use Fabric’s high-availability features
• Define and test failover procedures regularly
• Test the DR plan like a fire drill: don’t wait for a real incident

21. Security and Access Control for ML Assets

Roles in Microsoft Fabric Workspaces

graph TD
    WS[Fabric Workspace] --> ADM[Admin\nFull access\nManage members\nDelete workspace]
    WS --> MEM[Member\nAdd members\nReshare items\nCreate/Edit DB mirroring]
    WS --> CON[Contributor\nRead and write\nin the workspace]
    WS --> VIE[Viewer\nRead-only\naccess to items]

    style ADM fill:#FF0000,color:#fff
    style MEM fill:#FFC000,color:#000
    style CON fill:#70AD47,color:#fff
    style VIE fill:#4472C4,color:#fff

Permission Matrix by Role

Demo — Assigning Roles in Fabric

1. Open the workspace in Power BI Service
2. Click "Manage access"
3. Click "Add people or groups"
4. Search for the user (e.g.: Raghav Kumar)
5. Select the user → click "Add"
   → Default role: Viewer
6. Change the role from the dropdown menu
   → Switch to Contributor, Member, or Admin
7. Repeat for other users

Security Best Practices

graph LR
    subgraph Practices["🔒 Best practices"]
        LP[Principle of least\nprivilege\nGrant only the\nminimum necessary]
        LA[Activity logging\nFor audits and compliance]
        RP[Regular review\nof access policies]
        MFA[Multi-factor\nAuthentication MFA]
        DM[Data Masking\nMask sensitive data]
        ENC[Encryption\nData in transit and at rest]
    end

22. Compliance and Security Best Practices

Security Features in Fabric

Compliance Features

Complete Security Architecture

graph TD
    subgraph EXT["External Access"]
        USR[Users]
        APP[Applications]
        API_EXT[External APIs]
    end
    subgraph AUTH["Authentication & Authorization"]
        AAD[Azure Active Directory\nSSO + MFA]
        RBAC[Role-Based Access Control\nAdmin/Member/Contributor/Viewer]
    end
    subgraph DATA["Data Protection"]
        ENC[Encryption\nAzure Key Vault]
        DM[Data Masking\nMicrosoft Purview]
        DLP[Data Loss Prevention]
    end
    subgraph AUDIT["Governance & Audit"]
        LOG[Activity Logs\nFull logging]
        AT[Audit Trails\nRegulatory compliance]
        LIN[Lineage Tracking\nData traceability]
    end

    EXT --> AUTH
    AUTH --> DATA
    DATA --> AUDIT

Security and Compliance Checklist

☐  Enable MFA for all users
☐  Apply least-privilege principle (RBAC)
☐  Configure data retention policies
☐  Enable activity logging
☐  Implement data masking for PII
☐  Configure Microsoft Purview for governance
☐  Regularly test the disaster recovery plan
☐  Train teams on security practices
☐  Implement Secure Coding practices
☐  Monitor threats and vulnerabilities

Appendix — Key Concepts Summary

Glossary

Course Module Overview

graph LR
    M1[Module 1\nFoundations\nArchitecture + Use cases\nPlatform comparison] --> M2
    M2[Module 2\nWorkflows\nData preparation\nTraining + PREDICT] --> M3
    M3[Module 3\nAI + BI\nSemantic Links\nNLP + AI Skills] --> M4
    M4[Module 4\nGovernance\nLineage + DR\nSecurity + Compliance]

    style M1 fill:#4472C4,color:#fff
    style M2 fill:#70AD47,color:#fff
    style M3 fill:#FFC000,color:#000
    style M4 fill:#FF0000,color:#fff

Search Terms

azure · data · science · microsoft · fabric · fundamentals · platforms · databases · sql · lineage · security · available · spark · experiment · model · semantic · tracking · via · analysis · api · automl · capabilities · compliance · distributed