Manage Data with Azure Databricks and Azure Data Lake

Level: Intermediate / Advanced | Platform: Azure Databricks + ADLS Gen2

Table of Contents

1. ADLS Gen2 Architecture and Foundations
2. Azure Databricks Integration with ADLS Gen2
3. Security with RBAC and Managed Identities
4. Authentication Mechanisms
5. Azure Key Vault — Secure Secret Management
6. Connecting to ADLS Gen2 from Databricks
7. Mount Points vs Direct Access
8. Troubleshooting Connectivity Issues
9. PySpark vs SQL — Decision Guide
10. Data Ingestion from ADLS Gen2
11. Schemas: Inference and Evolution
12. Auto Loader — Incremental Ingestion
13. Optimization: Parquet vs Delta Lake
14. Unity Catalog — Centralized Governance
15. Configuring Unity Catalog in a Workspace
16. Sharing Data Between Workspaces
17. Data Lineage with Unity Catalog
18. Access Control Models Compared
19. Delta Sharing — Cross-Organization Sharing
20. Fine-Grained Permissions: Rows and Columns
21. Summary and Best Practices
22. Glossary

1. ADLS Gen2 Architecture and Foundations

1.1 Why Azure Data Lake Storage Gen2?

Azure Data Lake Storage Gen2 (ADLS Gen2) is Microsoft’s storage solution optimized for large-scale data analytics. It combines two technologies:

• Azure Blob Storage: scalability, durability, cost
• Hierarchical Namespace (HNS): real folders with atomic operations

graph TB
    subgraph "Classic Blob Storage"
        BS[Flat Structure]
        BS --> BL1[container/folder1/file1.csv]
        BS --> BL2[container/folder1/file2.csv]
        BS --> BL3[container/folder2/file3.csv]
        BL1 -.->|Rename = copy + delete| BL1
        note1["❌ Rename = O\(N\) expensive\n❌ No real hierarchy"]
    end
    
    subgraph "ADLS Gen2 with HNS"
        ADLS[Hierarchical Namespace]
        ADLS --> D1[raw/]
        ADLS --> D2[processed/]
        ADLS --> D3[output/]
        D1 --> F1[marketing/campaign_data.json]
        D1 --> F2[hr/employees.csv]
        D2 --> F3[year=2024/month=01/]
        note2["✅ Rename = O\(1\) atomic\n✅ True file system\n✅ ACLs at folder level"]
    end

1.2 ADLS Gen2 Distinctive Features

1.3 ABFSS Path Format

abfss://<container>@<storage_account>.dfs.core.windows.net/<path>

Examples:

# Container "raw", account "salesdatalake"
raw_path = "abfss://raw@salesdatalake.dfs.core.windows.net/"

# Specific path to a file
file_path = "abfss://raw@salesdatalake.dfs.core.windows.net/marketing/campaign_2024.json"

# Partitioned path
partition_path = "abfss://raw@salesdatalake.dfs.core.windows.net/hr/year=2024/month=01/"

1.4 Lakehouse Architecture with ADLS Gen2

graph LR
    subgraph "Data Sources"
        S1[CSV/JSON/Parquet\nFlat Files]
        S2["(SQL Databases\nSQL Server/Oracle)"]
        S3[APIs/Streaming\nEvent Hubs/Kafka]
        S4[SaaS Applications\nSalesforce/SAP]
    end
    
    subgraph "ADLS Gen2"
        subgraph "Zones"
            RAW[Container: raw\nUntouched raw data]
            PROC[Container: processed\nCleaned data]
            OUT[Container: output\nFinal business data]
        end
    end
    
    subgraph "Azure Databricks"
        DL[Delta Lake\nTransaction Log + Parquet]
        Spark[Apache Spark + Photon]
        UC[Unity Catalog\nGovernance]
        
        subgraph "Personas"
            DE[Data Engineering]
            SQL3[SQL Analytics]
            ML[Machine Learning]
        end
    end
    
    subgraph "Consumption"
        PBI[Power BI]
        TB[Tableau]
        API2[REST APIs]
        DS[Data Science Models]
    end
    
    S1 & S2 & S3 & S4 --> RAW
    RAW --> DL
    DL --> PROC
    PROC --> OUT
    DL <--> Spark
    Spark --> UC
    UC --> DE & SQL3 & ML
    OUT --> PBI & TB & API2 & DS

Lakehouse = Data Lake + Data Warehouse:

• Data Lake flexibility: all file formats, all structures
• Data Warehouse performance: fast SQL queries, ACID, schema

2. Azure Databricks Integration with ADLS Gen2

2.1 Integration Architecture

sequenceDiagram
    participant Dev as Developer
    participant DB as Databricks Cluster
    participant AAD as Azure Active Directory
    participant KV as Azure Key Vault
    participant ADLS as ADLS Gen2

    Dev->>DB: Submit Spark job
    DB->>AAD: Authenticate (service principal / managed identity)
    AAD-->>DB: OAuth 2.0 Token
    DB->>KV: Retrieve credentials (if SP)
    KV-->>DB: Encrypted credentials
    DB->>ADLS: Access data with token
    ADLS->>AAD: Validate RBAC permissions
    AAD-->>ADLS: Access granted
    ADLS-->>DB: Data returned
    DB-->>Dev: Job results

2.2 Benefits of Databricks + ADLS Gen2 Integration

3. Security with RBAC and Managed Identities

3.1 Azure RBAC — Role-Based Access Control

Azure RBAC controls who can access what in your Azure environment:

graph TB
    subgraph "Azure RBAC Roles for ADLS Gen2"
        R1[Storage Blob Data Reader\n• Read blobs/files\n• List containers]
        R2[Storage Blob Data Contributor\n• Read + Write + Delete blobs\n• Create/delete containers]
        R3[Storage Blob Data Owner\n• Full access\n• Manage POSIX ACLs]
        R4[Storage Account Contributor\n• Manage storage account\n• No data access]
    end
    
    R1 -.->|Inherits from| R2
    R2 -.->|Inherits from| R3

3.2 Types of Managed Identities

3.3 Configuring RBAC for Databricks

# Variables
SUBSCRIPTION_ID="your-subscription-id"
RESOURCE_GROUP="DataRG"
STORAGE_ACCOUNT="salesdatalake"
DATABRICKS_WORKSPACE_NAME="databricks-workspace"

# Get the managed identity ID of the Databricks workspace
PRINCIPAL_ID=$(az resource show \
  --resource-group $RESOURCE_GROUP \
  --resource-type "Microsoft.Databricks/workspaces" \
  --name $DATABRICKS_WORKSPACE_NAME \
  --query "identity.principalId" -o tsv)

echo "Principal ID: $PRINCIPAL_ID"

# Assign Storage Blob Data Contributor role to the managed identity
SCOPE="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.Storage/storageAccounts/${STORAGE_ACCOUNT}"

az role assignment create \
  --assignee "$PRINCIPAL_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "$SCOPE"

echo "Role assigned successfully"

3.4 End-to-End Authentication

sequenceDiagram
    participant Spark as Spark Job
    participant MI as Managed Identity
    participant AAD as Azure AD
    participant RBAC as RBAC Check
    participant ADLS as ADLS Gen2

    Spark->>MI: Request access token
    MI->>AAD: Authenticate automatically
    AAD-->>MI: OAuth 2.0 Token
    MI-->>Spark: Token returned
    Spark->>ADLS: Read/write request
    ADLS->>RBAC: Check permissions
    RBAC->>AAD: Validate token and roles
    AAD-->>RBAC: Role = Storage Blob Data Contributor
    RBAC-->>ADLS: Access granted
    ADLS-->>Spark: Data returned

4. Authentication Mechanisms

4.1 Comparing Authentication Methods

4.2 AAD Credential Passthrough — Configuration

⚠️ Deprecated: The Credential Passthrough feature has been deprecated since Databricks Runtime 15.0. Use Unity Catalog for new projects.

# Enable Credential Passthrough in cluster configuration
# (via the interface or API — NOT in Python code)

# Cluster configuration to enable:
# Advanced Options > Enable credential passthrough for user-level data access
# OR via JSON:
cluster_config = {
    "spark_conf": {
        "spark.databricks.passthrough.enabled": "true",
        "spark.databricks.repl.allowedLanguages": "python,sql"
    }
}

# In notebook — transparent access with user's identity
spark.conf.set("spark.databricks.passthrough.enabled", "true")

# Read data — the user's identity is used automatically
df = spark.read.csv(
    "abfss://raw@salesdatalake.dfs.core.windows.net/hr/employees.csv",
    header=True
)
df.show()

4.3 Service Principal — Complete Configuration

# Secure retrieval from Azure Key Vault via Databricks Secret Scope
client_id = dbutils.secrets.get(scope="kv-datalake-secrets", key="sp-client-id")
client_secret = dbutils.secrets.get(scope="kv-datalake-secrets", key="sp-client-secret")
tenant_id = dbutils.secrets.get(scope="kv-datalake-secrets", key="tenant-id")
storage_account = "salesdatalake"

# Configure OAuth authentication for ADLS Gen2
spark.conf.set(
    f"fs.azure.account.auth.type.{storage_account}.dfs.core.windows.net",
    "OAuth"
)
spark.conf.set(
    f"fs.azure.account.oauth.provider.type.{storage_account}.dfs.core.windows.net",
    "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.id.{storage_account}.dfs.core.windows.net",
    client_id
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.secret.{storage_account}.dfs.core.windows.net",
    client_secret
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.endpoint.{storage_account}.dfs.core.windows.net",
    f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
)

print("Service Principal authentication configured successfully")

# Test access
files = dbutils.fs.ls(f"abfss://raw@{storage_account}.dfs.core.windows.net/")
for f in files:
    print(f"  {f.path} ({f.size} bytes)")

5. Azure Key Vault — Secure Secret Management

5.1 Key Vault Architecture with Databricks

graph LR
    subgraph "Azure Key Vault"
        KV[kv-datalake-secrets]
        S1[Secret: sp-client-id]
        S2[Secret: sp-client-secret]
        S3[Secret: tenant-id]
        S4[Secret: adls-account-name]
        KV --> S1 & S2 & S3 & S4
    end
    
    subgraph "Databricks"
        SS[Secret Scope\nkv-datalake-secrets]
        NB[Notebook]
        NB -->|dbutils.secrets.get| SS
        SS -->|Secure request| KV
    end
    
    SP[Service Principal\nsp-datalake-access] -->|Authenticates with| KV
    Databricks -->|Via Managed Identity| KV

5.2 Create an Azure Key Vault

# Create the Key Vault
az keyvault create \
  --name "kv-datalake-secrets" \
  --resource-group "DataRG" \
  --location "eastus" \
  --sku standard \
  --enabled-for-template-deployment true

# Configure authorization model (Access Policy — required for Databricks)
az keyvault update \
  --name "kv-datalake-secrets" \
  --resource-group "DataRG" \
  --set properties.accessPolicies=[]

# Add Access Policy for your user
USER_OBJECT_ID=$(az ad signed-in-user show --query "id" -o tsv)
az keyvault set-policy \
  --name "kv-datalake-secrets" \
  --object-id "$USER_OBJECT_ID" \
  --secret-permissions get list set delete

# Add Access Policy for Databricks (if needed)
# Databricks workspace managed identity
az keyvault set-policy \
  --name "kv-datalake-secrets" \
  --object-id "$DATABRICKS_MANAGED_IDENTITY_ID" \
  --secret-permissions get list

5.3 Create a Service Principal and Store Secrets

# Create the Service Principal
SP_INFO=$(az ad sp create-for-rbac \
  --name "sp-datalake-access" \
  --skip-assignment \
  --output json)

SP_CLIENT_ID=$(echo $SP_INFO | jq -r '.appId')
SP_CLIENT_SECRET=$(echo $SP_INFO | jq -r '.password')
SP_TENANT_ID=$(echo $SP_INFO | jq -r '.tenant')

echo "Client ID: $SP_CLIENT_ID"
echo "Tenant ID: $SP_TENANT_ID"
# ⚠️ Never display the secret in logs!

# Store secrets in Key Vault
az keyvault secret set \
  --vault-name "kv-datalake-secrets" \
  --name "sp-client-id" \
  --value "$SP_CLIENT_ID"

az keyvault secret set \
  --vault-name "kv-datalake-secrets" \
  --name "sp-client-secret" \
  --value "$SP_CLIENT_SECRET"

az keyvault secret set \
  --vault-name "kv-datalake-secrets" \
  --name "tenant-id" \
  --value "$SP_TENANT_ID"

# Assign role on ADLS Gen2 to the Service Principal
STORAGE_SCOPE="/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/DataRG/providers/Microsoft.Storage/storageAccounts/salesdatalake"
az role assignment create \
  --assignee "$SP_CLIENT_ID" \
  --role "Storage Blob Data Contributor" \
  --scope "$STORAGE_SCOPE"

echo "Service Principal configured and secrets stored in Key Vault"

5.4 Create a Databricks Secret Scope Pointing to Key Vault

# Create Secret Scope via Databricks API
# Navigate to: https://<workspace>.azuredatabricks.net/#secrets/createScope

# OR via Databricks CLI
databricks secrets create-scope \
  --scope "kv-datalake-secrets" \
  --scope-backend-type AZURE_KEYVAULT \
  --resource-id "/subscriptions/{sub-id}/resourceGroups/DataRG/providers/Microsoft.KeyVault/vaults/kv-datalake-secrets" \
  --dns-name "https://kv-datalake-secrets.vault.azure.net/"

# Verify the scope
databricks secrets list-scopes

# Usage in a Databricks notebook
# Secrets are automatically masked in logs and outputs

# Retrieve a secret
client_id = dbutils.secrets.get(scope="kv-datalake-secrets", key="sp-client-id")
print(f"Client ID retrieved: {client_id[:8]}...")  # Display only first 8 chars

# List available secrets (without showing values)
secrets = dbutils.secrets.list(scope="kv-datalake-secrets")
for s in secrets:
    print(f"  Available secret: {s.key}")

6. Connecting to ADLS Gen2 from Databricks

6.1 Via Service Principal and Key Vault

"""
Secure connection script to ADLS Gen2
Uses Service Principal + Azure Key Vault
"""

# Authentication configuration
def configure_adls_access(storage_account: str, secret_scope: str):
    """Configure access to ADLS Gen2 via Service Principal."""
    
    # Retrieve credentials from Key Vault via Databricks Secret Scope
    client_id = dbutils.secrets.get(scope=secret_scope, key="sp-client-id")
    client_secret = dbutils.secrets.get(scope=secret_scope, key="sp-client-secret")
    tenant_id = dbutils.secrets.get(scope=secret_scope, key="tenant-id")
    
    # Configuration prefix
    prefix = f"fs.azure.account"
    
    # Configure OAuth 2.0
    spark.conf.set(
        f"{prefix}.auth.type.{storage_account}.dfs.core.windows.net",
        "OAuth"
    )
    spark.conf.set(
        f"{prefix}.oauth.provider.type.{storage_account}.dfs.core.windows.net",
        "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
    )
    spark.conf.set(
        f"{prefix}.oauth2.client.id.{storage_account}.dfs.core.windows.net",
        client_id
    )
    spark.conf.set(
        f"{prefix}.oauth2.client.secret.{storage_account}.dfs.core.windows.net",
        client_secret
    )
    spark.conf.set(
        f"{prefix}.oauth2.client.endpoint.{storage_account}.dfs.core.windows.net",
        f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
    )
    
    print(f"✅ ADLS Gen2 access configured for: {storage_account}")
    return f"abfss://raw@{storage_account}.dfs.core.windows.net/"

# Call
STORAGE_ACCOUNT = "salesdatalake"
SECRET_SCOPE = "kv-datalake-secrets"
base_path = configure_adls_access(STORAGE_ACCOUNT, SECRET_SCOPE)

# Verify access
print("Contents of the raw container:")
for f in dbutils.fs.ls(base_path):
    print(f"  📁 {f.name} ({f.size} bytes)")

6.2 Via Unity Catalog (recommended for new architectures)

-- Step 1: Create a Storage Credential (Unity Catalog admin only)
CREATE STORAGE CREDENTIAL adls_gen2_credential
USING MANAGED IDENTITY;

-- Step 2: Verify the credential
SHOW CREDENTIALS;

-- Step 3: Create an External Location
CREATE EXTERNAL LOCATION raw_data
URL 'abfss://raw@salesdatalake.dfs.core.windows.net'
WITH (STORAGE CREDENTIAL adls_gen2_credential)
COMMENT 'Raw data - raw container';

-- Step 4: Validate access
VALIDATE STORAGE CREDENTIAL adls_gen2_credential
ON LOCATION 'abfss://raw@salesdatalake.dfs.core.windows.net';

-- Step 5: List external locations
SHOW EXTERNAL LOCATIONS;

# Reading with Unity Catalog — no access configuration needed
# The credential is managed transparently

# Read from external location
df = spark.read.json(
    "abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning_management.json"
)
df.show(10)

# Or create an external table pointing to ADLS
spark.sql("""
    CREATE TABLE IF NOT EXISTS hr_catalog.learning_schema.learning_management
    USING DELTA
    LOCATION 'abfss://raw@salesdatalake.dfs.core.windows.net/tables/learning_management/'
""")

7. Mount Points vs Direct Access

7.1 Comparing the Two Approaches

graph TB
    subgraph "Mount Point Access"
        M1[dbutils.fs.mount\n'Mount' a container] --> M2["/mnt/raw/\nSimplified path"]
        M2 --> M3["Easy reading\nspark.read.csv\n'/mnt/raw/file.csv')"]
        M4[✅ Short paths\n✅ Interactive exploration\n⚠️ Workspace-scoped\n❌ No AAD passthrough\n⚠️ Not recommended with Unity Catalog]
    end
    
    subgraph "Direct Access"
        D1[spark.read / spark.write\nFull ABFSS path] --> D2[Explicit authentication\nSP or Unity Catalog]
        D2 --> D3["Direct reading\nspark.read.csv\n'abfss://raw@...')"]
        D4[✅ Production-ready\n✅ Unity Catalog compatible\n✅ Fine-grained access\n✅ Audit logging\n⚠️ Longer paths]
    end

7.2 Create a Mount Point with Key Vault

def mount_adls_container(
    storage_account: str,
    container_name: str,
    mount_point: str,
    secret_scope: str
):
    """Mount an ADLS Gen2 container with OAuth authentication via Key Vault."""
    
    # Check if already mounted
    mounts = [m.mountPoint for m in dbutils.fs.mounts()]
    if mount_point in mounts:
        print(f"⚠️  Already mounted: {mount_point}")
        return
    
    # Retrieve credentials
    client_id = dbutils.secrets.get(scope=secret_scope, key="sp-client-id")
    client_secret = dbutils.secrets.get(scope=secret_scope, key="sp-client-secret")
    tenant_id = dbutils.secrets.get(scope=secret_scope, key="tenant-id")
    
    # OAuth configuration
    configs = {
        "fs.azure.account.auth.type": "OAuth",
        "fs.azure.account.oauth.provider.type": 
            "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
        "fs.azure.account.oauth2.client.id": client_id,
        "fs.azure.account.oauth2.client.secret": client_secret,
        "fs.azure.account.oauth2.client.endpoint": 
            f"https://login.microsoftonline.com/{tenant_id}/oauth2/token",
    }
    
    # Mount the container
    dbutils.fs.mount(
        source=f"abfss://{container_name}@{storage_account}.dfs.core.windows.net/",
        mount_point=mount_point,
        extra_configs=configs
    )
    
    print(f"✅ Container mounted: {container_name} → {mount_point}")

# Unmount a mount point
def unmount_adls(mount_point: str):
    """Unmount a mount point if it exists."""
    mounts = [m.mountPoint for m in dbutils.fs.mounts()]
    if mount_point in mounts:
        dbutils.fs.unmount(mount_point)
        print(f"✅ Unmounted: {mount_point}")
    else:
        print(f"⚠️  {mount_point} is not mounted")

# Usage
mount_adls_container(
    storage_account="salesdatalake",
    container_name="raw",
    mount_point="/mnt/raw",
    secret_scope="kv-datalake-secrets"
)

# Read with simplified path
df = spark.read.csv("/mnt/raw/marketing/campaign_data.csv", header=True)
df.show(5)

8. Troubleshooting Connectivity Issues

8.1 Common Errors and Solutions

8.2 Diagnostic Script

def diagnose_adls_connection(storage_account: str, container: str, secret_scope: str):
    """Complete ADLS Gen2 connection diagnostic."""
    
    print("=== ADLS Gen2 Connection Diagnostic ===\n")
    
    # 1. Check secrets
    print("1. Checking Key Vault secrets...")
    try:
        secrets = dbutils.secrets.list(scope=secret_scope)
        available_keys = [s.key for s in secrets]
        print(f"   ✅ Available secrets: {available_keys}")
        
        required = ["sp-client-id", "sp-client-secret", "tenant-id"]
        missing = [k for k in required if k not in available_keys]
        if missing:
            print(f"   ❌ Missing secrets: {missing}")
            return False
    except Exception as e:
        print(f"   ❌ Key Vault access error: {e}")
        return False
    
    # 2. Configure and test access
    print("\n2. Testing ADLS Gen2 connection...")
    try:
        configure_adls_access(storage_account, secret_scope)
        path = f"abfss://{container}@{storage_account}.dfs.core.windows.net/"
        files = dbutils.fs.ls(path)
        print(f"   ✅ Connection successful! {len(files)} objects found")
        for f in files[:5]:  # Show max 5 items
            print(f"      📄 {f.name}")
    except Exception as e:
        print(f"   ❌ Connection error: {e}")
        
        # Additional diagnostics
        if "AuthenticationFailed" in str(e):
            print("   💡 Action: Verify that credentials are not expired")
        elif "Forbidden" in str(e) or "403" in str(e):
            print("   💡 Action: Verify that SP has the 'Storage Blob Data Contributor' role")
        elif "Connection" in str(e):
            print("   💡 Action: Check Key Vault and ADLS firewall rules")
        return False
    
    # 3. Test a read
    print("\n3. Testing file read...")
    try:
        files = dbutils.fs.ls(path)
        if files:
            first_file = files[0].path
            if first_file.endswith('/'):
                sub_files = dbutils.fs.ls(first_file)
                if sub_files:
                    first_file = sub_files[0].path
            
            if first_file.endswith(('.csv', '.json', '.parquet')):
                df = spark.read.option("header", "true").csv(first_file) \
                    if first_file.endswith('.csv') \
                    else spark.read.json(first_file)
                row_count = df.count()
                print(f"   ✅ Read successful: {row_count} rows")
    except Exception as e:
        print(f"   ⚠️  Read warning: {e}")
    
    print("\n✅ Diagnostic complete - Connection functional")
    return True

# Run diagnostic
diagnose_adls_connection(
    storage_account="salesdatalake",
    container="raw",
    secret_scope="kv-datalake-secrets"
)

9. PySpark vs SQL — Decision Guide

9.1 Decision Matrix

graph TD
    Start[Which approach to choose?] --> Q1{Complex transformation\nwith conditions/UDFs?}
    Q1 -->|Yes| PySpark[PySpark]
    Q1 -->|No| Q2{Automated\npipeline?}
    Q2 -->|Yes| PySpark
    Q2 -->|No| Q3{Ad hoc query\nor BI dashboard?}
    Q3 -->|Yes| SQL[SQL]
    Q3 -->|No| Q4{Dynamic data\nor variable schema?}
    Q4 -->|Yes| PySpark
    Q4 -->|No| SQL

9.2 Detailed Comparison Table

9.3 Equivalent Code Examples

# Transformation with PySpark
from pyspark.sql import functions as F
from pyspark.sql.window import Window

# Calculate rank by department
window_spec = Window.partitionBy("department").orderBy(F.desc("salary"))

employees_ranked = employees_df \
    .withColumn("salary_rank", F.rank().over(window_spec)) \
    .withColumn("percentile_in_dept", 
                F.percent_rank().over(window_spec)) \
    .filter(F.col("salary_rank") <= 3)

employees_ranked.show()

-- Equivalent transformation in SQL
WITH ranked AS (
    SELECT 
        *,
        RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS salary_rank,
        PERCENT_RANK() OVER (PARTITION BY department ORDER BY salary DESC) AS percentile_in_dept
    FROM employees
)
SELECT * FROM ranked WHERE salary_rank <= 3;

10. Data Ingestion from ADLS Gen2

10.1 Read JSON and CSV from ADLS Gen2

from pyspark.sql import functions as F

# Read a JSON file (multiline — complete JSON object across multiple lines)
learning_df = (
    spark.read
    .format("json")
    .option("multiLine", "true")
    .option("mode", "PERMISSIVE")
    .option("columnNameOfCorruptRecord", "_corrupt_record")
    .load("abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning_management.json")
)

print(f"Automatically inferred schema:")
learning_df.printSchema()
print(f"Number of rows: {learning_df.count():,}")
learning_df.show(10, truncate=False)

# Read a CSV with all options
demographics_df = (
    spark.read
    .format("csv")
    .option("header", "true")
    .option("inferSchema", "true")
    .option("sep", ",")
    .option("quote", '"')
    .option("escape", "\\")
    .option("encoding", "UTF-8")
    .option("nullValue", "NULL")
    .option("emptyValue", "")
    .load("abfss://raw@salesdatalake.dfs.core.windows.net/hr/demographics.csv")
)

demographics_df.printSchema()
print(f"Demographic data: {demographics_df.count():,} rows")

10.2 Read Nested JSON Data

# JSON with nested structure (common for API data)
complex_json_df = spark.read.json(
    "abfss://raw@salesdatalake.dfs.core.windows.net/marketing/campaign_data.json"
)

# Display nested schema
complex_json_df.printSchema()

# Flatten nested structures
flattened_df = complex_json_df.select(
    F.col("campaign_id"),
    F.col("campaign_name"),
    F.col("metrics.impressions").alias("impressions"),
    F.col("metrics.clicks").alias("clicks"),
    F.col("metrics.conversions").alias("conversions"),
    F.col("audience.age_group").alias("age_group"),
    F.col("audience.region").alias("region"),
    F.explode("targeting_keywords").alias("keyword")  # Explode arrays
)

flattened_df.show(10, truncate=False)

11. Schemas: Inference and Evolution

11.1 Automatic Inference vs Manual Schema

from pyspark.sql.types import *

# ❌ OPTION 1: InferSchema — slow and potentially incorrect
df_inferred = spark.read.option("inferSchema", "true") \
    .csv("abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning.csv", 
         header=True)

# ✅ OPTION 2: Manual schema — fast and reliable for production
learning_schema = StructType([
    StructField("employee_id",    IntegerType(),   nullable=False),
    StructField("employee_name",  StringType(),    nullable=True),
    StructField("department",     StringType(),    nullable=True),
    StructField("course_id",      StringType(),    nullable=False),
    StructField("course_name",    StringType(),    nullable=True),
    StructField("completion_date", DateType(),     nullable=True),
    StructField("score",          DoubleType(),    nullable=True),
    StructField("certification",  BooleanType(),   nullable=True),
    StructField("country",        StringType(),    nullable=True),
])

df_schema = spark.read.schema(learning_schema) \
    .csv("abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning.csv",
         header=True)

print("With manual schema:")
df_schema.printSchema()

11.2 mergeSchema and rescueDataColumn

# mergeSchema — Delta schema evolution
df_new_data = spark.read.json(
    "abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning_v2.json"
)

# Write with schema merge
df_new_data.write \
    .format("delta") \
    .mode("append") \
    .option("mergeSchema", "true") \
    .save("abfss://raw@salesdatalake.dfs.core.windows.net/tables/learning/")

print("Schema evolved successfully — new columns added")

# rescueDataColumn — capture unexpected fields
df_rescued = (
    spark.read
    .format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("rescuedDataColumn", "_rescued_data")  
    .option("cloudFiles.schemaLocation",
            "abfss://raw@salesdatalake.dfs.core.windows.net/schema/learning/")
    .load("abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning_stream/")
)

# Analyze rescued fields
rescued_fields = df_rescued.filter(F.col("_rescued_data").isNotNull())
print(f"Records with unknown fields: {rescued_fields.count()}")
rescued_fields.select("_rescued_data").show(truncate=False)

12. Auto Loader — Incremental Ingestion

12.1 Auto Loader Architecture

graph LR
    subgraph "ADLS Gen2"
        DIR[Source directory\n/hr/learning_stream/]
        F1[file_2024_01.json]
        F2[file_2024_02.json]
        F3[file_2024_03.json\nNEW]
        DIR --> F1 & F2 & F3
    end
    
    subgraph "Auto Loader"
        AL[CloudFiles\nformat: cloudFiles]
        CL[Checkpoint\nLocation]
        Schema[Schema Location\nPersistent inference]
        AL --> CL
        AL --> Schema
    end
    
    subgraph "Output"
        DT[Delta Table\nhr_catalog.learning.courses]
    end
    
    DIR -->|New file detected| AL
    F1 & F2 -->|Already processed\nIgnored| AL
    F3 -->|Processed| AL
    AL -->|writeStream| DT
    
    note[✅ Exactly once\n✅ Auto-inferred schema\n✅ Fault-tolerant]

12.2 Complete Auto Loader Pipeline

from pyspark.sql import functions as F

def create_autoloader_pipeline(
    source_path: str,
    checkpoint_path: str,
    schema_path: str,
    target_table: str,
    file_format: str = "json"
):
    """
    Create an Auto Loader pipeline for incremental ingestion.
    
    Args:
        source_path: ADLS Gen2 path of source directory
        checkpoint_path: Path to store stream state
        schema_path: Path to store/evolve inferred schema
        target_table: Target Delta table (catalog.schema.table)
        file_format: File format (json, csv, parquet, avro)
    """
    
    # Auto Loader stream configuration
    stream = (
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", file_format)
        .option("cloudFiles.schemaLocation", schema_path)  # Persists schema
        .option("cloudFiles.inferColumnTypes", "true")     # Type inference
        .option("cloudFiles.maxFilesPerTrigger", 1000)     # Limit per batch
        .option("mergeSchema", "true")                      # Schema evolution
        .option("rescuedDataColumn", "_rescued_data")       # Capture unknowns
        .load(source_path)
    )
    
    # Add ingestion metadata
    stream_enriched = stream \
        .withColumn("_ingestion_time", F.current_timestamp()) \
        .withColumn("_source_file", F.input_file_name())
    
    # Stream write to Delta table
    query = (
        stream_enriched.writeStream
        .format("delta")
        .outputMode("append")
        .option("checkpointLocation", checkpoint_path)
        .option("mergeSchema", "true")
        .trigger(availableNow=True)  # Process all available, then stop
        .toTable(target_table)
    )
    
    # Wait for processing to complete
    query.awaitTermination()
    print(f"✅ Auto Loader complete → {target_table}")
    return query

# Start the pipeline
create_autoloader_pipeline(
    source_path="abfss://raw@salesdatalake.dfs.core.windows.net/hr/learning_stream/",
    checkpoint_path="abfss://raw@salesdatalake.dfs.core.windows.net/checkpoints/learning/",
    schema_path="abfss://raw@salesdatalake.dfs.core.windows.net/schema/learning/",
    target_table="hr_catalog.learning_schema.courses_ingested",
    file_format="json"
)

12.3 Auto Loader Trigger Modes

13. Optimization: Parquet vs Delta Lake

13.1 Why Delta Lake Is Superior for Production

import time

base_path = "abfss://raw@salesdatalake.dfs.core.windows.net/"

# Write same data in Parquet and Delta for comparison
data_df = spark.read.json(
    f"{base_path}hr/learning_management.json"
).dropna()

# Parquet write
start = time.time()
data_df.write \
    .mode("overwrite") \
    .partitionBy("department") \
    .format("parquet") \
    .save(f"{base_path}output/learning_parquet/")
parquet_write_time = time.time() - start
print(f"Parquet write time: {parquet_write_time:.2f}s")

# Delta write
start = time.time()
data_df.write \
    .mode("overwrite") \
    .partitionBy("department") \
    .format("delta") \
    .save(f"{base_path}output/learning_delta/")
delta_write_time = time.time() - start
print(f"Delta write time: {delta_write_time:.2f}s")

# Read and performance comparison
start = time.time()
parquet_count = spark.read.parquet(f"{base_path}output/learning_parquet/").count()
parquet_read_time = time.time() - start

start = time.time()
delta_count = spark.read.format("delta").load(f"{base_path}output/learning_delta/").count()
delta_read_time = time.time() - start

print(f"\n=== Comparative Results ===")
print(f"Parquet - Read: {parquet_read_time:.2f}s ({parquet_count} rows)")
print(f"Delta   - Read: {delta_read_time:.2f}s ({delta_count} rows)")
print(f"\nDelta is ~{parquet_read_time/delta_read_time:.1f}x faster")
print("(thanks to Data Skipping, Column Statistics, and Transaction Log)")

13.2 Partitioning and Compression

# Partitioning best practices
# ✅ Partition on columns with medium cardinality
data_df.write \
    .mode("overwrite") \
    .partitionBy("country", "year", "month") \
    .format("delta") \
    .save(f"{base_path}output/learning_partitioned/")

# ❌ Avoid: partition on a very high cardinality column
# data_df.write.partitionBy("employee_id")  # Too many small files!

# ZSTD compression (best compression/speed ratio)
spark.conf.set("spark.sql.parquet.compression.codec", "zstd")

# Or via Table Property
spark.sql("""
    ALTER TABLE hr_catalog.learning_schema.courses
    SET TBLPROPERTIES ('delta.parquet.vorder.enabled' = 'true')
""")

14. Unity Catalog — Centralized Governance

14.1 Unity Catalog Hierarchical Model

graph TB
    UC[Unity Catalog Metastore\nOne per Azure organization] --> C1[hr_catalog]
    UC --> C2[marketing_catalog]
    UC --> C3[finance_catalog]
    UC --> C4[hive_metastore\nlegacy]
    
    C1 --> S1[learning_schema]
    C1 --> S2[workforce_schema]
    C1 --> S3[raw_schema]
    
    S1 --> T1[Table: courses]
    S1 --> T2[Table: completions]
    S1 --> V1[View: vw_active_employees]
    
    S2 --> T3[Table: demographics]
    S2 --> T4[Table: org_chart]
    
    subgraph "Permissions"
        P1[GRANT SELECT ON TABLE]
        P2[GRANT ALL ON SCHEMA]
        P3[Row Filter Functions]
        P4[Column Masks]
    end

14.2 Advantages vs Legacy Hive Metastore

15. Configuring Unity Catalog in a Workspace

15.1 Create Unity Catalog Resources

# Admin notebook - Create Unity Catalog structure

# Create a domain-specific catalog
spark.sql("""
    CREATE CATALOG IF NOT EXISTS hr_catalog
    COMMENT 'HR data catalog'
""")

spark.sql("USE CATALOG hr_catalog")

# Create schemas
spark.sql("""
    CREATE SCHEMA IF NOT EXISTS hr_catalog.learning_schema
    COMMENT 'Schema for training data'
    LOCATION 'abfss://processed@salesdatalake.dfs.core.windows.net/hr/'
""")

spark.sql("""
    CREATE SCHEMA IF NOT EXISTS hr_catalog.workforce_schema
    COMMENT 'Schema for workforce data'
""")

# Create external table pointing to ADLS Gen2
spark.sql("""
    CREATE TABLE IF NOT EXISTS hr_catalog.workforce_schema.demographics
    USING DELTA
    LOCATION 'abfss://raw@salesdatalake.dfs.core.windows.net/tables/demographics/'
    COMMENT 'Employee demographic data'
""")

# Verify structure
spark.sql("SHOW CATALOGS").show()
spark.sql("SHOW SCHEMAS IN hr_catalog").show()
spark.sql("SHOW TABLES IN hr_catalog.workforce_schema").show()

15.2 Create a Table via PySpark and Register in Unity Catalog

from pyspark.sql.types import *

# Read data from ADLS Gen2
demographics_df = spark.read.csv(
    "abfss://raw@salesdatalake.dfs.core.windows.net/hr/demographics.csv",
    header=True,
    inferSchema=True
)

# Add metadata
demographics_clean = demographics_df \
    .withColumn("ingestion_date", F.current_date()) \
    .withColumn("source_system", F.lit("hr_demographics_v1"))

# Save as Delta table in Unity Catalog
demographics_clean.write \
    .mode("overwrite") \
    .format("delta") \
    .option("path", "abfss://raw@salesdatalake.dfs.core.windows.net/tables/demographics/") \
    .saveAsTable("hr_catalog.workforce_schema.demographics")

print("Table created in Unity Catalog:")
spark.sql("DESCRIBE TABLE EXTENDED hr_catalog.workforce_schema.demographics").show(30, False)

16. Sharing Data Between Workspaces

16.1 Cross-Workspace Sharing Architecture

graph LR
    subgraph "Primary Workspace (main-databricks)"
        UC1[Unity Catalog\nhr_catalog]
        T1[Table: demographics]
        T2[Table: courses]
        UC1 --> T1 & T2
    end
    
    subgraph "Secondary Workspace (hr-ws)"
        UC2[Unity Catalog\nRead access]
        UC2 -->|Read| T1 & T2
    end
    
    subgraph "Finance Workspace (finance-ws)"
        UC3[Unity Catalog\nRestricted access]
        UC3 -->|Read allowed columns| T1
    end
    
    Admin[Unity Catalog Administrator] -->|Assign workspace| UC1
    Admin -->|GRANT SELECT| UC2 & UC3

16.2 Share a Catalog Between Workspaces

-- In Unity Catalog interface > Catalog > hr_catalog > Workspaces
-- OR via SQL (Unity Catalog admin required)

-- Once shared, in secondary workspace:
SHOW CATALOGS;  -- View shared catalogs

USE CATALOG hr_catalog;
SELECT * FROM workforce_schema.demographics LIMIT 10;

-- Create a local view on shared data
CREATE VIEW IF NOT EXISTS local_schema.employee_summary AS
SELECT 
    department,
    country,
    COUNT(*) AS employee_count,
    AVG(age) AS avg_age
FROM hr_catalog.workforce_schema.demographics
GROUP BY department, country;

17. Data Lineage with Unity Catalog

17.1 Automatic Lineage Capture

Unity Catalog automatically captures column-level lineage — who reads what, who creates what:

graph LR
    S1[demographics\nSource table] -->|Columns: gender, country| V1[vw_male_employees\nFiltered view]
    V1 -->|Inherited columns| V2[vw_usa_male_employees\nChained view]
    
    S2[courses\nSource table] -->|Columns: employee_id, score| AGG1[vw_avg_scores\nAggregation]
    
    V1 & AGG1 -->|Join| FINAL[report_employees_performance\nFinal report]
    
    style S1 fill:#e8f5e9
    style S2 fill:#e8f5e9
    style V1 fill:#fff3e0
    style V2 fill:#fff3e0
    style AGG1 fill:#fff3e0
    style FINAL fill:#fce4ec

17.2 Create Views and View Their Lineage

-- Create a filtered view
CREATE OR REPLACE VIEW hr_catalog.workforce_schema.vw_male_employees AS
SELECT 
    employee_id,
    employee_name,
    department,
    country,
    age
FROM hr_catalog.workforce_schema.demographics
WHERE gender = 'Male';

-- Create a derived view (chained lineage)
CREATE OR REPLACE VIEW hr_catalog.workforce_schema.vw_usa_employees AS
SELECT 
    employee_id,
    employee_name,
    department,
    age
FROM hr_catalog.workforce_schema.vw_male_employees
WHERE country = 'USA';

-- Check lineage in the interface:
-- Unity Catalog > hr_catalog > workforce_schema > vw_male_employees > Lineage
-- OR via SQL:
DESCRIBE EXTENDED hr_catalog.workforce_schema.vw_male_employees;

# Access lineage via Unity Catalog API
from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Get table lineage (beta API)
lineage = w.table_constraints.list("hr_catalog.workforce_schema.demographics")
for constraint in lineage:
    print(f"Constraint: {constraint}")

18. Access Control Models Compared

18.1 Evolution of Access Control in Databricks

timeline
    title Evolution of Databricks Access Control
    
    "Before Unity Catalog" : ADLS ACLs at storage level
                           : Table ACLs workspace-only
                           : Credential Passthrough
                           : Manual configuration per workspace
    
    "With Unity Catalog" : Centralized multi-workspace governance
                        : Column-level security
                        : Row-level security via Row Filters
                        : Column Masks for PII
                        : Automatic audit logs
                        : Column-to-column lineage

18.2 Model Comparison Table

18.3 Unity Catalog Grants

-- Grant granular permissions
-- On a complete catalog
GRANT USE CATALOG ON CATALOG hr_catalog TO `data-analysts@company.com`;
GRANT USE SCHEMA ON SCHEMA hr_catalog.learning_schema TO `data-analysts@company.com`;
GRANT SELECT ON TABLE hr_catalog.learning_schema.courses TO `data-analysts@company.com`;

-- On a schema
GRANT ALL PRIVILEGES ON SCHEMA hr_catalog.learning_schema TO `hr-admins@company.com`;

-- Create a restricted view for a group
CREATE VIEW hr_catalog.workforce_schema.vw_safe_demographics AS
SELECT 
    employee_id,
    department,
    country,
    -- Hide sensitive information
    REGEXP_REPLACE(employee_name, '.', 'X') AS employee_name_masked,
    age
FROM hr_catalog.workforce_schema.demographics;

-- Grant access to the view (not the source table)
GRANT SELECT ON VIEW hr_catalog.workforce_schema.vw_safe_demographics 
TO `external-partners@company.com`;

-- View granted permissions
SHOW GRANTS ON TABLE hr_catalog.workforce_schema.demographics;

19. Delta Sharing — Cross-Organization Sharing

19.1 Delta Sharing Architecture

graph LR
    subgraph "Provider Organization (your organization)"
        UC[Unity Catalog]
        Share[Delta Share\n'hr_external_share']
        T1[Table: demographics\nVersion 10]
        T2[Table: courses]
        UC --> T1 & T2
        Share --> T1 & T2
    end
    
    subgraph "Recipient (partner organization)"
        R1[Databricks Workspace\nDatabricks Recipient]
        R2[Power BI\nOpen Recipient]
        R3[Pandas/Spark\nOpen Recipient]
        
        Token[Bearer Token\nSecured]
        Token -->|Authenticates| R1 & R2 & R3
    end
    
    Share -->|Live Data\nNo copy!| R1 & R2 & R3

19.2 Configure Delta Sharing

# Prerequisites: Unity Catalog with Delta Sharing enabled in Account Console

# Create a Share (via Unity Catalog interface > Delta Sharing > Shared by me)
# OR via SQL:

-- Create a share
CREATE SHARE IF NOT EXISTS hr_external_share
COMMENT 'HR data sharing with external partners';

-- Add tables to the share
ALTER SHARE hr_external_share ADD TABLE hr_catalog.workforce_schema.demographics
PARTITIONS (country IN ('USA', 'Canada'));  -- Share only specific partitions

ALTER SHARE hr_external_share ADD TABLE hr_catalog.learning_schema.courses;

-- Create an external recipient (non-Databricks)
CREATE RECIPIENT IF NOT EXISTS external_partner_john
USING ID 'your-sharing-id@delta.io'
COMMENT 'External partner - HR data access';

-- Grant access to the share
GRANT SELECT ON SHARE hr_external_share TO RECIPIENT external_partner_john;

-- View created shares
SHOW SHARES;
SHOW RECIPIENTS;

19.3 Differences Between Delta Sharing and Cross-Workspace Sharing

20. Fine-Grained Permissions: Rows and Columns

20.1 Row Filters — Row-Level Filters

-- Create a row filter function
-- Only one filter per table
CREATE OR REPLACE FUNCTION hr_catalog.workforce_schema.country_filter(country STRING)
RETURNS BOOLEAN
RETURN 
    CASE 
        WHEN IS_ACCOUNT_GROUP_MEMBER('hr-admins') THEN TRUE  -- Admins see everything
        WHEN CURRENT_USER() = 'global-analyst@company.com' THEN TRUE  -- Privileged user
        ELSE country = 'USA'  -- Others only see US employees
    END;

-- Apply the filter to the table
ALTER TABLE hr_catalog.workforce_schema.demographics
SET ROW FILTER hr_catalog.workforce_schema.country_filter ON (country);

-- Verify applied filter
DESCRIBE TABLE EXTENDED hr_catalog.workforce_schema.demographics;

-- Test: a regular user only sees USA employees
SELECT country, COUNT(*) FROM hr_catalog.workforce_schema.demographics GROUP BY country;
-- → Returns only USA

-- Remove the filter (admin only)
ALTER TABLE hr_catalog.workforce_schema.demographics DROP ROW FILTER;

20.2 Column Masks — Sensitive Column Masking

-- Create a masking function for names
CREATE OR REPLACE FUNCTION hr_catalog.workforce_schema.mask_employee_name(name STRING)
RETURNS STRING
RETURN 
    CASE 
        WHEN IS_ACCOUNT_GROUP_MEMBER('hr-admins') THEN name  -- Admins: full name
        ELSE REGEXP_REPLACE(name, '(\\w+)$', 'XXXXXX')  -- Others: mask last name
    END;

-- Apply the mask to the column
ALTER TABLE hr_catalog.workforce_schema.demographics
ALTER COLUMN employee_name
SET MASK hr_catalog.workforce_schema.mask_employee_name;

-- Test: a non-admin user sees masked names
SELECT employee_id, employee_name FROM hr_catalog.workforce_schema.demographics LIMIT 5;
-- → John XXXXXX, Marie XXXXXX, etc.

-- Full masking for highly sensitive columns (SSN, etc.)
CREATE OR REPLACE FUNCTION hr_catalog.workforce_schema.hide_ssn(ssn STRING)
RETURNS STRING
RETURN 
    CASE 
        WHEN IS_ACCOUNT_GROUP_MEMBER('payroll-admins') THEN ssn
        ELSE '***-**-****'
    END;

20.3 Testing Fine-Grained Permissions

# Test script to validate permissions
def test_row_and_column_security():
    """Test Row Filters and Column Masks."""
    
    print("=== Fine-Grained Security Test ===\n")
    
    # 1. Test Row Filter
    print("1. Row Filter test (country_filter):")
    all_countries = spark.sql("""
        SELECT country, COUNT(*) AS num_employees
        FROM hr_catalog.workforce_schema.demographics
        GROUP BY country
        ORDER BY country
    """).collect()
    
    for row in all_countries:
        print(f"   {row.country}: {row.num_employees} visible employees")
    
    # 2. Test Column Mask
    print("\n2. Column Mask test (mask_employee_name):")
    sample = spark.sql("""
        SELECT employee_id, employee_name
        FROM hr_catalog.workforce_schema.demographics
        LIMIT 5
    """).collect()
    
    for row in sample:
        print(f"   ID {row.employee_id}: {row.employee_name}")
    
    print("\n✅ Test complete - Verify that masks are properly applied")

test_row_and_column_security()

21. Summary and Best Practices

21.1 Recommended End-to-End Architecture

graph TB
    subgraph "Security"
        KV[Azure Key Vault\nSecrets]
        SP[Service Principal\nor Managed Identity]
        UC[Unity Catalog\nGovernance]
    end
    
    subgraph "Ingestion"
        AL[Auto Loader\nIncremental]
        DirectR[Direct Read\nBatch]
    end
    
    subgraph "Storage (ADLS Gen2)"
        RAW["/raw container/\nRaw data"]
        PROC["/processed container/\nProcessed data"]
        OUTPUT["/output container/\nBusiness data"]
    end
    
    subgraph "Databricks"
        Bronze[Bronze Table\nRaw Delta]
        Silver[Silver Table\nClean Delta]
        Gold[Gold Table\nAggregated Delta]
        
        RF[Row Filters]
        CM[Column Masks]
    end
    
    SP -->|Auth| RAW
    KV -->|Credentials| SP
    AL & DirectR --> RAW --> Bronze
    Bronze --> Silver --> Gold
    UC --> RF & CM
    RF & CM --> Gold
    Gold --> OUTPUT

21.2 Security and Governance Checklist

22. Glossary

Search Terms

manage · data · azure · databricks · lake · spark · engineering · analytics · catalog · gen2 · adls · architecture · unity · delta · sharing · vault · access · auto · loader · authentication · between · control · integration · lineage