Security and Compliance in Azure Databricks

Level: Intermediate / Advanced | Platform: Azure Databricks Premium

Table of Contents

1. Databricks Security Architecture
2. Shared Responsibility Model
3. Identity and Access Management (IAM)
4. Role-Based Access Control (RBAC)
5. Single Sign-On (SSO) and MFA
6. Network Security — Private Link and VNet Injection
7. Encryption at Rest and In Transit
8. Azure Key Vault — Secret Management
9. Secure Access to Azure Storage
10. Unity Catalog — Data Governance
11. Granular Access Control
12. Row-Level Security and Column Masking
13. Audit Logging and Compliance
14. GDPR and HIPAA — Regulatory Compliance
15. Security Monitoring with Azure Monitor
16. Azure Defender for Databricks
17. Securing Notebooks and Jobs
18. Cluster Policies — Secure Standardization
19. Enterprise Security Checklist
20. Summary and Best Practices
21. Glossary

1. Databricks Security Architecture

1.1 Secure Architecture Overview

graph TB
    subgraph "Network Level"
        PVTLINK[Private Link\nNo internet exposure]
        VNET[VNet Injection\nPrivate client network]
        FW[Azure Firewall\nFiltering]
    end
    
    subgraph "Identity Level"
        AAD[Microsoft Entra ID\nAuthentication]
        SCIM[SCIM\nAuto provisioning]
        MFA[MFA\nTwo-factor authentication]
        SSO[SSO\nSingle Sign-On]
    end
    
    subgraph "Access Level"
        RBAC[RBAC\nRoles and permissions]
        UC[Unity Catalog\nCentralized governance]
        RLS[Row-Level Security\nRow filters]
        CM[Column Masking\nPII masking]
    end
    
    subgraph "Data Level"
        ENC[AES-256 Encryption\nAt rest and in transit]
        KV[Azure Key Vault\nKey management]
        CMK[Customer Managed Keys\nClient keys]
    end
    
    subgraph "Audit Level"
        AL[Audit Logs\nAll accesses]
        DEF[Azure Defender\nThreat Detection]
        MON[Azure Monitor\nReal-time alerts]
    end

1.2 The Two Levels of Databricks Security

2. Shared Responsibility Model

2.1 Who is Responsible for What?

graph LR
    subgraph "Microsoft Azure"
        AZURE[Infrastructure security\n• Physical datacenters\n• Network hardware\n• Hypervisor\n• Azure network layer]
    end
    
    subgraph "Databricks Inc."
        DATABRICKS[Platform security\n• Data encryption\n• Access Control Layer\n• Identity Management\n• Spark security patches]
    end
    
    subgraph "Your Organization (Client)"
        CLIENT[Workload security\n• User access management\n• Policy configuration\n• Workload monitoring\n• Data compliance\n• Secrets and credentials]
    end
    
    AZURE --> DATABRICKS --> CLIENT

2.2 Detailed Responsibility Matrix

3. Identity and Access Management (IAM)

3.1 Microsoft Entra ID (Azure Active Directory) Integration

sequenceDiagram
    participant User as User
    participant DB as Databricks
    participant AAD as Microsoft Entra ID
    participant KV as Key Vault
    participant ADLS as ADLS Gen2

    User->>DB: Connection attempt
    DB->>AAD: Redirect to Entra ID
    AAD->>User: Login page (SSO)
    User->>AAD: Credentials + MFA
    AAD-->>DB: OAuth 2.0 / JWT Token
    DB->>DB: Validate token, load permissions
    DB-->>User: Access granted based on roles
    
    User->>DB: Access a Delta table
    DB->>AAD: Check Unity Catalog permissions
    AAD-->>DB: Permissions validated
    DB->>ADLS: Read data (via Managed Identity)
    ADLS-->>DB: Data returned
    DB-->>User: Query results

3.2 Automatic Provisioning with SCIM

SCIM (System for Cross-domain Identity Management) automates user provisioning:

# Configure SCIM Provisioning from Azure Portal
# 1. Azure AD → Enterprise Applications
# 2. Search "Azure Databricks SCIM Provisioning Connector"
# 3. Configure the Databricks tenant URL
# 4. Add the Personal Access Token of a Databricks admin

# Databricks Tenant URL for SCIM
# https://adb-xxxx.azuredatabricks.net/api/2.0/preview/scim

What SCIM manages automatically:

4. Role-Based Access Control (RBAC)

4.1 Databricks Permission Hierarchy

graph TB
    subgraph "Account Level (Unity Catalog)"
        MA[Metastore Admin\nFull access to everything]
        CA[Catalog Admin\nManage a catalog]
    end
    
    subgraph "Workspace Level"
        WA[Workspace Admin\nAdminister the workspace]
        WU[Workspace User\nUse the workspace]
    end
    
    subgraph "Resource Level"
        CP[Can Manage\nManage + use]
        CR[Can Restart\nRestart only]
        CAT[Can Attach To\nAttach notebooks]
    end
    
    MA --> CA --> WA --> WU
    WA --> CP --> CR --> CAT

4.2 Fundamental Security Principles

# PRINCIPLE 1: Least Privilege
# ✅ Do: Grant only necessary permissions
spark.sql("""
    GRANT SELECT ON TABLE taxicatalog.rides.yellow_taxis 
    TO `analyst-group`
""")

# ❌ Avoid: Giving overly broad access
spark.sql("""
    GRANT ALL PRIVILEGES ON CATALOG taxicatalog 
    TO `analyst-group`  -- Too permissive!
""")

# PRINCIPLE 2: Separation of Duties
spark.sql("GRANT SELECT ON SCHEMA taxicatalog.rides TO `data-analysts`")
spark.sql("GRANT MODIFY ON SCHEMA taxicatalog.rides TO `data-engineers`")
spark.sql("GRANT MANAGE ON SCHEMA taxicatalog.rides TO `schema-admins`")

# PRINCIPLE 3: Groups rather than individual users
# ✅ Do: Assign to groups
spark.sql("GRANT SELECT ON TABLE taxicatalog.rides.yellow_taxis TO `finance-team`")

4.3 Recommended RBAC Matrix by Role

5. Single Sign-On (SSO) and MFA

5.1 SSO with Microsoft Entra ID

Databricks uses Microsoft Entra ID natively — no manual SAML/OIDC configuration is required:

graph LR
    User[User] -->|1. Access Databricks| DB[Databricks UI]
    DB -->|2. Redirect to AAD| AAD[Microsoft Entra ID]
    AAD -->|3. Verify identity| User
    User -->|4. Credentials + MFA| AAD
    AAD -->|5. JWT Token| DB
    DB -->|6. Access granted| User
    
    note1[✅ Same credentials as\nMicrosoft 365, Azure, Teams\n→ Unified experience]

5.2 Configure MFA with Conditional Access

# Configuration via Microsoft Entra admin center:
# Identity → Protection → Conditional Access → Create new policy

# Recommended settings:
# Name: "Databricks - Require MFA"
# Assignments:
#   - Users: All users (or specific Databricks users group)
#   - Cloud apps: Azure Databricks
# Conditions:
#   - Locations: Any location (or exclude corporate network)
# Grant:
#   - Grant access
#   - Require multifactor authentication  ✅
# Enable policy: ON

6. Network Security — Private Link and VNet Injection

6.1 Secure Network Architecture

graph TB
    subgraph "Standard Architecture (Public)"
        Internet[Public Internet] <-->|Exposed traffic| DB_STD[Standard Databricks]
        DB_STD --> ADLS_STD[ADLS Gen2]
    end
    
    subgraph "Secure Architecture (Private)"
        subgraph "Client VNet"
            subgraph "Databricks Subnets"
                Public[Public Subnet\nCluster nodes]
                Private[Private Subnet\nIsolated cluster nodes]
            end
            PE[Private Endpoint\nDatabricks Workspace]
            PE2[Private Endpoint\nADLS Gen2]
        end
        
        Corp[Corporate network] -->|VPN/ExpressRoute| PE
        PE --> Public & Private
        Private --> PE2
        PE2 --> ADLS[ADLS Gen2]
        
        note2[✅ No internet traffic\n✅ Total isolation\n✅ DLP possible]
    end

6.2 Azure Private Link — Configuration

# Create a Private Endpoint for the Databricks workspace
az network private-endpoint create \
  --name "databricks-private-endpoint" \
  --resource-group "RG" \
  --vnet-name "my-vnet" \
  --subnet "databricks-private-subnet" \
  --private-connection-resource-id "$WORKSPACE_ID" \
  --group-id "databricks_ui_api" \
  --connection-name "databricks-private-connection" \
  --location "eastus2"

# Create a Private DNS Zone to resolve Databricks names
az network private-dns zone create \
  --resource-group "RG" \
  --name "privatelink.azuredatabricks.net"

# Link the DNS Zone to the VNet
az network private-dns link vnet create \
  --resource-group "RG" \
  --zone-name "privatelink.azuredatabricks.net" \
  --name "databricks-dns-link" \
  --virtual-network "my-vnet" \
  --registration-enabled false

7. Encryption at Rest and In Transit

7.1 Encryption at Rest

7.2 Customer Managed Keys (CMK)

# Configure Customer Managed Keys for Databricks
# (Requires Databricks Premium)

# 1. Create Key Vault with deletion protection
az keyvault create \
  --name "databricks-cmk-vault" \
  --resource-group "RG" \
  --location "eastus2" \
  --enable-soft-delete true \
  --enable-purge-protection true

# 2. Create encryption key
az keyvault key create \
  --vault-name "databricks-cmk-vault" \
  --name "databricks-encryption-key" \
  --kty RSA \
  --size 2048

# 3. Assign permissions to the Databricks workspace
WORKSPACE_PRINCIPAL_ID=$(az databricks workspace show \
  --name "my-workspace" \
  --resource-group "RG" \
  --query "identity.principalId" -o tsv)

az keyvault set-policy \
  --name "databricks-cmk-vault" \
  --object-id "$WORKSPACE_PRINCIPAL_ID" \
  --key-permissions get wrapKey unwrapKey

# 4. Configure Databricks to use CMK
az databricks workspace update \
  --name "my-workspace" \
  --resource-group "RG" \
  --key-vault-uri "https://databricks-cmk-vault.vault.azure.net/" \
  --key-name "databricks-encryption-key" \
  --key-version "latest"

8. Azure Key Vault — Secret Management

8.1 Secure Secrets Architecture

graph LR
    subgraph "Azure Key Vault"
        K1[Secret: adls-access-key]
        K2[Secret: sql-password]
        K3[Secret: api-token]
        K4[Secret: sp-client-secret]
    end
    
    subgraph "Databricks"
        SS[Secret Scope\nkv-databricks-secrets]
        NB[Notebook Code]
        SS -->|Masked in logs| NB
    end
    
    NB -->|dbutils.secrets.get| SS
    SS -->|Secure API| K1 & K2 & K3 & K4
    
    note[✅ Never hardcoded secrets\n✅ Centralized rotation\n✅ Access audit]

8.2 Key Vault Best Practices

# ✅ GOOD PRACTICE: Retrieve from Key Vault
storage_account = "taxidatadls"
client_id = dbutils.secrets.get(scope="kv-databricks-secrets", key="sp-client-id")
client_secret = dbutils.secrets.get(scope="kv-databricks-secrets", key="sp-client-secret")
tenant_id = dbutils.secrets.get(scope="kv-databricks-secrets", key="tenant-id")

# Values are automatically masked in outputs: [REDACTED]
print(f"Client ID: {client_id}")  # Displays: [REDACTED]

# ❌ BAD PRACTICE: Hardcoding secrets
# client_secret = "abc123xyz"  # Visible in source code and logs!

# Secure Spark configuration
spark.conf.set(
    f"fs.azure.account.auth.type.{storage_account}.dfs.core.windows.net",
    "OAuth"
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.id.{storage_account}.dfs.core.windows.net",
    client_id
)
spark.conf.set(
    f"fs.azure.account.oauth2.client.secret.{storage_account}.dfs.core.windows.net",
    client_secret  # Never shown in logs
)

9. Secure Access to Azure Storage

9.1 Secure Access Methods Comparison

9.2 Mounting ADLS Gen2 Securely

def secure_mount_adls(
    storage_account: str,
    container: str,
    mount_point: str,
    secret_scope: str
):
    """Mounts ADLS Gen2 securely via OAuth + Key Vault."""
    
    existing_mounts = [m.mountPoint for m in dbutils.fs.mounts()]
    if mount_point in existing_mounts:
        print(f"ℹ️  Already mounted: {mount_point}")
        return
    
    # Retrieve credentials from Key Vault (never hardcoded!)
    configs = {
        "fs.azure.account.auth.type": "OAuth",
        "fs.azure.account.oauth.provider.type":
            "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
        "fs.azure.account.oauth2.client.id":
            dbutils.secrets.get(scope=secret_scope, key="sp-client-id"),
        "fs.azure.account.oauth2.client.secret":
            dbutils.secrets.get(scope=secret_scope, key="sp-client-secret"),
        "fs.azure.account.oauth2.client.endpoint":
            f"https://login.microsoftonline.com/{dbutils.secrets.get(scope=secret_scope, key='tenant-id')}/oauth2/token"
    }
    
    dbutils.fs.mount(
        source=f"abfss://{container}@{storage_account}.dfs.core.windows.net/",
        mount_point=mount_point,
        extra_configs=configs
    )
    
    print(f"✅ Successfully mounted: {container} → {mount_point}")

# Usage
secure_mount_adls(
    storage_account="taxidatadls",
    container="raw",
    mount_point="/mnt/raw",
    secret_scope="kv-databricks-secrets"
)

10. Unity Catalog — Data Governance

10.1 Unity Catalog Security Architecture

graph TB
    subgraph "Unity Catalog — Central Source of Truth"
        MS[Metastore\nOrganization Central]
        
        subgraph "hr_catalog"
            S1[workforce_schema]
            S2[learning_schema]
            T1[Table: demographics\n🔐 PII data]
            T2[Table: salaries\n🔐 Confidential]
            S1 --> T1 & T2
        end
        
        subgraph "taxicatalog"
            S3[rides_schema]
            T3[Table: yellow_taxis\n📊 Analytics]
            S3 --> T3
        end
        
        MS --> hr_catalog & taxicatalog
    end
    
    subgraph "Governance"
        ACL[Fine-grained ACLs\nColumn and row]
        LIN[Data Lineage\nColumn traceability]
        AUD[Audit Logs\nWho accesses what]
        TAG[Data Tags\nPII, Confidential]
    end
    
    MS --> ACL & LIN & AUD & TAG

10.2 Compliance Tags Policy

-- Create tags to identify sensitive data

-- Tag a column as PII
ALTER TABLE hr_catalog.workforce_schema.demographics
ALTER COLUMN ssn
SET TAGS ('pii' = 'true', 'classification' = 'confidential', 'gdpr_applicable' = 'true');

-- Tag a table as containing HIPAA data
ALTER TABLE hr_catalog.workforce_schema.medical_records
SET TAGS (
    'hipaa_applicable' = 'true',
    'data_classification' = 'restricted',
    'retention_years' = '7'
);

-- List all PII columns
SELECT 
    table_catalog,
    table_schema,
    table_name,
    column_name,
    tag_name,
    tag_value
FROM system.information_schema.column_tags
WHERE tag_name = 'pii' AND tag_value = 'true'
ORDER BY table_catalog, table_schema, table_name;

11. Granular Access Control

11.1 GRANT and REVOKE in Unity Catalog

-- ═══════════════════════════════════════════════════
-- GRANT: Grant permissions
-- ═══════════════════════════════════════════════════

-- On a catalog
GRANT USE CATALOG ON CATALOG hr_catalog TO `data-analysts`;
GRANT CREATE ON CATALOG hr_catalog TO `data-engineers`;

-- On a schema
GRANT USE SCHEMA ON SCHEMA hr_catalog.workforce_schema TO `data-analysts`;
GRANT CREATE TABLE ON SCHEMA hr_catalog.workforce_schema TO `data-engineers`;

-- On a specific table
GRANT SELECT ON TABLE hr_catalog.workforce_schema.demographics TO `data-analysts`;
GRANT MODIFY ON TABLE hr_catalog.workforce_schema.demographics TO `data-engineers`;

-- On a view
GRANT SELECT ON VIEW hr_catalog.workforce_schema.vw_safe_demographics TO `external-partners`;

-- ═══════════════════════════════════════════════════
-- REVOKE: Revoke permissions
-- ═══════════════════════════════════════════════════

-- Revoke access to a table
REVOKE SELECT ON TABLE hr_catalog.workforce_schema.salaries FROM `contractors`;

-- ═══════════════════════════════════════════════════
-- SHOW GRANTS: Audit permissions
-- ═══════════════════════════════════════════════════

-- View all permissions on a table
SHOW GRANTS ON TABLE hr_catalog.workforce_schema.demographics;

-- View permissions of a group
SHOW GRANTS TO `data-analysts`;

12. Row-Level Security and Column Masking

12.1 Row-Level Security with Row Filters

-- Create a Row Filter function
CREATE OR REPLACE FUNCTION hr_catalog.security.row_filter_by_department(
    dept_col STRING
)
RETURNS BOOLEAN
RETURN 
    is_account_group_member('hr-admins')  -- HR admins see everything
    OR current_user() = 'data-admin@company.com'  -- Specific admin
    OR dept_col = (
        SELECT department 
        FROM hr_catalog.workforce_schema.user_departments 
        WHERE user_email = current_user()
    );

-- Apply row filter to a table
ALTER TABLE hr_catalog.workforce_schema.employees
SET ROW FILTER hr_catalog.security.row_filter_by_department ON (department);

-- Test the filter (each user only sees their own rows)
SELECT * FROM hr_catalog.workforce_schema.employees;

12.2 Column Masking for PII

-- Create a masking function for Social Security Numbers
CREATE OR REPLACE FUNCTION hr_catalog.security.mask_ssn(ssn STRING)
RETURNS STRING
RETURN 
    CASE 
        WHEN is_account_group_member('hr-managers') THEN ssn  -- Managers see full
        ELSE CONCAT('***-**-', RIGHT(ssn, 4))  -- Others see masked
    END;

-- Apply mask to the SSN column
ALTER TABLE hr_catalog.workforce_schema.employees
ALTER COLUMN ssn
SET MASK hr_catalog.security.mask_ssn;

13. Audit Logging and Compliance

13.1 Configure Diagnostic Logs

# Enable diagnostic logs for Databricks
az monitor diagnostic-settings create \
  --name "databricks-audit-logs" \
  --resource "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Databricks/workspaces/my-workspace" \
  --logs '[{"category": "accounts", "enabled": true}, {"category": "clusters", "enabled": true}, {"category": "workspace", "enabled": true}, {"category": "notebook", "enabled": true}, {"category": "secrets", "enabled": true}]' \
  --workspace "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.OperationalInsights/workspaces/my-workspace"

13.2 Audit Queries

-- Monitor access to sensitive data
SELECT
    timestamp,
    userIdentity.email AS user_email,
    actionName AS action,
    requestParams.tableName AS table_accessed,
    requestParams.clusterId AS cluster_id
FROM system.access.audit
WHERE actionName IN ('runCommand', 'tableAccess')
AND (requestParams.tableName LIKE '%pii%' 
     OR requestParams.tableName LIKE '%sensitive%')
AND timestamp > NOW() - INTERVAL 7 DAYS
ORDER BY timestamp DESC;

14. GDPR and HIPAA — Regulatory Compliance

14.1 GDPR Implementation in Databricks

# Right to erasure (right to be forgotten)
def gdpr_right_to_erasure(user_id: str, affected_tables: list):
    """
    Implements the GDPR right to erasure.
    Deletes or anonymizes all personal data for a user.
    """
    audit_log = []
    
    for table in affected_tables:
        try:
            row_count = spark.sql(f"""
                SELECT COUNT(*) as cnt 
                FROM {table} 
                WHERE user_id = '{user_id}'
            """).collect()[0]["cnt"]
            
            if row_count > 0:
                spark.sql(f"""
                    DELETE FROM {table}
                    WHERE user_id = '{user_id}'
                """)
                
                audit_log.append({
                    "table": table,
                    "action": "DELETE",
                    "rows_deleted": row_count,
                    "timestamp": datetime.now().isoformat(),
                    "gdpr_request_id": str(uuid4())
                })
        
        except Exception as e:
            audit_log.append({
                "table": table,
                "action": "ERROR",
                "error": str(e),
                "timestamp": datetime.now().isoformat()
            })
    
    return audit_log

# Execute GDPR erasure
erasure_result = gdpr_right_to_erasure(
    user_id="user_12345",
    affected_tables=[
        "hr_catalog.workforce_schema.employees",
        "hr_catalog.learning_schema.training_records"
    ]
)

15. Security Monitoring with Azure Monitor

15.1 Security Alerts

# Create an alert for suspicious accesses
az monitor scheduled-query-rules create \
  --name "databricks-suspicious-access" \
  --resource-group "rg-databricks" \
  --condition-query "DatabricksAccounts | where Category == 'accounts' | where ActionName == 'login' | where ResultStatus == 'Failure' | summarize FailedLogins=count() by UserName, bin(TimeGenerated, 1h) | where FailedLogins > 5" \
  --condition-time-aggregation "Count" \
  --condition-operator "GreaterThan" \
  --condition-threshold 0 \
  --evaluation-frequency "PT15M" \
  --window-size "PT1H" \
  --severity 2

19. Enterprise Security Checklist

Core Security

21. Glossary

Search Terms

security · compliance · azure · databricks · spark · data · engineering · analytics · access · architecture · secure · catalog · unity · audit · column · configure · control · encryption · entra · gdpr · link · management · masking · matrix