Administering Clusters and Configuring Policies with Databricks

Level: Intermediate / Advanced

Table of Contents

1. Overview and Objectives
2. Databricks Architecture — Foundations
3. Databricks Cluster Types
4. Databricks Runtime (DBR) Versions
5. Autoscaling and Capacity Management
6. Cluster Modes: Standard vs High Concurrency
7. Defining Cluster Policies on Azure Databricks
8. Constraint Types in a Cluster Policy
9. Policy Deployment Best Practices
10. Demonstrations: Cluster Policies in Practice
11. Configuring Cluster Resource Access
12. Entitlements and Permissions on Resources
13. Instance Pools — Reducing Startup Time
14. Cluster Tags and Cost Attribution
15. Init Scripts — Cluster Customization
16. Spark Configuration Parameters
17. Termination Policies and Timeouts
18. Unity Catalog and Cluster Access
19. User and Group Management
20. Service Principals — Application Accounts
21. Azure Active Directory and SCIM Integration
22. Managing and Using Personal Access Tokens
23. Databricks REST API — Complete Reference
24. Databricks CLI — Command-Line Management
25. Terraform / Infrastructure as Code for Databricks
26. Summary and Key Points
27. Review Questions
28. Glossary

1. Overview and Objectives

1.1 Why Administer Databricks Clusters?

Administering a Databricks workspace involves a wide variety of tasks, but one of the most critical — and often overlooked — is cluster and cluster policy management. Without proper governance, the consequences can be severe:

• Budget overruns: Clusters left running without workloads can consume DBUs (Databricks Units) unnecessarily, generating exponential costs.
• Inconsistent environments: Different team members create clusters with very different configurations, making reproducibility impossible.
• Security risks: Unauthorized users may access sensitive data if permissions are not properly configured.
• Debugging difficulties: Without standardized runtimes and libraries, environment-related bugs become very hard to reproduce.

Cluster Policies address all these problems by defining rules that govern cluster creation. They act as configuration templates that users must follow, while still allowing controlled flexibility.

1.2 Services Available in a Databricks Workspace

A Databricks workspace is a container for all Azure Databricks assets. It comprises three main services:

graph LR
    WS[Databricks Workspace] --> SQL[Databricks SQL]
    WS --> DSE[Data Science & Engineering]
    WS --> ML[Databricks Machine Learning]

    SQL --> SQL1[SQL analytical queries]
    SQL --> SQL2[Visualizations and dashboards]
    SQL --> SQL3[SQL Warehouses]

    DSE --> DSE1[Interactive notebooks]
    DSE --> DSE2[All-Purpose / Job Clusters]
    DSE --> DSE3[Delta Lake / Unity Catalog]
    DSE --> DSE4[Workflows / Jobs]

    ML --> ML1[AutoML]
    ML --> ML2[Feature Store]
    ML --> ML3[MLflow Tracking]
    ML --> ML4[Model Registry]

1.3 Course Learning Objectives

By the end of this course, you will be able to:

1. Understand why cluster policies are necessary in an organization
2. Define and deploy cluster policies with different constraint types
3. Manage permissions on clusters, instance pools, and other Databricks resources
4. Administer users and groups in a Databricks workspace
5. Generate and use Personal Access Tokens for programmatic access
6. Automate cluster management via the Databricks REST API
7. Integrate Azure Active Directory with Databricks via SCIM
8. Apply enterprise administration best practices

2. Databricks Architecture — Foundations

2.1 Architecture Overview

Azure Databricks is built on a two-plane architecture (Control Plane / Data Plane) that clearly separates infrastructure management from data execution.

graph TB
    subgraph Azure["Microsoft Azure"]
        subgraph CP["Control Plane (managed by Databricks)"]
            UI[Web Interface / UI]
            API[REST API]
            JM[Job Manager]
            CM[Cluster Manager]
            DBFS[DBFS Metadata]
        end
        
        subgraph DP["Data Plane (your Azure subscription)"]
            subgraph RG["Client Resource Group"]
                VNet[Virtual Network]
                subgraph Cluster["Spark Cluster"]
                    Driver[Driver Node]
                    W1[Worker 1]
                    W2[Worker 2]
                    W3[Worker N]
                end
                ADLS[Azure Data Lake Storage]
                KeyVault[Azure Key Vault]
            end
        end
    end
    
    User([User]) --> UI
    User --> API
    CM --> Cluster
    Driver --> ADLS
    Driver --> KeyVault

2.2 Control Plane vs Data Plane

Important note: Data never passes through the Control Plane. It stays in your Data Plane, which is a major advantage for compliance (GDPR, HIPAA, etc.).

2.3 Spark Cluster Components

Each Databricks cluster consists of Apache Spark nodes with a master-slave architecture:

graph TB
    subgraph Cluster["Databricks Cluster"]
        subgraph Driver["Driver Node (Master)"]
            SC[SparkContext]
            DAG[DAG Scheduler]
            TS[Task Scheduler]
        end
        
        subgraph Executor1["Worker Node 1"]
            E1[Executor]
            T1[Task 1]
            T2[Task 2]
            Cache1[Cache / Memory]
        end
        
        subgraph Executor2["Worker Node 2"]
            E2[Executor]
            T3[Task 3]
            T4[Task 4]
            Cache2[Cache / Memory]
        end
        
        subgraph Executor3["Worker Node N"]
            E3[Executor]
            T5[Task N]
            Cache3[Cache / Memory]
        end
    end
    
    Driver --> Executor1
    Driver --> Executor2
    Driver --> Executor3

3. Databricks Cluster Types

3.1 Cluster Type Comparison Overview

A Databricks cluster is a set of virtual machines (VMs) running Apache Spark. Three main categories exist:

graph TD
    A[Workspace Databricks] --> B[All-Purpose Cluster]
    A --> C[Job Cluster]
    A --> D[SQL Warehouse]

    B --> B1[Interactive processing]
    B --> B2[Collaborative notebooks]
    B --> B3[Manual start/stop]
    B --> B4[Shareable among users]

    C --> C1[Batch / ETL processing]
    C --> C2[Auto-start when job launches]
    C --> C3[Auto-stop after job ends]
    C --> C4[Not reusable across jobs]

    D --> D1[Databricks SQL Analytics]
    D --> D2[SQL queries by analysts]
    D --> D3[Dashboards and visualizations]
    D --> D4[Serverless or Classic]

3.2 Detailed Comparison Table

3.3 All-Purpose Cluster — Complete Lifecycle

stateDiagram-v2
    [*] --> Pending : Create / Start
    Pending --> Running : Machines provisioned and Spark initialized
    Running --> Resizing : Autoscaling triggered
    Resizing --> Running : Capacity adjusted
    Running --> Terminating : Auto-termination / Manual stop
    Terminating --> Terminated : VMs fully stopped
    Terminated --> Pending : Restart
    Running --> Error : Configuration / network error
    Error --> Terminated : Auto-cleanup
    Pending --> Error : Provisioning failure

Typical transition times:

• Pending → Running: 5-15 minutes (cold start) or 1-2 minutes (with Instance Pool)
• Running → Terminated: 1-3 minutes
• Terminated → Running (restart): same as initial startup

3.4 Job Cluster — Automatic Lifecycle

The Job Cluster is automatically created when a Databricks job launches and destroyed at the end. Here is its complete lifecycle:

sequenceDiagram
    participant S as Databricks Scheduler
    participant CM as Cluster Manager
    participant Azure as Azure (VMs)
    participant Job as Job Runner

    S->>CM: Trigger job (schedule/trigger)
    CM->>Azure: Provision VMs according to config
    Azure-->>CM: VMs ready
    CM->>Job: Start job execution
    Job->>Job: Execute Spark tasks
    Job-->>CM: Job completed (success/failure)
    CM->>Azure: Release VMs
    Azure-->>CM: VMs deallocated
    CM-->>S: Result report

Key points about Job Clusters:

• A job cluster cannot be restarted after the job ends
• Configuration is defined in the job’s task definition
• Supports cluster policies to standardize production environments
• Cannot be shared across multiple simultaneous jobs
• Provides complete isolation: fresh environment for each run

3.5 SQL Warehouse — Variants and Sizing

The SQL Warehouse is a cluster optimized for analytical SQL queries. It comes in three variants:

T-shirt sizing for SQL Warehouses:

4. Databricks Runtime (DBR) Versions

4.1 What is the Databricks Runtime?

The Databricks Runtime (DBR) is a software layer that runs on clusters. It includes:

• Apache Spark (specific version, optimized by Databricks)
• Delta Lake (native Databricks optimizations)
• Python, R, Scala libraries pre-installed and tested
• Databricks optimizations (Photon, JVM optimizations, etc.)
• Connectors to Azure services (ADLS, Azure SQL, etc.)

4.2 Version Naming Convention

DBR 13.3 LTS
│    │  │
│    │  └── LTS = Long Term Support (minimum 2 years of support)
│    └───── 3 = minor version
└────────── 13 = Databricks major version

Recommendation: Always use an LTS version in production to ensure stability and long-term support.

4.3 Databricks Runtime Families

4.4 Example spark_version Values in Policies

{
  "spark_version": {
    "type": "fixed",
    "value": "13.3.x-scala2.12"
  }
}

Common values usable in cluster policies:

5. Autoscaling and Capacity Management

5.1 Autoscaling Principles

Autoscaling allows a Databricks cluster to automatically adjust the number of worker nodes based on workload. This avoids both under-sizing (jobs too slow) and over-sizing (unnecessary costs).

graph LR
    subgraph "Without Autoscaling"
        Fixed[Fixed cluster: 10 nodes] 
        Low[Low load] --> Fixed
        High[High load] --> Fixed
        Fixed --> Waste[Waste under low load]
        Fixed --> Bottleneck[Bottleneck under high load]
    end

    subgraph "With Autoscaling"
        Auto[Autoscaling cluster: 2-20 nodes]
        LowLoad[Low load] --> Auto
        HighLoad[High load] --> Auto
        Auto --> Scale2[2 active nodes]
        Auto --> Scale20[20 active nodes]
    end

5.2 Configuring Autoscaling in a Policy

{
  "autoscale.min_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 5,
    "defaultValue": 2
  },
  "autoscale.max_workers": {
    "type": "range",
    "minValue": 2,
    "maxValue": 20,
    "defaultValue": 10
  }
}

5.3 Enhanced Autoscaling

Since DBR 10.4+, Databricks offers a smarter Enhanced Autoscaling:

5.4 Fixed Worker Count vs Autoscaling

# Example JSON configuration of a cluster with autoscaling
cluster_config_autoscaling = {
    "cluster_name": "autoscale-cluster",
    "spark_version": "13.3.x-scala2.12",
    "node_type_id": "Standard_DS3_v2",
    "autoscale": {
        "min_workers": 2,
        "max_workers": 10
    },
    "autotermination_minutes": 30
}

# Example with fixed worker count
cluster_config_fixed = {
    "cluster_name": "fixed-cluster",
    "spark_version": "13.3.x-scala2.12",
    "node_type_id": "Standard_DS3_v2",
    "num_workers": 4,
    "autotermination_minutes": 60
}

6. Cluster Modes: Standard vs High Concurrency

6.1 Cluster Mode Comparison

Important: With Unity Catalog (the new Databricks governance), cluster modes are evolving. Unity Catalog recommends Single User clusters with data access managed via the Catalog, rather than Table ACLs in High Concurrency mode.

6.2 Constraining Mode via a Policy

{
  "cluster_type": {
    "type": "fixed",
    "value": "dbu"
  },
  "data_security_mode": {
    "type": "fixed",
    "value": "SINGLE_USER"
  }
}

To enforce High Concurrency mode with Table ACL:

{
  "spark_conf.spark.databricks.repl.allowedLanguages": {
    "type": "fixed",
    "value": "python,sql",
    "hidden": true
  },
  "spark_conf.spark.databricks.acl.dfAclsEnabled": {
    "type": "fixed",
    "value": "true",
    "hidden": true
  }
}

7. Defining Cluster Policies on Azure Databricks

7.1 Why Cluster Policies?

Without cluster policies, any user with the allow-cluster-create permission can create a cluster with any configuration. This causes several major enterprise issues:

mindmap
  root((Problems without Policies))
    Uncontrolled costs
      Oversized clusters
      Clusters never stopped
      Unnecessary premium DBU types
    Inconsistency
      Different DBR versions
      Incompatible libraries
      Ad hoc configurations
    Security
      Ungoverned data access
      Exposed credentials
      No mandatory tags
    Compliance
      Difficult auditing
      Impossible cost attribution
      No traceability

Solution: Cluster Policies define configurations that users must follow. They can:

• Force a specific value (e.g.: DBR version)
• Restrict possible values (e.g.: VM size from a list)
• Define default values modifiable by the user
• Hide attributes to simplify the interface

7.2 Anatomy of a Cluster Policy

A cluster policy is defined in JSON. Here is its general structure:

{
  "cluster_attribute": {
    "type": "constraint_type",
    "value": "value_or_list",
    "hidden": false,
    "defaultValue": "default_value"
  }
}

Complete example of a production policy:

{
  "spark_version": {
    "type": "fixed",
    "value": "13.3.x-scala2.12",
    "hidden": false
  },
  "node_type_id": {
    "type": "allowlist",
    "values": [
      "Standard_DS3_v2",
      "Standard_DS4_v2",
      "Standard_DS5_v2"
    ],
    "defaultValue": "Standard_DS3_v2"
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 60,
    "hidden": true
  },
  "autoscale.min_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 5,
    "defaultValue": 2
  },
  "autoscale.max_workers": {
    "type": "range",
    "minValue": 2,
    "maxValue": 20,
    "defaultValue": 8
  },
  "custom_tags.team": {
    "type": "fixed",
    "value": "data-engineering",
    "hidden": true
  },
  "custom_tags.cost_center": {
    "type": "regex",
    "pattern": "CC-[0-9]{4}",
    "hidden": false
  }
}

8. Constraint Types in a Cluster Policy

8.1 Constraint Types Overview

Databricks offers several constraint types usable in policies:

8.2 fixed Type — Enforced Value

{
  "spark_version": {
    "type": "fixed",
    "value": "13.3.x-scala2.12"
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 30,
    "hidden": true
  }
}

The hidden: true attribute hides a field from the user in the cluster creation interface. This is useful for:

• Avoiding overloading users with technical information
• Hiding internal configuration details
• Applying “invisible” restrictions for A/B testing

8.3 allowlist Type — Whitelist

{
  "node_type_id": {
    "type": "allowlist",
    "values": [
      "Standard_DS3_v2",
      "Standard_DS4_v2",
      "Standard_DS5_v2"
    ],
    "defaultValue": "Standard_DS3_v2"
  }
}

The allowlist is useful for limiting choices to VM types approved by the infrastructure department, preventing costly GPUs or very large VMs.

8.4 blocklist Type — Blacklist

{
  "spark_version": {
    "type": "blocklist",
    "values": [
      "9.0.x-scala2.12",
      "9.1.x-scala2.12",
      "10.0.x-scala2.12"
    ]
  }
}

The blocklist is useful for blocking specific versions with known CVEs or documented bugs, while allowing any other version.

8.5 range Type — Numeric Range

{
  "num_workers": {
    "type": "range",
    "minValue": 1,
    "maxValue": 10,
    "defaultValue": 3
  }
}

Ideal for controlling cluster size without fixing it, but limiting it to a reasonable range.

8.6 regex Type — Regular Expression

{
  "custom_tags.cost_center": {
    "type": "regex",
    "pattern": "^CC-[0-9]{4}$"
  },
  "cluster_name": {
    "type": "regex",
    "pattern": "^(dev|staging|prod)-[a-z0-9-]{3,50}$"
  }
}

Regex allows enforcing naming formats, very useful for cost tags or standardized cluster names.

8.7 Commonly Configured Attributes in Policies

graph LR
    Policy[Cluster Policy] --> A["spark_version\n(DBR version)"]
    Policy --> B["node_type_id\n(VM type)"]
    Policy --> C["driver_node_type_id\n(Driver VM type)"]
    Policy --> D["autotermination_minutes\n(Auto-stop)"]
    Policy --> E["num_workers / autoscale\n(Cluster size)"]
    Policy --> F["spark_conf.*\n(Spark config)"]
    Policy --> G["custom_tags.*\n(Cost tags)"]
    Policy --> H["data_security_mode\n(Security mode)"]
    Policy --> I["init_scripts\n(Init scripts)"]

9. Policy Deployment Best Practices

9.1 Three-Step Process

When deploying cluster policies in an organization, it is recommended to follow three steps:

flowchart LR
    A[1. Communication] --> B[2. Testing]
    B --> C[3. Deployment]

    A --> A1[Announce policies]
    A --> A2[Explain motivations]
    A --> A3[Show impact on workflows]

    B --> B1[Test with pilot group]
    B --> B2[Verify edge cases]
    B --> B3[Collect feedback]
    B --> B4[Adjust if necessary]

    C --> C1[Deploy by team]
    C --> C2[User training]
    C --> C3[Support and guidance]

9.2 Step 1: Communication

During the communication phase, it is essential to:

1. Announce policy details: Which attributes are controlled? What are the allowed values?
2. Explain motivations: Cost control? Compliance? Environment standardization?
3. Show impact: How will new workflows be affected? Which clusters will need to be migrated?
4. Offer a transition period: Don’t force immediately, allow a reasonable delay.

Tip: Prepare a clear migration document with before/after examples for each role (data engineer, data scientist, analyst).

9.3 Step 2: Testing

Before production deployment, test policies with:

• A pilot group of 2-5 volunteer users
• Edge cases (very long jobs, very large clusters, atypical configurations)
• Error scenarios (what happens if a user tries to circumvent the policy?)

9.4 Step 3: Deployment

Deployment should be progressive:

9.5 Organizing Policies by Role

// Policy for Data Engineers
{
  "spark_version": {
    "type": "allowlist",
    "values": ["13.3.x-scala2.12", "14.3.x-scala2.12"],
    "defaultValue": "13.3.x-scala2.12"
  },
  "node_type_id": {
    "type": "allowlist",
    "values": ["Standard_DS3_v2", "Standard_DS4_v2", "Standard_DS5_v2"]
  },
  "autoscale.max_workers": {
    "type": "range",
    "maxValue": 20
  },
  "autotermination_minutes": {
    "type": "range",
    "minValue": 10,
    "maxValue": 120,
    "defaultValue": 60
  },
  "custom_tags.role": {
    "type": "fixed",
    "value": "data-engineering",
    "hidden": true
  }
}

// Policy for Data Scientists
{
  "spark_version": {
    "type": "allowlist",
    "values": [
      "13.3.x-cpu-ml-scala2.12",
      "14.3.x-cpu-ml-scala2.12",
      "13.3.x-gpu-ml-scala2.12"
    ]
  },
  "node_type_id": {
    "type": "allowlist",
    "values": [
      "Standard_DS3_v2",
      "Standard_DS4_v2",
      "Standard_NC6s_v3",
      "Standard_NC12s_v3"
    ]
  },
  "autoscale.max_workers": {
    "type": "range",
    "maxValue": 8
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 60,
    "hidden": true
  },
  "custom_tags.role": {
    "type": "fixed",
    "value": "data-science",
    "hidden": true
  }
}

10. Demonstrations: Cluster Policies in Practice

10.1 Creating a Simple Cluster Policy (fixed)

Steps to create a simple policy via the Databricks interface:

1. In the left menu, navigate to Compute → Cluster Policies
2. Click Create Policy
3. Give it a name (e.g.: cp-runtime-visible)
4. Enter the JSON definition:

{
  "spark_version": {
    "type": "fixed",
    "value": "13.3.x-scala2.12"
  }
}

5. Click Create

Effect: When a user creates a cluster with this policy, the DBR version is automatically set to 13.3.x-scala2.12 and the user can see this value.

10.2 Creating a Policy with Hidden Attributes

{
  "spark_version": {
    "type": "fixed",
    "value": "13.3.x-scala2.12",
    "hidden": true
  },
  "autotermination_minutes": {
    "type": "fixed",
    "value": 30,
    "hidden": true
  }
}

Effect: The user creates a cluster without seeing either the DBR version or the auto-termination. These values are applied silently.

10.3 Policy with Allowlist and Default Value

{
  "spark_version": {
    "type": "allowlist",
    "values": [
      "13.3.x-scala2.12",
      "14.3.x-scala2.12",
      "15.4.x-scala2.12"
    ],
    "defaultValue": "13.3.x-scala2.12"
  }
}

Effect: The user only sees these 3 versions in the dropdown. The default version is already selected.

10.4 Policy with Blocklist

{
  "spark_version": {
    "type": "blocklist",
    "values": [
      "9.0.x-scala2.12",
      "9.1.x-scala2.12",
      "10.0.x-scala2.12"
    ]
  }
}

Effect: All DBR versions are available except the 3 listed. Useful for blocking EOL versions or versions with CVEs.

10.5 Assigning Permissions on a Cluster Policy

Policy permissions determine who can use the policy to create clusters:

graph LR
    Admin[Databricks Admin] -->|Creates the policy| Policy[Cluster Policy]
    Policy -->|Can Use| Group1[Group: devs]
    Policy -->|Can Use| Group2[Group: analysts]
    Group1 --> User1[User A]
    Group1 --> User2[User B]
    Group2 --> User3[User C]
    User1 -->|Creates cluster from policy| Cluster[Created Cluster]

Via UI:

1. Go to Cluster Policies → Select the policy
2. Click Permissions
3. Add the group/user with the Can Use permission
4. Save

Best practice: Always assign permissions to groups rather than individual users. This simplifies management during team rotations.

10.6 Entitlements Required to Use Policies

For a user to create clusters via policies, they must have:

A user without allow-cluster-create but with Can Use on a policy can create clusters only via that policy.

11. Configuring Cluster Resource Access

11.1 Permission Levels in Databricks

Access to Databricks resources is governed by a multi-level permission system:

graph TB
    subgraph "Workspace Level"
        WA[workspace-access]
        CC[allow-cluster-create]
        SQL[databricks-sql-access]
        IP[allow-instance-pool-create]
    end
    
    subgraph "Resource Level"
        subgraph Cluster["Cluster Permissions"]
            CM2[Can Manage]
            CR[Can Restart]
            CA[Can Attach To]
        end
        
        subgraph Pool["Instance Pool Permissions"]
            PM[Can Manage]
            PU[Can Attach To]
        end
        
        subgraph Policy["Policy Permissions"]
            PUse[Can Use]
        end
    end
    
    WA --> Cluster
    CC --> Cluster
    WA --> Pool
    IP --> Pool

11.2 Access Control Lists (ACL) — Complete Matrix

Cluster Permissions

Instance Pool Permissions

Cluster Policy Permissions

11.3 Practical Configuration Examples via REST API

# Set permissions on a cluster via API
CLUSTER_ID="0123-456789-abcdef"
TOKEN="your_token_here"
WORKSPACE_URL="https://adb-xxxx.azuredatabricks.net"

curl -X PUT "${WORKSPACE_URL}/api/2.0/permissions/clusters/${CLUSTER_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "access_control_list": [
      {
        "group_name": "devs",
        "permission_level": "CAN_MANAGE"
      },
      {
        "group_name": "analysts",
        "permission_level": "CAN_ATTACH_TO"
      }
    ]
  }'

# Python equivalent using the requests library
import requests
import json

workspace_url = "https://adb-xxxx.azuredatabricks.net"
token = "your_token_here"
cluster_id = "0123-456789-abcdef"

headers = {
    "Authorization": f"Bearer {token}",
    "Content-Type": "application/json"
}

permissions = {
    "access_control_list": [
        {"group_name": "devs", "permission_level": "CAN_MANAGE"},
        {"group_name": "analysts", "permission_level": "CAN_ATTACH_TO"},
        {"user_name": "admin@example.com", "permission_level": "CAN_MANAGE"}
    ]
}

response = requests.put(
    f"{workspace_url}/api/2.0/permissions/clusters/{cluster_id}",
    headers=headers,
    json=permissions
)

if response.status_code == 200:
    print("Permissions updated successfully")
else:
    print(f"Error: {response.status_code} - {response.text}")

12. Entitlements and Permissions on Resources

12.1 Workspace-Level Entitlements

Entitlements are rights that can be granted to users, service principals, and groups:

graph LR
    User[User / Group] --> E1[workspace-access\nAccess to workspace]
    User --> E2[allow-cluster-create\nCreate clusters freely]
    User --> E3[databricks-sql-access\nAccess Databricks SQL]
    User --> E4[allow-instance-pool-create\nCreate instance pools]

    E1 --> R1[Connect to the interface]
    E2 --> R2[Create any cluster type]
    E3 --> R3[Run SQL queries]
    E4 --> R4[Manage machine pools]

12.2 Assigning Entitlements via the SCIM API

# Assign the allow-cluster-create entitlement to a group
GROUP_ID="xxxx-yyyy-zzzz"

curl -X PATCH "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Groups/${GROUP_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
    "Operations": [
      {
        "op": "add",
        "path": "entitlements",
        "value": [
          {"value": "allow-cluster-create"}
        ]
      }
    ]
  }'

# Revoke the allow-cluster-create entitlement from a group
curl -X PATCH "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Groups/${GROUP_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
    "Operations": [
      {
        "op": "remove",
        "path": "entitlements[value eq \"allow-cluster-create\"]"
      }
    ]
  }'

12.3 Recommended Entitlements Matrix by Role

13. Instance Pools — Reducing Startup Time

13.1 What is an Instance Pool?

An Instance Pool is a set of virtual machines pre-allocated and maintained in a standby (idle) state in Azure. When a cluster needs new VMs, it takes them from this pool instead of provisioning them on demand.

graph LR
    subgraph "Without Instance Pool"
        J1[New Job] --> Provision[Provision VM\n5-15 minutes]
        Provision --> Ready[VM Ready]
        Ready --> Execute[Execute]
    end

    subgraph "With Instance Pool"
        J2[New Job] --> Pool[Instance Pool\nVMs already available]
        Pool --> Execute2[Execute\n1-2 minutes]
    end

13.2 Instance Pool Advantages

13.3 Instance Pool Configuration

{
  "instance_pool_name": "production-pool",
  "min_idle_instances": 2,
  "max_capacity": 20,
  "node_type_id": "Standard_DS3_v2",
  "preloaded_spark_versions": ["13.3.x-scala2.12"],
  "idle_instance_autotermination_minutes": 60,
  "azure_attributes": {
    "availability": "SPOT_WITH_FALLBACK_AZURE",
    "spot_bid_max_price": -1
  }
}

13.4 Using an Instance Pool in a Cluster Policy

{
  "instance_pool_id": {
    "type": "fixed",
    "value": "0123-456789-pool1",
    "hidden": true
  },
  "driver_instance_pool_id": {
    "type": "fixed",
    "value": "0123-456789-pool1",
    "hidden": true
  }
}

13.5 Creating an Instance Pool via REST API

# Create an instance pool
curl -X POST "${WORKSPACE_URL}/api/2.0/instance-pools/create" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "instance_pool_name": "production-pool",
    "min_idle_instances": 2,
    "max_capacity": 20,
    "node_type_id": "Standard_DS3_v2",
    "preloaded_spark_versions": ["13.3.x-scala2.12"],
    "idle_instance_autotermination_minutes": 60
  }'

14. Cluster Tags and Cost Attribution

14.1 Why Tags on Clusters?

Tags allow attributing cluster costs to the right teams, projects, or cost centers. In enterprises, this is essential for:

• Chargeback: billing costs to user teams
• Showback: showing teams their consumption without billing
• Auditing: identifying who uses what and for what purpose
• Optimization: spotting unnecessary spending

14.2 Tag Levels in Databricks

14.3 Enforcing Tags via a Cluster Policy

{
  "custom_tags.team": {
    "type": "fixed",
    "value": "data-engineering",
    "hidden": true
  },
  "custom_tags.cost_center": {
    "type": "regex",
    "pattern": "^CC-[0-9]{4}$",
    "hidden": false
  },
  "custom_tags.environment": {
    "type": "allowlist",
    "values": ["dev", "staging", "production"],
    "defaultValue": "dev"
  },
  "custom_tags.project": {
    "type": "unlimited",
    "defaultValue": "undefined"
  }
}

14.4 Visualizing Costs with Tags

# PySpark example: analyze costs by team from billing logs
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.getOrCreate()

# Read Azure billing data (exported to ADLS)
billing_df = spark.read.parquet(
    "abfss://billing@mystorageaccount.dfs.core.windows.net/azure-costs/"
)

# Filter on Databricks costs
databricks_costs = billing_df.filter(
    F.col("ServiceName").isin(["Azure Databricks", "Virtual Machines"])
)

# Aggregate by team (custom tag)
team_costs = (
    databricks_costs
    .groupBy(
        F.col("Tags.team").alias("team"),
        F.col("Tags.cost_center").alias("cost_center"),
        F.col("Tags.environment").alias("environment"),
        F.date_trunc("month", F.col("Date")).alias("month")
    )
    .agg(
        F.sum("Cost").alias("total_cost_usd"),
        F.count("*").alias("num_resources")
    )
    .orderBy("month", "team")
)

team_costs.show(50, truncate=False)

15. Init Scripts — Cluster Customization

15.1 What is an Init Script?

An Init Script is a shell script that runs on each node of a Databricks cluster before Spark starts. It allows:

• Installing system packages (apt-get, yum)
• Configuring environment variables
• Installing custom drivers
• Downloading configuration files
• Configuring connections to external systems

15.2 Init Script Types

15.3 Init Script Examples

Python package installation script:

#!/bin/bash
# init_script_python_packages.sh

set -e  # Stop on error

echo "=== Installing additional Python packages ==="

# Install required pip packages
/databricks/python/bin/pip install \
    great-expectations==0.18.0 \
    pandera==0.18.0 \
    pyarrow==14.0.0 \
    --quiet

echo "=== Installation complete ==="

SSL certificate configuration script:

#!/bin/bash
# init_script_ssl_certs.sh

# Copy custom CA certificates
cat /dbfs/company-certs/internal-ca.crt >> /etc/ssl/certs/ca-certificates.crt

# Set environment variables for certificates
echo "REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt" >> /etc/environment
echo "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" >> /etc/environment

echo "SSL certificates configured successfully"

15.4 Configuring an Init Script in a Policy

{
  "init_scripts.0.dbfs.destination": {
    "type": "fixed",
    "value": "dbfs:/company/init-scripts/standard-init.sh",
    "hidden": true
  }
}

16. Spark Configuration Parameters

16.1 Important Spark Configurations

{
  "spark_conf.spark.sql.adaptive.enabled": {
    "type": "fixed",
    "value": "true",
    "hidden": true
  },
  "spark_conf.spark.sql.shuffle.partitions": {
    "type": "range",
    "minValue": 100,
    "maxValue": 2000,
    "defaultValue": 200
  },
  "spark_conf.spark.databricks.delta.optimizeWrite.enabled": {
    "type": "fixed",
    "value": "true",
    "hidden": true
  }
}

16.2 Recommended Spark Configurations for Production

16.3 Advanced Configuration Example via API

import requests

# Complete production cluster configuration
cluster_config = {
    "cluster_name": "prod-etl-cluster",
    "spark_version": "13.3.x-scala2.12",
    "node_type_id": "Standard_DS4_v2",
    "driver_node_type_id": "Standard_DS4_v2",
    "autoscale": {
        "min_workers": 2,
        "max_workers": 10
    },
    "autotermination_minutes": 30,
    "spark_conf": {
        "spark.sql.adaptive.enabled": "true",
        "spark.sql.adaptive.coalescePartitions.enabled": "true",
        "spark.databricks.delta.optimizeWrite.enabled": "true",
        "spark.databricks.delta.autoCompact.enabled": "true",
        "spark.sql.shuffle.partitions": "400"
    },
    "custom_tags": {
        "team": "data-engineering",
        "environment": "production",
        "cost_center": "CC-1234"
    },
    "policy_id": "E0004A97B2B5F000",
    "init_scripts": [
        {
            "dbfs": {
                "destination": "dbfs:/company/init-scripts/standard-init.sh"
            }
        }
    ]
}

response = requests.post(
    f"{workspace_url}/api/2.0/clusters/create",
    headers={"Authorization": f"Bearer {token}"},
    json=cluster_config
)

cluster_id = response.json().get("cluster_id")
print(f"Cluster created with ID: {cluster_id}")

17. Termination Policies and Timeouts

17.1 Auto-termination

Auto-termination is one of the most important features for cost control. It automatically stops a cluster after a defined inactivity period.

{
  "autotermination_minutes": {
    "type": "range",
    "minValue": 10,
    "maxValue": 60,
    "defaultValue": 30
  }
}

Golden rule: Always configure auto-termination on All-Purpose Clusters. A value of 30-60 minutes is generally a good compromise between cost and productivity.

17.2 Recommended Auto-termination by Usage

17.3 Enforcing Auto-termination in a Policy

{
  "autotermination_minutes": {
    "type": "fixed",
    "value": 30,
    "hidden": true
  }
}

With hidden: true, the user doesn’t even know this value is fixed. The cluster will automatically stop after 30 minutes of inactivity without the user being able to modify it.

18. Unity Catalog and Cluster Access

18.1 Impact of Unity Catalog on Clusters

Unity Catalog is Databricks’ centralized data governance solution. It fundamentally changes how clusters access data:

graph TB
    subgraph "Before Unity Catalog"
        Cluster1[Hive Metastore Cluster] --> Meta1[Per-workspace Hive Metastore]
        Meta1 --> Data1[Direct ADLS data]
        Cluster1 --> Mount[DBFS mount points]
    end
    
    subgraph "With Unity Catalog"
        Cluster2[Unity Catalog Cluster] --> UC[Unity Catalog\nCentralized Metastore]
        UC --> Data2[ADLS data via Storage Credentials]
        UC --> Govern[Centralized governance]
        Govern --> Audit[Full audit]
        Govern --> Lineage[Data lineage]
        Govern --> ABAC[Attribute-based access control]
    end

18.2 Data Security Mode and Unity Catalog

18.3 Enforcing Unity Catalog in a Policy

{
  "data_security_mode": {
    "type": "allowlist",
    "values": ["SINGLE_USER", "USER_ISOLATION"],
    "defaultValue": "SINGLE_USER"
  },
  "runtime_engine": {
    "type": "fixed",
    "value": "PHOTON"
  }
}

19. User and Group Management

19.1 Entities in Databricks

There are three entity types in a Databricks workspace:

graph LR
    subgraph "Databricks Entities"
        U[Users\nHumans\nAzure AD account]
        SP[Service Principals\nApplications/Scripts\nAPI/CLI access]
        G[Groups\nCollections of entities\nPermission management]
    end
    
    G --> U
    G --> SP
    G --> G
    
    U --> WS[Databricks Workspace]
    SP --> WS

19.2 Special Groups in Databricks

Two groups are automatically created when a workspace is created and cannot be deleted:

19.3 Creating and Managing Users via the SCIM API

# List all users
curl -X GET "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Users" \
  -H "Authorization: Bearer ${TOKEN}"

# Add a user
cat > user.json << 'EOF'
{
  "schemas": ["urn:ietf:params:scim:schemas:core:2.0:User"],
  "userName": "alice@example.com",
  "groups": [
    {"value": "DEV_GROUP_ID"}
  ],
  "entitlements": [
    {"value": "workspace-access"}
  ]
}
EOF

curl -X POST "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Users" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @user.json

# Delete a user (get ID first)
USER_ID="xxxx-yyyy-zzzz"
curl -X DELETE "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Users/${USER_ID}" \
  -H "Authorization: Bearer ${TOKEN}"

19.4 Creating and Managing Groups via the SCIM API

# Create a group with members
cat > group.json << 'EOF'
{
  "schemas": ["urn:ietf:params:scim:schemas:core:2.0:Group"],
  "displayName": "data-engineers",
  "members": [
    {"value": "USER_ID_1"},
    {"value": "USER_ID_2"}
  ]
}
EOF

curl -X POST "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Groups" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @group.json

# Add a member to an existing group
cat > add_member.json << 'EOF'
{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    {
      "op": "add",
      "path": "members",
      "value": [{"value": "NEW_MEMBER_ID"}]
    }
  ]
}
EOF

curl -X PATCH "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Groups/${GROUP_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @add_member.json

# Remove a member from a group
cat > remove_member.json << 'EOF'
{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    {
      "op": "remove",
      "path": "members[value eq \"MEMBER_ID_TO_REMOVE\"]"
    }
  ]
}
EOF

curl -X PATCH "${WORKSPACE_URL}/api/2.0/preview/scim/v2/Groups/${GROUP_ID}" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @remove_member.json

20. Service Principals — Application Accounts

20.1 What is a Service Principal?

A Service Principal is an account intended for applications and scripts that interact with Databricks, unlike human users. It accesses Databricks only via the CLI or REST API, never via the web interface.

graph LR
    SP[Service Principal] -->|REST API| DB[Databricks]
    SP -->|CLI| DB
    
    Human[Human User] -->|Web UI| DB
    Human -->|REST API| DB
    Human -->|CLI| DB
    
    App[CI/CD Application] --> SP
    Script[Automation Script] --> SP
    Pipeline["ADF/ADB Pipeline"] --> SP

20.2 Creating a Service Principal

# File sp.json
cat > sp.json << 'EOF'
{
  "schemas": ["urn:ietf:params:scim:schemas:core:2.0:ServicePrincipal"],
  "displayName": "etl-service-principal",
  "groups": [
    {"value": "DATA_ENGINEERS_GROUP_ID"}
  ],
  "entitlements": [
    {"value": "workspace-access"}
  ]
}
EOF

curl -X POST "${WORKSPACE_URL}/api/2.0/preview/scim/v2/ServicePrincipals" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @sp.json

20.3 Listing and Deleting Service Principals

# List all Service Principals
curl -X GET "${WORKSPACE_URL}/api/2.0/preview/scim/v2/ServicePrincipals" \
  -H "Authorization: Bearer ${TOKEN}"

# Delete a Service Principal
SP_ID="xxxx-yyyy-zzzz"
curl -X DELETE \
  "${WORKSPACE_URL}/api/2.0/preview/scim/v2/ServicePrincipals/${SP_ID}" \
  -H "Authorization: Bearer ${TOKEN}"

20.4 Service Principal Best Practices

21. Azure Active Directory and SCIM Integration

21.1 Why Automate User Provisioning?

Manual user provisioning is tedious and error-prone. With the Azure SCIM Provisioning Connector, it is possible to fully automate synchronization between Azure Active Directory and Databricks.

sequenceDiagram
    participant AAD as Azure Active Directory
    participant SCIM as SCIM Provisioning App
    participant DB as Databricks Workspace

    AAD->>SCIM: Users and AD groups
    SCIM->>SCIM: Scheduled synchronization
    SCIM->>DB: Create/Update/Delete users
    DB-->>SCIM: Confirmation
    SCIM->>DB: Create/Update groups
    DB-->>SCIM: Confirmation
    
    Note over AAD,DB: Continuous automatic synchronization

21.2 SCIM Provisioning Connector Configuration

Configuration steps:

1. In the Azure portal, access Azure Active Directory
2. Go to Enterprise Applications → New Application
3. Search for “Azure Databricks SCIM Provisioning Connector”
4. Create the application
5. In the created application, go to Provisioning
6. Configure with:
   • Tenant URL: https://your-workspace.azuredatabricks.net/api/2.0/preview/scim
   • Secret Token: Personal Access Token from a Databricks admin
7. Test the connection and enable provisioning

21.3 Azure AD ↔ Databricks Mappings

21.4 Automatic Provisioning Limitations

22. Managing and Using Personal Access Tokens

22.1 What is a Personal Access Token?

A Personal Access Token (PAT) is an authentication token that allows programmatic access to Databricks (API, CLI, third-party tools). It is the equivalent of a password for technical access.

22.2 Creating a PAT from the Interface

1. In the Databricks menu, go to Settings → User Settings
2. Access Tokens tab
3. Click Generate New Token
4. Add a descriptive comment (e.g.: “Token for Azure DevOps CI/CD”)
5. Set a lifetime (in days, maximum 730 days)
6. Copy and store immediately the token — it will not be visible again

Critical security: Never store a PAT in plaintext in source code, unencrypted configuration files, or emails. Use Azure Key Vault or CI/CD pipeline secrets.

22.3 Managing PATs via REST API

# List all current user tokens
curl -X GET "${WORKSPACE_URL}/api/2.0/token/list" \
  -H "Authorization: Bearer ${TOKEN}"

# Create a new token
curl -X POST "${WORKSPACE_URL}/api/2.0/token/create" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "comment": "Token for CI/CD pipeline",
    "lifetime_seconds": 7776000
  }'

# Revoke a token
curl -X POST "${WORKSPACE_URL}/api/2.0/token/delete" \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"token_id": "TOKEN_ID_TO_REVOKE"}'

22.4 Enterprise PAT Management Policy

22.5 Using a PAT in Python Code

import os
from databricks.sdk import WorkspaceClient

# Option 1: Via environment variable (recommended)
# Export: DATABRICKS_HOST=https://... and DATABRICKS_TOKEN=dapi...
client = WorkspaceClient()

# Option 2: Via explicit parameters (less recommended)
client = WorkspaceClient(
    host=os.environ.get("DATABRICKS_HOST"),
    token=os.environ.get("DATABRICKS_TOKEN")
)

# List clusters
clusters = client.clusters.list()
for cluster in clusters:
    print(f"Cluster: {cluster.cluster_name} | State: {cluster.state}")

# Using Azure Key Vault (recommended for production)
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

credential = DefaultAzureCredential()
key_vault_client = SecretClient(
    vault_url="https://my-keyvault.vault.azure.net",
    credential=credential
)

databricks_token = key_vault_client.get_secret("databricks-pat").value
databricks_host = key_vault_client.get_secret("databricks-host").value

client = WorkspaceClient(host=databricks_host, token=databricks_token)

23. Databricks REST API — Complete Reference

23.1 API Structure

The Databricks REST API is organized around several main endpoints:

https://{workspace-url}/api/2.0/{endpoint}

Main endpoints:
├── /clusters/
│   ├── create          POST  Create a cluster
│   ├── edit            POST  Modify a cluster
│   ├── start           POST  Start a cluster
│   ├── restart         POST  Restart a cluster
│   ├── delete          POST  Delete a cluster
│   ├── get             GET   Cluster details
│   └── list            GET   List clusters
├── /policies/clusters/
│   ├── create          POST  Create a policy
│   ├── edit            POST  Modify a policy
│   ├── delete          POST  Delete a policy
│   ├── get             GET   Policy details
│   └── list            GET   List policies
├── /instance-pools/
│   ├── create          POST  Create a pool
│   ├── edit            POST  Modify a pool
│   ├── delete          POST  Delete a pool
│   └── list            GET   List pools
└── /preview/scim/v2/
    ├── Users           CRUD  User management
    ├── Groups          CRUD  Group management
    └── ServicePrincipals CRUD Service principal management

23.2 Creating a Cluster Policy via API

# 1. Set environment variables
export DATABRICKS_TOKEN="dapi_your_token_here"
export WORKSPACE_URL="https://adb-xxxx.xx.azuredatabricks.net"

# 2. Create a new policy
curl -X POST "${WORKSPACE_URL}/api/2.0/policies/clusters/create" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "policy-data-engineering-prod",
    "definition": "{\"spark_version\":{\"type\":\"fixed\",\"value\":\"13.3.x-scala2.12\"},\"autotermination_minutes\":{\"type\":\"fixed\",\"value\":60,\"hidden\":true},\"node_type_id\":{\"type\":\"allowlist\",\"values\":[\"Standard_DS3_v2\",\"Standard_DS4_v2\"]}}"
  }'

23.3 Creating a Cluster from a Policy via API

# Cluster JSON configuration file
cat > devcluster.json << 'EOF'
{
  "cluster_name": "my-api-cluster",
  "spark_version": "13.3.x-scala2.12",
  "node_type_id": "Standard_DS3_v2",
  "num_workers": 3,
  "autotermination_minutes": 30,
  "policy_id": "E0004A97B2B5F000",
  "custom_tags": {
    "team": "data-engineering",
    "environment": "development"
  }
}
EOF

# Submit the creation request
curl -X POST "${WORKSPACE_URL}/api/2.0/clusters/create" \
  -H "Authorization: Bearer ${DATABRICKS_TOKEN}" \
  -H "Content-Type: application/json" \
  --data @devcluster.json

23.4 Complete Administration Script via REST API

#!/usr/bin/env python3
"""
Databricks administration script via REST API
Cluster, policy and permission management
"""
import os
import json
import requests
from typing import Optional, Dict, List, Any

class DatabricksAdminClient:
    """Administration client for the Databricks REST API."""
    
    def __init__(self, workspace_url: str, token: str):
        self.workspace_url = workspace_url.rstrip('/')
        self.headers = {
            "Authorization": f"Bearer {token}",
            "Content-Type": "application/json"
        }
    
    def _request(self, method: str, endpoint: str, 
                  data: Optional[Dict] = None) -> Dict:
        """Makes a request to the Databricks API."""
        url = f"{self.workspace_url}/api/2.0/{endpoint}"
        response = requests.request(
            method, url, 
            headers=self.headers,
            json=data
        )
        response.raise_for_status()
        return response.json()
    
    # ── Policy Management ──────────────────────────────────
    
    def create_policy(self, name: str, definition: Dict) -> str:
        """Creates a cluster policy and returns its ID."""
        result = self._request("POST", "policies/clusters/create", {
            "name": name,
            "definition": json.dumps(definition)
        })
        policy_id = result["policy_id"]
        print(f"Policy created: {name} (ID: {policy_id})")
        return policy_id
    
    def list_policies(self) -> List[Dict]:
        """Lists all cluster policies."""
        result = self._request("GET", "policies/clusters/list")
        return result.get("policies", [])
    
    def delete_policy(self, policy_id: str) -> None:
        """Deletes a cluster policy."""
        self._request("POST", "policies/clusters/delete", {
            "policy_id": policy_id
        })
        print(f"Policy {policy_id} deleted")
    
    # ── Cluster Management ──────────────────────────────────
    
    def create_cluster(self, config: Dict) -> str:
        """Creates a cluster and returns its ID."""
        result = self._request("POST", "clusters/create", config)
        cluster_id = result["cluster_id"]
        print(f"Cluster created: {config.get('cluster_name')} (ID: {cluster_id})")
        return cluster_id
    
    def list_clusters(self) -> List[Dict]:
        """Lists all clusters."""
        result = self._request("GET", "clusters/list")
        return result.get("clusters", [])
    
    def terminate_cluster(self, cluster_id: str) -> None:
        """Terminates a cluster."""
        self._request("POST", "clusters/delete", {"cluster_id": cluster_id})
        print(f"Cluster {cluster_id} stopping")
    
    # ── Permission Management ──────────────────────────────
    
    def set_cluster_permissions(self, cluster_id: str, 
                                  acl: List[Dict]) -> None:
        """Sets permissions on a cluster."""
        self._request("PUT", f"permissions/clusters/{cluster_id}", {
            "access_control_list": acl
        })
        print(f"Permissions updated for cluster {cluster_id}")
    
    def set_policy_permissions(self, policy_id: str, 
                                 acl: List[Dict]) -> None:
        """Sets permissions on a cluster policy."""
        self._request("PUT", 
                      f"permissions/cluster-policies/{policy_id}", {
            "access_control_list": acl
        })
        print(f"Permissions updated for policy {policy_id}")

# Using the client
if __name__ == "__main__":
    client = DatabricksAdminClient(
        workspace_url=os.environ["DATABRICKS_HOST"],
        token=os.environ["DATABRICKS_TOKEN"]
    )
    
    # Create a data engineering policy
    policy_definition = {
        "spark_version": {
            "type": "allowlist",
            "values": ["13.3.x-scala2.12", "14.3.x-scala2.12"],
            "defaultValue": "13.3.x-scala2.12"
        },
        "autotermination_minutes": {
            "type": "fixed",
            "value": 60,
            "hidden": True
        },
        "autoscale.max_workers": {
            "type": "range",
            "maxValue": 10
        }
    }
    
    policy_id = client.create_policy(
        "data-engineering-standard", 
        policy_definition
    )
    
    # Assign permissions to a group
    client.set_policy_permissions(policy_id, [
        {"group_name": "data-engineers", "permission_level": "CAN_USE"},
        {"group_name": "admins", "permission_level": "CAN_MANAGE"}
    ])
    
    # Create a production cluster
    cluster_id = client.create_cluster({
        "cluster_name": "prod-etl-cluster",
        "spark_version": "13.3.x-scala2.12",
        "node_type_id": "Standard_DS4_v2",
        "autoscale": {"min_workers": 2, "max_workers": 8},
        "autotermination_minutes": 60,
        "policy_id": policy_id,
        "custom_tags": {
            "team": "data-engineering",
            "environment": "production"
        }
    })
    
    print(f"\nSummary:")
    print(f"  Policy created: {policy_id}")
    print(f"  Cluster created: {cluster_id}")

24. Databricks CLI — Command-Line Management

24.1 Installation and Configuration

# Install via pip
pip install databricks-cli

# Or via the new CLI (recommended)
pip install databricks-sdk

# Configuration
databricks configure --token
# Enter: Databricks Host: https://adb-xxxx.azuredatabricks.net
# Enter: Token: dapi_your_token

# Verify configuration
databricks clusters list

24.2 Essential CLI Commands

# ── Cluster management ──────────────────────────────────────

# List clusters
databricks clusters list

# Create a cluster from a JSON file
databricks clusters create --json-file cluster-config.json

# Start a cluster
databricks clusters start --cluster-id 0123-456789-abcdef

# Stop a cluster
databricks clusters delete --cluster-id 0123-456789-abcdef

# Detailed cluster info
databricks clusters get --cluster-id 0123-456789-abcdef

# ── Policy management ──────────────────────────────────────

# List policies
databricks cluster-policies list

# Create a policy
databricks cluster-policies create --json-file policy.json

# ── DBFS management ─────────────────────────────────────

# Upload an init script
databricks fs cp init_script.sh dbfs:/company/init-scripts/

# List DBFS files
databricks fs ls dbfs:/company/

# ── Job management ─────────────────────────────────────────

# List jobs
databricks jobs list

# Trigger a job
databricks jobs run-now --job-id 12345

# Check run status
databricks runs get --run-id 98765

24.3 New Databricks SDK for Python

# Installation
# pip install databricks-sdk

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.compute import (
    ClusterSpec, 
    AutoScale,
    ClusterLogConf,
    DbfsStorageInfo
)

# Initialize (uses environment variables automatically)
w = WorkspaceClient()

# Create a cluster with the SDK
cluster = w.clusters.create(
    cluster_name="sdk-test-cluster",
    spark_version="13.3.x-scala2.12",
    node_type_id="Standard_DS3_v2",
    autoscale=AutoScale(min_workers=1, max_workers=5),
    autotermination_minutes=30,
    custom_tags={"team": "data-engineering", "env": "dev"}
)

print(f"Cluster ID: {cluster.cluster_id}")

# List active clusters
for c in w.clusters.list():
    print(f"  {c.cluster_name}: {c.state.value}")

# Delete a cluster
w.clusters.permanent_delete(cluster_id=cluster.cluster_id)

25. Terraform / Infrastructure as Code for Databricks

25.1 Why Use Terraform with Databricks?

Terraform allows managing Databricks infrastructure (clusters, policies, permissions, workspaces) as code, offering:

• Reproducibility: the same code produces the same result each time
• Versioning: infrastructure changes are tracked in Git
• Collaboration: teams can review changes via Pull Requests
• Automation: automatic deployment in CI/CD pipelines
• Rollback: ability to revert to a previous configuration

25.2 Databricks Terraform Provider

# versions.tf
terraform {
  required_providers {
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.0"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 3.0"
    }
  }
}

# Authentication via Personal Access Token
provider "databricks" {
  host  = var.databricks_host
  token = var.databricks_token
}

# Or Azure AD authentication (recommended for production)
provider "databricks" {
  host                = azurerm_databricks_workspace.main.workspace_url
  azure_workspace_resource_id = azurerm_databricks_workspace.main.id
}

25.3 Deploying a Databricks Workspace with Terraform

# main.tf — Databricks Workspace on Azure

resource "azurerm_resource_group" "rg" {
  name     = "databricks-rg"
  location = "East US 2"
}

resource "azurerm_databricks_workspace" "main" {
  name                = "my-databricks-workspace"
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  sku                 = "premium"
  
  custom_parameters {
    no_public_ip = true
  }
  
  tags = {
    Environment = "production"
    Team        = "platform"
  }
}

output "workspace_url" {
  value = "https://${azurerm_databricks_workspace.main.workspace_url}"
}

25.4 Managing Cluster Policies with Terraform

# policies.tf

# Policy for data engineers
resource "databricks_cluster_policy" "data_engineering" {
  name = "data-engineering-standard"
  
  definition = jsonencode({
    "spark_version" = {
      "type"         = "allowlist"
      "values"       = ["13.3.x-scala2.12", "14.3.x-scala2.12"]
      "defaultValue" = "13.3.x-scala2.12"
    }
    "node_type_id" = {
      "type"         = "allowlist"
      "values"       = ["Standard_DS3_v2", "Standard_DS4_v2"]
      "defaultValue" = "Standard_DS3_v2"
    }
    "autotermination_minutes" = {
      "type"        = "range"
      "minValue"    = 10
      "maxValue"    = 120
      "defaultValue" = 60
    }
    "autoscale.max_workers" = {
      "type"     = "range"
      "maxValue" = 15
    }
    "custom_tags.team" = {
      "type"   = "fixed"
      "value"  = "data-engineering"
      "hidden" = true
    }
  })
}

# Data engineers group
resource "databricks_group" "data_engineers" {
  display_name = "data-engineers"
}

# Policy permission for the group
resource "databricks_permissions" "policy_permissions" {
  cluster_policy_id = databricks_cluster_policy.data_engineering.id
  
  access_control {
    group_name       = databricks_group.data_engineers.display_name
    permission_level = "CAN_USE"
  }
}

# Users in the group
resource "databricks_user" "alice" {
  user_name = "alice@example.com"
}

resource "databricks_group_member" "alice_in_engineers" {
  group_id  = databricks_group.data_engineers.id
  member_id = databricks_user.alice.id
}

25.5 Managing Instance Pools with Terraform

# instance_pools.tf

resource "databricks_instance_pool" "production_pool" {
  instance_pool_name                    = "production-standard-pool"
  min_idle_instances                    = 2
  max_capacity                          = 30
  node_type_id                          = "Standard_DS4_v2"
  idle_instance_autotermination_minutes = 60
  
  azure_attributes {
    availability       = "SPOT_WITH_FALLBACK_AZURE"
    spot_bid_max_price = -1  # Market price
  }
  
  preloaded_spark_versions = ["13.3.x-scala2.12"]
  
  custom_tags = {
    Team        = "platform"
    Environment = "production"
  }
}

# Permission on the pool
resource "databricks_permissions" "pool_permissions" {
  instance_pool_id = databricks_instance_pool.production_pool.id
  
  access_control {
    group_name       = databricks_group.data_engineers.display_name
    permission_level = "CAN_ATTACH_TO"
  }
}

25.6 CI/CD Workflow with Terraform

# .github/workflows/databricks-infra.yml
name: Databricks Infrastructure

on:
  push:
    branches: [main]
    paths: ['terraform/**']
  pull_request:
    branches: [main]
    paths: ['terraform/**']

jobs:
  terraform:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v2
      with:
        terraform_version: 1.5.0
    
    - name: Terraform Init
      run: terraform init
      working-directory: ./terraform
      env:
        ARM_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
        ARM_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
        ARM_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
        ARM_SUBSCRIPTION_ID: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
    
    - name: Terraform Plan
      run: |
        terraform plan \
          -var="databricks_host=${{ secrets.DATABRICKS_HOST }}" \
          -var="databricks_token=${{ secrets.DATABRICKS_TOKEN }}"
      working-directory: ./terraform
    
    - name: Terraform Apply (main only)
      if: github.ref == 'refs/heads/main'
      run: |
        terraform apply -auto-approve \
          -var="databricks_host=${{ secrets.DATABRICKS_HOST }}" \
          -var="databricks_token=${{ secrets.DATABRICKS_TOKEN }}"
      working-directory: ./terraform

26. Summary and Key Points

26.1 Fundamental Concepts Recap

mindmap
  root((Databricks\nAdministration))
    Cluster Policies
      Constraint types
        fixed
        allowlist
        blocklist
        range
        regex
      Best practices
        Communication
        Testing
        Progressive deployment
    Permissions
      Levels
        Workspace
        Cluster
        Instance Pool
        Policy
      Key entitlements
        workspace-access
        allow-cluster-create
        Can Attach To
        Can Manage
    Users
      Types
        Humans
        Service Principals
        Groups
      Provisioning
        Manual UI
        SCIM API
        Azure AD SCIM
    API and automation
      REST API v2.0
      Databricks CLI
      Python SDK
      Terraform

26.2 Databricks Administration Checklist

Initial workspace configuration:

• Workspace created with Premium SKU (required for policies)
• admins group configured with administrators
• Groups created by team (data-engineers, data-scientists, analysts)
• Cluster policies defined for each role
• Instance Pool created if necessary
• Mandatory tags configured via policies

Cluster governance:

• allow-cluster-create permission not granted directly to users
• Auto-termination enforced in all policies
• Cluster sizes limited by role
• Mandatory cost tags (cost_center, team, environment)
• Standardized init scripts

Security:

• PATs with limited lifetime
• Service Principals created for each application
• Azure Key Vault for secret storage
• Unity Catalog enabled and configured
• Data Security Mode configured in policies

26.3 Management Methods Comparison Table

27. Review Questions

Module 1: Cluster Policies

1. What is the difference between fixed and allowlist in a cluster policy?

   • fixed: a single enforced value, the user cannot change it
   • allowlist: a list of allowed values the user chooses from

2. Which constraint type would you use to block DBR 9.x while allowing all others?

   • blocklist with 9.x versions listed

3. How do you hide a policy attribute from the end user?

   • Add "hidden": true in the attribute definition

4. Can a user without allow-cluster-create create clusters?

   • Yes, if they have the Can Use permission on a cluster policy

5. Why assign policy permissions to groups rather than individual users?

   • Better scalability, easier management during team changes

Module 2: Permissions and Access

6. Which permission allows attaching a notebook to a cluster without being able to modify it?

   • Can Attach To

7. What are the two groups automatically created in every Databricks workspace?

   • admins and users

8. What is the difference between a User and a Service Principal?

   • User: account for humans (UI + API), Service Principal: account for applications (API/CLI only)

Module 3: API and Automation

9. How do you authenticate a request to the Databricks REST API?

   • Via the Authorization: Bearer {TOKEN} header

10. Which endpoint creates a cluster policy via the API?

    • POST /api/2.0/policies/clusters/create

28. Glossary

Search Terms

administering · clusters · configuring · policies · databricks · azure · spark · data · engineering · analytics · cluster · policy · api · via · instance · terraform · access · configuration · managing · permissions · pool · type · autoscaling · entitlements