Databricks Identity & Access Management: AIM, SCIM & PIM — 2026 Edition

Feature	SCIM Provisioning	Automatic Identity Management (AIM)
Sync Targets	Users and Groups (direct members only)	Users, Groups, and Service Principals
Nested Group Support	No — direct memberships only	Yes — transitive memberships supported
Entra ID App Requirement	Requires a separate Enterprise Application	No separate application required
Default Availability	Manual configuration	On by default for accounts created after Aug 1, 2025
Identity Federation	Required	Required

Identity Status	Technical Definition / Trigger
Inactive: No Usage	Exists in the IdP but has never logged into Databricks (Users/SPs), or has not been added to a workspace (Groups).
Active	Has successfully authenticated or been actively provisioned into the Databricks environment.
Active: Removed From IdP	Was active but has been deleted in the IdP; will deactivate in Databricks on the next sync cycle.
Deactivated	Explicitly deactivated in the IdP, or automatically deactivated by Databricks following an IdP deletion.
Denied	On the account access denylist — logins blocked, and the principal is excluded from sharing dialogs.

Lesson Content · Full Text

Course Modules — DIAM‑2026

Five modules in a progressive build order — cloud differences, sync mechanics, troubleshooting, provisioning boundaries, and PIM — each with a practice quiz drawn from the 25 sourced scenario questions.

Module 1Enterprise IDP Integration & Cloud Differences＋

Workforce identity behaves differently on each cloud. Azure Databricks treats Entra ID as a first-party, deeply-integrated source of record. AWS and GCP require more explicit configuration to reach the same experience.

Native integration patterns

Azure DatabricksDeep Entra ID integration with AIM as the primary source of record. Managed Identities are highly recommended for Unity Catalog storage access to eliminate secret management. Billing is unified via Microsoft commercial agreements.

Databricks on AWSAWS Unified Login streamlines authentication across the account and all workspaces at once. Connects to Entra ID or Okta via SSO/SCIM. IAM Role Assumption is the preferred "secretless" pattern for S3 access.

Google Cloud (GCP)Shifting to a Bring-Your-Own-IdP (BYO-IdP) model, replacing default Google Identity configurations. Requires credential configuration during the "Test Connection" phase to link GCP projects with an enterprise identity source.

Exam tip

The "secretless" strategic recommendation applies across every cloud: Azure Managed Identities or AWS IAM Role Assumption — never long-lived service principal keys.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Module 2Automatic Identity Management (AIM): GA & Sync Mechanics＋

AIM's General Availability removes the barrier of pre-provisioning entirely. Under AIM, you can share Unity Catalog objects or AI/BI dashboards with any user in your Entra ID directory — the platform searches the whole IdP directory in real time, and the user is provisioned automatically on first access.

Sync intervals and operational latency

Browser logins: sync if >5 minutes elapsed Token/job authentication: sync if >40 minutes elapsed

Nested groups, service principals & JIT

Inheritance: members of child groups automatically inherit the parent group's permissions.
UI visibility limit: only the first level of nesting is visible on the group detail page in the Account Console.
API/Terraform blind spot: nested groups and service principals not directly provisioned to the account are view-only — they cannot be retrieved or managed via the Databricks Terraform provider or SCIM APIs unless explicitly provisioned.
JIT is mandatory with AIM and cannot be disabled — new users are created automatically on first login. Service principals must complete a "first use" (token auth or job run) before they appear in the directory, even inside a synced group.

databricks accounts groups create --display-name "DB-Metastore-Admins" \
  --external-id "<ENTRA_ID_GROUP_OBJECT_ID>"

Exam tip

Note the 2026 behavior change: the default workspace "users" system group will have no entitlements — workspace entitlements must be granted explicitly when adding principals, rather than inherited by default.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Module 3Troubleshooting Identity Divergences (The AIM Prep Notebook)＋

Identity divergence occurs when the IdP's unique ID (ObjectID) and the Databricks externalId fall out of sync — typically surfacing as duplicate entries in sharing modals. Treat this as a pre-migration governance audit requirement.

AIM Enablement Prep Script — 3 phases

Phase 1 (Workspace Compatibility): identifies non-federated workspaces, since AIM requires identity federation.
Phase 2 (Identity Gathering): aggregates all existing account identities into CSV reports.
Phase 3 (Divergence Check): compares Databricks records against the IdP to flag mismatches.

Diagnostic guide

Error	Technical Meaning	Action to Take
`EXTERNAL_ID_NOT_IN_IDP`	externalId does not exist in the IdP	Update to a valid ObjectID, or remove it
`EXTERNAL_ID_MATCH_NAME_MISMATCH`	Correct ID, but the username differs	Users/SPs: update username (support ticket). Groups: rename group.
`NAME_MATCH_EXTERNAL_ID_MISMATCH`	Name matches the IdP, but the ID is wrong	Update the Databricks externalId to the IdP's ObjectID
`GROUP_HAS_LOCAL_MEMBERS...`	Group contains manually-added local members	Remove local members to restore the IdP as source of truth

# Resolve NAME_MATCH_EXTERNAL_ID_MISMATCH via SCIM PATCH
PATCH /api/2.1/accounts/<accountId>/scim/v2/Users/<databricksId>
{
  "schemas": ["urn:ietf:params:scim:api:messages:2.0:PatchOp"],
  "Operations": [
    { "op": "replace", "path": "externalId", "value": "<ENTRA_ID_OBJECT_ID>" }
  ]
}

Exam tip

Renaming a group in the IdP is not proactive — the change only reflects in Databricks once an administrator opens that group's detail page in the Account Console. Also note: removing an externalId (setting it empty) is currently supported for Groups only.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Module 4Just-In-Time Provisioning & Sharing Boundaries＋

SSO-based JIT provisioning enables frictionless sharing of Databricks Apps, Genie Rooms, and AI/BI Dashboards to unprovisioned IdP users — while SCIM remains active as a fallback deprovisioning channel to handle hard deletes and credential deactivation.

Context-Based Ingress (CBI) — Public Preview

CBI allows administrators to enforce dynamic, zero-trust policies that restrict user access to sensitive platform layers, while safely exposing specific data apps, AI/BI dashboards, and Genie endpoints to unprovisioned external users — without exposing the full workspace UI.

Account-level vs. workspace-level asset sharing

Workspace-level assets require the admin to first add the group to the workspace; account-level assets can be shared with any Entra ID group directly, since AIM searches the whole directory on demand.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Module 5Securing the Admin Layer with PIM＋

For sensitive roles like Account Admin, enforce "Least Privilege" via Microsoft Entra ID Privileged Identity Management (PIM), using Entra ID P2 or Governance licenses to grant time-bound, audited group role activation.

The synchronization delay

PIM activation in the Azure portal is instant for Azure RBAC. But because Databricks relies on its own internal sync cycles, there's a known delay of up to 40 minutes before the elevated group membership is recognized inside Databricks under legacy SCIM — advise users to activate PIM roughly 45 minutes before performing sensitive operations if SCIM is still in use. AIM's 5-minute browser-login cache window shortens this dramatically.

Governance & auditing

endpoint: "autoUserCreation" — flags AIM-driven operations groupMembershipType: "IdentityProvider" — filters IdP-synced memberships

Exam tip

PIM is designed for human users — it does not support Service Principals. Success always depends on the AIM sync cycle actually recognizing the dynamic group membership update.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Applied Scenarios

Three Full Case Studies

Worked scenarios showing the mechanics above applied to real production incidents.

Case Study A — Mitigating JIT Administrative Role Elevation Latency (PIM Integration)

The Problem

A platform administrator activates their "Metastore Admin" group role via Azure PIM. Immediately after, they hit a Permission Denied error trying to configure Unity Catalog storage credentials — because the organization's SCIM connector only syncs every 20–40 minutes, while Entra ID updated instantly.

Under the Hood

SCIM is a push model on a scheduled interval; it has no idea the PIM activation happened until its next sync cycle runs. AIM instead uses direct Microsoft Graph API queries triggered by authentication checks — group memberships refresh dynamically based on activity (5 min for browser logins, 40 min for token/job auth).

Resolution

Decommission the legacy SCIM connector via Security → User Provisioning in the Account Console. Register the PIM-managed group in Databricks with its Entra ID ObjectId as the externalId. Once activated, a fresh browser login lets AIM query Graph API within its 5-minute cache window — privileges resolve almost instantly.

Case Study B — Nested Group Access Cascades and Programmatic API Boundaries

The Problem

An architect builds Marketing-All (parent, explicitly provisioned) with nested children Marketing-US and Marketing-EU. A nightly audit script queries the SCIM API for members of Marketing-US and gets an empty result or 404 — even though those users are actively working in the workspace.

Under the Hood

AIM grants access transitively without pre-provisioning the entire hierarchy, to avoid account-limit bloat in tenants with deep nesting. Because the child groups were never directly provisioned, they're visible in the Account Console UI but cannot be retrieved, searched, or managed via the SCIM API or Terraform.

Resolution

If only access inheritance is needed, nothing changes — permissions cascade transitively. To make a child group independently queryable, explicitly provision it as its own group shell via POST /api/2.1/accounts/<accountId>/scim/v2/Groups with its Entra ID ObjectId set as the externalId.

Case Study C — Troubleshooting Identity Divergences on AIM Transition

The Problem

A 10,000-user organization migrates from a legacy SCIM connector to AIM. Admins notice duplicate identities in sharing modals, new Entra ID groups sharing names with legacy local groups failing to sync, and dashboard-share member counts not matching the actual member list.

Under the Hood

AIM relies on the provisioned identity's externalId matching the IdP's ObjectId. Manually-created or outdated-SCIM-mapped identities may have a missing or mismatched externalId, so Databricks treats the local and IdP identities as two separate objects. Databricks also enforces a strict unique group-name constraint at the account level, which blocks provisioning when a legacy group already reserved that name.

Resolution

Run the AIM Enablement Prep Script's divergence scan (an account-admin service principal executes run_divergence against config.py's ACCOUNT_ID). Triage NAME_MATCH_EXTERNAL_ID_MISMATCH entries via SCIM PATCH to correct the externalId, and triage GROUP_HAS_LOCAL_MEMBERS_WITHOUT_EXTERNAL_ID by removing local members and re-adding legitimate ones directly in Entra ID.

Hands-On Lab

Secretless Unity Catalog Storage with Terraform

A complete, production-grade Terraform blueprint for onboarding ADLS Gen2 storage into Unity Catalog using Azure Managed Identities — eliminating manual secret rotation entirely.

🗄️

ADLS Gen2 Storage

Hierarchical namespace enabled (is_hns_enabled = true) — mandatory for Delta Lake directory-level transactional guarantees, with public access denied by default.

🔗

Access Connector

A System-Assigned Managed Identity — the first-party, secretless cloud identity layer Databricks recommends for metadata connectivity.

🛡️

Role Assignment

Grants the Managed Identity "Storage Blob Data Contributor" scoped directly to the target storage account — no keys, no client secrets.

📜

UC Storage Credential & External Location

Registers the Access Connector as a Storage Credential, then scopes it to a safe abfss:// path as an External Location.

# Access Connector — hosts the System-Assigned Managed Identity
resource "azurerm_databricks_access_connector" "uc_access_connector" {
  name                = var.access_connector_name
  resource_group_name = azurerm_resource_group.rg.name
  location            = azurerm_resource_group.rg.location
  identity { type = "SystemAssigned" }
}

# Register it in Unity Catalog as a Storage Credential — no secrets, no keys
resource "databricks_storage_credential" "uc_credential" {
  name = var.storage_credential_name
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.uc_access_connector.id
  }
  comment    = "Secretless credential managed by Terraform using Databricks Access Connector System Identity"
  depends_on = [azurerm_role_assignment.storage_data_contributor]
}

# External Location scoped to a governed abfss:// path
resource "databricks_external_location" "uc_external_location" {
  name            = var.external_location_name
  url             = "abfss://${azurerm_storage_container.container.name}@${azurerm_storage_account.adls_gen2.name}.dfs.core.windows.net/"
  credential_name = databricks_storage_credential.uc_credential.id
  comment         = "Governed external path for production Silver and Gold medallion tables"
}

# RBAC: Admins get full control, Data Engineers get read/write files only
resource "databricks_grants" "location_grants" {
  external_location = databricks_external_location.uc_external_location.id
  grant {
    principal  = var.admin_group_name
    privileges = ["ALL_PRIVILEGES"]
  }
  grant {
    principal  = var.data_engineers_group_name
    privileges = ["READ_FILES", "WRITE_FILES"]
  }
}

Full blueprint (~230 lines) additionally covers: storage-account network firewall rules with IP/subnet allow-lists, the resource group and container provisioning, and Terraform outputs for the connector's principal ID and the registered External Location URL. Deploy with the standard lifecycle: terraform init → terraform plan -out=tfplan → terraform apply tfplan.

Spaced Practice · 20 Cards

Flashcards — Core IAM Facts

A rapid-recall deck spanning sync intervals, error diagnostics, provisioning rules, and multi-cloud identity patterns. Read the question, think of your answer, then flip the card to check yourself. Mark each card "Got It" or "Still Learning" to build a focused review queue.

Card 1 / 20

Question

Loading…

Click to flip ↻

Answer

Click to flip ↻

Academic & Technical Bibliography

References & Further Reading

Compiled from a Databricks IAM study handbook, an architecture infographic, a production Terraform blueprint, and current Databricks product announcements.

The Databricks Identity and Access Management HandbookFive-module deep dive: IDP integration, AIM sync mechanics, troubleshooting divergences, JIT & sharing, and PIM — the core source for this course.

"The Modern Databricks IAM Architecture" — InfographicFull visual reference embedded above: AIM vs. SCIM provisioning, the SCIM deprovisioning lifecycle, the PIM & JIT lifecycle, multi-cloud identity integration, and troubleshooting identity divergence.

unity-catalog-storage.tf — Production Terraform BlueprintFull onboarding of ADLS Gen2 storage into Unity Catalog using Azure Access Connector Managed Identities and strict RBAC grants.

Databricks Data + AI Summit 2026 — Platform AnnouncementsGeneral Availability of AIM for Entra ID on AWS and GCP; AIM for Okta entering Public Preview; Context-Based Ingress in Public Preview.

AIM Enablement Prep Script DocumentationThe 3-phase (Workspace Compatibility → Identity Gathering → Divergence Check) audit workflow referenced throughout Module 3.

SME Scenario Question Bank (25 questions, sourced)Advanced exam-style scenario questions with hints, used to build the module quizzes and the practice quiz below.

All 25 Sourced Scenario Questions

Sample Scenario Questions

All 25 fully-sourced scenario questions from the course material, grouped exactly as they appear in the modules above. Click any option to reveal the answer and hint. (Two questions from the original 27-question bank — #24 and #27 — did not extract cleanly from the source document and are omitted rather than invented.)

Lesson Assessment · Qualify for Certificate

Practice Quiz — Databricks Identity & Access Management

Ten questions drawn from the module content above, spanning all five modules. Answer all correctly to unlock your certificate — retry as many times as you like.

You have studied:

AIM vs. SCIM architecture, sync intervals, and nested-group inheritance limits
Diagnosing and resolving identity divergence via the SCIM PATCH API
PIM latency, multi-cloud identity patterns, and secretless Terraform storage credentials

Quick Reference

Frequently Asked Questions

Is "Databricks Subject Matter Expert" an official Databricks certification?

No. Databricks Inc.'s official certifications are the Data Engineer Associate/Professional, Data Analyst Associate, ML Associate/Professional, Generative AI Engineer Associate, Context Engineering Associate, and Associate Developer for Apache Spark. "Subject Matter Expert" identity badges (e.g. via EdChart on Credly) are third-party credentials, not issued by Databricks. This course gives genuine technical depth on Databricks IAM — content that also directly supports the Identity and Access domain of the official Azure Databricks Certified Associate Platform Administrator exam — without misrepresenting its certification status.

Do I need prior Databricks experience?

No formal prerequisite, but familiarity with Unity Catalog fundamentals and basic Databricks administration will help the material land faster, especially in Modules 2 and 3.

Why are only 25 of the 27 source scenario questions included?

Questions 24 and 27 did not extract cleanly from the source document — their question text was missing even though the answer key listed answers for them. Rather than inventing plausible-sounding content to fill the gap, this course presents only the 25 questions with fully verified, sourced text.

What's the most-repeated "gotcha" across the question bank?

The distinction between what AIM grants transitively (access) versus what it actually provisions (a queryable local object). Nested child groups get real access through inheritance, but stay invisible to the SCIM API and Terraform unless explicitly provisioned as their own group shell — this exact pattern appears across several scenario questions.

Is this course affiliated with Databricks?

No — this is an independent, free study course compiled from a study handbook, an architecture infographic, and a Terraform blueprint, cross-checked against Databricks' public product announcements, for educational purposes only.

Databricks Identity& Access Management

Master the shift from SCIM to Automatic Identity Management

⚙ SCIM → AIM: THE ARCHITECTURAL SHIFT

After completing this course

Navigate Multi-Cloud IDP Integration

Master AIM Sync Mechanics

Diagnose Identity Divergence

Configure JIT Provisioning

Harden the Admin Layer with PIM

Deploy Secretless Storage Credentials

SCIM vs. AIM, and the Identity Status Lifecycle

The Modern Databricks IAM Architecture — At a Glance

Course Modules — DIAM‑2026

Native integration patterns

Exam tip

Practice Questions — Module Quiz

Sync intervals and operational latency

Nested groups, service principals & JIT

Exam tip

Practice Questions — Module Quiz

AIM Enablement Prep Script — 3 phases

Diagnostic guide

Exam tip

Practice Questions — Module Quiz

Context-Based Ingress (CBI) — Public Preview

Account-level vs. workspace-level asset sharing

Practice Questions — Module Quiz

The synchronization delay

Governance & auditing

Exam tip

Practice Questions — Module Quiz

Three Full Case Studies

Case Study A — Mitigating JIT Administrative Role Elevation Latency (PIM Integration)

The Problem

Under the Hood

Resolution

Case Study B — Nested Group Access Cascades and Programmatic API Boundaries

The Problem

Under the Hood

Resolution

Case Study C — Troubleshooting Identity Divergences on AIM Transition

The Problem

Under the Hood

Resolution

Secretless Unity Catalog Storage with Terraform

ADLS Gen2 Storage

Access Connector

Role Assignment

UC Storage Credential & External Location

Flashcards — Core IAM Facts

References & Further Reading

Sample Scenario Questions

Practice Quiz — Databricks Identity & Access Management

Print Your Certificate

Certificate of Course Completion

Frequently Asked Questions

Databricks Identity
& Access Management