Databricks Certified
Data Engineer Associate

The Data Lakehouse pairs the low-cost, flexible storage of a data lake with the ACID transactions and governance of a data warehouse. Delta Lake supplies the transactional storage layer; Unity Catalog supplies unified governance across data and AI assets.

Compute services — pick the right tool

Control Plane vs. Data Plane

• The Control Plane hosts the workspace UI, job scheduler, and notebook management.
• The Data Plane is where compute clusters actually process your data.

Exam tip

"Automated production ETL that should terminate immediately after completion" → always points to a Job Cluster, not All-Purpose.

Three ingestion tools cover almost every scenario on the exam. All of them land data into Unity Catalog–governed Delta tables by default.

Auto Loader file-detection modes

• Directory Listing (default, polling-based) — fine for smaller file sets, slows down at scale.
• File Notification (event-based, via cloud SQS/SNS or Event Grid) — the recommended mode once a landing zone reaches millions of files.

Handling malformed records

Auto Loader automatically routes rows that don't match the target schema into a _rescued_data column instead of dropping them — this prevents silent data loss during schema drift.

Selection matrix

Method	Volume / Frequency	Best For
Auto Loader	High / continuous	JSON, nested, unstructured data
COPY INTO	Moderate / scheduled batch	Structured, static file sets
Lakeflow Connect	Enterprise scale	SaaS & database sources

Exam tip

"Billions of files, exactly-once semantics" is Databricks' own description of Auto Loader — memorize that phrase.

This is the single largest domain — it covers the Medallion Architecture, PySpark/SQL cleaning, Lakeflow Spark Declarative Pipelines (SDP) dataset types, data quality Expectations, and Change Data Capture.

The Medallion Architecture

Three SDP dataset types

Data quality with Expectations

Expectations are true/false SQL constraints applied to every row. Three violation policies:

Quarantine pattern: run two flows from the same source — a main flow with DROP, and a mirrored flow with the inverse condition writing to a dedicated quarantine table, so bad data is investigated, not lost.

CDC with APPLY CHANGES INTO

Replaces manual MERGE INTO logic. Handles out-of-order events and deduplication automatically using a Sequencing Key (typically a timestamp or version column).

The Enzyme engine

Powers incremental recomputation in SDP: it builds a dependency graph and recomputes only the impacted segments of data rather than rescanning everything — append-only sources are cheapest, windowed aggregations rewrite only affected partitions.

Lazy evaluation — the classic "gotcha"

Transformations (.filter(), .select(), .groupBy(), .join()) are lazy — Spark only records them in a DAG. Nothing executes until an Action (.count(), .show(), .collect(), .save()) is called.

Exam tip

If a code block ends on a transformation with no Action, the correct answer is always "nothing happens — Spark uses lazy evaluation." Always check the last line first.

Lakeflow Jobs (formerly Databricks Workflows) orchestrate notebooks, SQL queries, dashboards, and SDP pipelines using a DAG-based task graph with dependencies, retries, and conditional branching.

Trigger types

Repair and Rerun

For complex DAGs, this feature re-executes only the specific task-level segments that failed, instead of restarting the entire job from scratch — the core answer to any "minimize downtime after partial failure" scenario.

Built-in reliability

Production workflows must be configured with automated failure alerts and automatic restarts to guarantee high availability of data pipelines.

Candidates must shift away from clicking through the UI toward code-first, version-controlled deployment using Declarative Automation Bundles (DABs) — the Infrastructure-as-Code standard for Databricks (formerly Databricks Asset Bundles).

The deployment pipeline

Key files & commands

Exam tip

"Move code from a dev catalog to a prod catalog without changing source code" → always environment-specific variables and overrides in the bundle configuration, never hardcoded catalog names.

The "Big Three" bottlenecks (Spark UI)

Modern optimization standards

• Liquid Clustering replaces legacy manual Z-Ordering / static partitioning — a self-tuning, Hilbert-curve-based layout that adapts to changing query patterns without expensive rewrites.
• Predictive Optimization is an autonomous background service that runs OPTIMIZE and VACUUM on Unity Catalog managed tables based on cost-benefit analysis, and captures statistics "on write."
• Deletion Vectors mark rows as deleted without an immediate rewrite, improving write performance.
• Fix skew in-flight with salting — append a random bucket suffix to skewed keys before grouping.
• Avoid a small-shuffle join cost with a broadcast join hint when one table is small.

Diagnostic & observability tools

Exam tip

"One task takes 45 minutes while others finish in 10 seconds" is the textbook description of data skew, not shuffle overhead or spilling.

The three-level namespace

catalog.schema.table

Managed vs. external tables

Feature	Managed Table	External Table
Data file ownership	Unity Catalog	User / external system
Storage location	Managed root storage	User-specified (S3/ADLS/GCS)
DROP TABLE result	Deletes metadata + data files	Deletes metadata only

Core privileges (GRANT / REVOKE / DENY)

Exam pro-tip: a principal needs USAGE on both the parent catalog and the parent schema to access an object — a common cause of "why can't this user see the table" scenarios.

Fine-grained security

Observability

Data Lineage traces column-level flow from ingestion to consumption for impact analysis; Audit Logging (system.access.audit) captures every governed action; Delta Sharing extends live data to non-Databricks consumers like Power BI or Snowflake.