Databricks Certified
Data Engineer Professional

At the Professional level, ingestion moves beyond a single Auto Loader stream into architecting for many concurrent, mixed-schema sources without multiplying operational overhead.

Core ingestion method comparison

Strategy selection: choose based on latency and volume — batch for nightly overwrites, incremental with Structured Streaming for low-latency, real-time needs.

The Multiplex Bronze pattern

A single "multiplex" Bronze table ingests multiple data streams with different schemas into one unified table, using JSON or binary columns for raw payload storage. This reduces cluster overhead versus managing hundreds of individual stream-to-table pipelines, and simplifies governance since one ingestion point lands data into Unity Catalog before routing to specific Silver tables.

Schema & late data management

• Schema inference and evolution: Auto Loader automatically detects structural drift and updates tables as source fields change.
• Late-arriving data: managed through Structured Streaming watermarks, which tell the system how long to wait for delayed events before finalizing aggregations.

Production reliability practices

Exam tip

Every stream needs its own dedicated checkpoint location — reusing one across streams causes state-metadata conflicts and breaks recovery/idempotency. This exact scenario appears repeatedly across the question bank.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

The Medallion flow at Professional level adds explicit quality-enforcement policies and a formal quarantine pattern rather than just "clean the data."

Data quality expectations & the quarantine pattern

The Quarantine Pipeline Pattern: a more sophisticated design than DROP — invalid records are routed to a separate quarantine table for auditing and manual remediation instead of being discarded, achieving zero-data-loss compliance.

Deduplication patterns

Constraints & integrity

Delta Lake supports boolean SQL CHECK constraints to prevent invalid data at the source. Foreign keys can be declared to inform the optimizer, but Databricks does not natively enforce them — actual referential integrity must be managed via lookup tables and modeling discipline.

Exam tip

If a scenario asks for "zero data loss" alongside quality enforcement, the answer is the quarantine pattern — not DROP, which silently discards records.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Modeling at this level covers advanced SCD implementation, referential integrity workarounds, and the normalized-vs-denormalized trade-off inside a Lakehouse.

SCD Type 1 vs. Type 2 with APPLY CHANGES INTO

Relationship modeling in the Lakehouse

Normalized vs. denormalized: normalized models reduce redundancy, but denormalized models are often preferred in Lakehouse environments to minimize expensive join operations during large-scale analysis. Use lookup tables for efficient code-to-description mapping without fully denormalizing every transaction table.

Handling missing foreign keys

Since Delta Lake does not natively enforce foreign key constraints, engineers must implement alternative logic (lookup validation, quality expectations) to maintain relational integrity — this is a frequent "gotcha" on the exam.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

The single largest domain — production-grade code across the full Bronze → Silver → Gold flow, using declarative Spark Declarative Pipelines patterns.

Bronze: unified multi-source ingestion

Ingest Cloud Storage, Database, IoT Sensor, and Log File sources into one Multiplex Bronze Table — avoiding the overhead of managing individual pipelines per source, and preventing the "small file" problem by matching trigger intervals to data volume.

Silver: automated CDC & data quality

Every ingested record runs through a quality gate: Meets Quality Expectation? → Yes writes to table; No applies the configured policy (Warn / Drop / Fail). SCD logic (Type 1 overwrite vs. Type 2 historical tracking) is handled automatically via APPLY CHANGES INTO.

Gold: physical layout & optimization

Liquid Clustering replaces traditional partitioning and Z-Ordering with a flexible, hands-off approach that co-locates data based on clustering keys (CLUSTER BY). Databricks uses file-level statistics in the Delta transaction log to perform data skipping, significantly reducing I/O for point-lookup queries.

Declarative engineering with SDP

Exam tip

If a question emphasizes "minimizing operational overhead of managing hundreds of individual streams," the answer is virtually always the Multiplex Bronze pattern.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

The domain shifts from "how to build" to "how to operate at scale" — programmatic job management and declarative deployment.

REST API 2.0 idempotency — a critical gotcha

Running 2.0/jobs/create multiple times with an identical JSON payload does not update anything — it creates multiple distinct jobs, each with a unique Job ID. This lacks the state-awareness of declarative tools.

Databricks Asset Bundles (DABs)

Unlike the raw REST API, databricks bundle deploy is idempotent — it updates existing state rather than duplicating resources. Variables like ${bundle.target} parameterize database names and cluster sizes across Dev/Test/Prod.

Troubleshooting & multi-task orchestration

Job status is determined by the collective outcome of the DAG. If a root task and one parallel branch succeed but another parallel branch fails, the job run is reported as Partially Completed — not Failed, not Succeeded.

Repair and Rerun restarts only the failed task and its downstream dependents, preserving work already completed — never re-running the entire workflow from scratch.

Exam tip

"Root task + Task B succeed, Task C fails" → the terminal job status is Partially Completed. Memorize this exact phrase; it appears across multiple question banks.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Production stability requires synthesizing metrics across Spark UI, Ganglia UI, and Unity Catalog lineage to maintain strict SLAs.

The observation suite

Alerting strategies

Production streaming jobs should be configured with unlimited retries capped at one concurrent run to prevent state-file corruption from overlapping executions. If an engineer receives multiple notifications for one threshold breach, the likely cause is multiple triggers configured for the same alert — not a data issue.

Logging & governance forensics

Integrate MLflow to track custom performance metrics, and leverage Unity Catalog's data lineage to trace the source of errors downstream and identify the blast radius of a data quality issue.

Exam tip

Setting Maximum Concurrent Runs to 1 on a streaming job cluster exists specifically to prevent conflicting access to a shared checkpoint directory — not for cost control.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

The second-largest domain — balancing compute cost against latency SLAs through modern, hands-off optimization features.

Compute optimization

Right-sizing: match worker type to the actual bottleneck — high-performance processors for CPU-intensive shuffles, large RAM modules for memory-intensive complex transforms.

Modern layout optimization

Liquid Clustering replaces legacy Z-Ordering's fixed column order with flexible, non-linear clustering — clustering keys can be redefined as query patterns evolve, without rewriting the underlying data.

Automated maintenance: Predictive Optimization

A hands-off feature that observes data growth and query performance, then automatically determines when to run OPTIMIZE and VACUUM — eliminating manually scheduled maintenance jobs entirely.

Data skipping & caching

Delta's transaction log stores min/max column statistics per file, enabling file skipping so irrelevant files are never read. Delta caching stores frequently accessed remote data on local NVMe SSDs to accelerate repeat reads.

Exam tip

The default OPTIMIZE target file size is 1GB — this exact figure is tested repeatedly across multiple question banks in this domain.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Moves beyond basic permissions into dynamic, attribute-driven governance and secure secrets handling for production credentials.

Dynamic views for masking & row-level security

A query flows into the dynamic view (security proxy) → identity check via is_member() or current_user() → authorized users see original data, unauthorized users see redacted/filtered rows.

Secrets management

Never hardcode credentials. Retrieve them with dbutils.secrets.get(scope="scope_name", key="secret_key"). Scope-level permissions are either Read (use the secret in a job) or Manage (administer the scope itself) — a service principal running a job only needs Read.

Permission hierarchy

Exam tip

Granted SELECT on a table but still can't query it? You're missing USAGE on the parent catalog and schema — access requires "entry" permission on every container above the object.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

Unity Catalog as the single source of truth for permission lifecycle, lineage, and cross-cloud consistency.

The unified governance hierarchy

Data is organized into a rigid Catalog → Schema → Object namespace for clear ownership and discovery. Cross-cloud governance means the same security policies apply regardless of the underlying cloud storage provider (AWS, Azure, GCP).

Permission inheritance

Privileges granted at a higher level (Catalog) are automatically inherited by lower-level objects (Schema, then Object) — reducing the need for repetitive per-table grants.

Automated lineage & auditing

Unity Catalog tracks Bronze → Silver → Gold lineage automatically, enabling proactive impact analysis — visualizing downstream effects of a change or deletion before it happens. Every access event is recorded for comprehensive audit trails.

Compute & workspace security

Exam tip

A user needing to attach a notebook to an existing cluster and run cells requires "Can Restart" permission — not "Can Manage" (too broad) or "Can Read" (insufficient to execute).

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.

The smallest domain by weight, but tests genuinely advanced interoperability concepts — sharing data without copying it, and querying external systems in place.

Delta Sharing — open data exchange

An open protocol allowing organizations to share live datasets with external users and tools without copying or moving the underlying files. Because it's an open standard, recipients don't need to be on the Databricks platform at all — enabling secure cross-organization collaboration (Power BI, Excel, external tools).

Lakehouse Federation — unified querying

Query external data sources (PostgreSQL, Snowflake, MySQL) directly through Databricks SQL without first ingesting the data into Delta Lake. External databases are mapped into Unity Catalog, so consistent governance applies and external tables can be joined with internal Delta tables in a single SQL query — reducing latency and avoiding costly ETL for frequently-updated source systems.

Iceberg UniForm — cross-platform interoperability

Allows Delta Lake tables to be read by Apache Iceberg–compatible readers without rewriting the data or creating duplicate copies. UniForm generates Iceberg metadata alongside Delta metadata, so the same physical data serves both ecosystems — a single Delta-format "source of truth" for hybrid, multi-engine stacks.

Exam tip

If a scenario says "external tool needs live access without copying data and isn't on Databricks," the answer is Delta Sharing. If it says "query an external database in place, joined with Delta tables," the answer is Lakehouse Federation.

Practice Questions — Module Quiz

Click an option to reveal whether it's correct.