Databricks Certified Data Engineer Professional Study Guide¶

Independent AI-assisted resource — SOURCES + OBJECTIVES CHECKED; HUMAN REVIEW PENDING. Objective coverage, citations, volatility labels, links, and exam-integrity compliance were checked on October 28, 2026. This is not a guarantee that the guide is error-free or current after that date. See the sources-and-objectives record. The official certification page and its linked exam guide are authoritative.

Study guide at a glance

Start here

Report an issue with DATABRICKS-DATA-ENGINEER-PROFESSIONAL

Blueprint checked2026-10-28

Review stateSources + objectives checked — human review pending

Guide-only active reading45 min–1 hr

Prerequisites: No formal prerequisite; the official guide highly recommends related training and one year of hands-on production data-engineering experience across Python/SQL/Spark, batch and streaming pipelines, CDC, orchestration, testing, sharing/federation, observability, cost/performance, privacy/governance, CI/CD and dimensional modeling.

Exam essentialsRead the objective map, key distinctions, and readiness checklist.

Deep understandingWork through every domain, including decisions, failure modes, and related items.

Hands-on labsBegin with Hands-on lab sequence, then use the remaining labs to produce evidence you can inspect and explain.

Official objective domains

See the objective map in this guide

The active-reading range uses 90–130 words per minute for technical material. Labs, troubleshooting, note-taking, spaced review, and prerequisite gaps add time; it is not a total preparation promise.

Library identifier: DATABRICKS-DATA-ENGINEER-PROFESSIONAL; Databricks does not publish a short exam code on the official page checked.
Current baseline: Detailed official exam guide for the live version as of July 3, 2026; live weighted page checked September 28, 2026.
Upcoming blueprint change: None announced as of September 28, 2026. Recheck the official guide two weeks before the appointment, as Databricks requests.
Lifecycle status: Active; valid for two years, with the currently live exam required for recertification.
Assessment: 59 scored multiple-choice questions, 120 minutes, USD 200, no test aids or API documentation, online or test-center delivery; English, Japanese, Brazilian Portuguese, and Korean listed.
Prerequisite: None required. The official guide highly recommends related training and one year of hands-on experience performing its data-engineering tasks.
Scope posture: This is a production-engineering guide. Be able to explain correctness, state, security, observability, cost, deployment and recovery—not only write a successful notebook cell.

Living-guide watch — October 28, 2026¶

Current platform context: The current official Professional exam guide is dated November 30, 2025. October releases make atomic transactions available for Unity Catalog managed Delta and Iceberg tables (catalog commits must be enabled) and make Lakeflow pipeline system tables generally available. These are useful current engineering practices for transaction boundaries and pipeline observability, but the live exam guide remains the scope authority. See the current certification page and October release notes.

How to use this guide¶

Build one evolving system rather than ten isolated demos. For every pipeline, retain source contract, schema, checkpoint/state, target grain, quality results, code/package revision, bundle target, run/update IDs, identity, grants, cost/performance evidence, failure artifacts and cleanup.

contract -> governed acquisition -> replayable bronze -> tested transformations
-> quality/quarantine -> CDC and consumer model -> shared or federated access
-> orchestrated deployment -> observability and alert -> repair/recovery evidence
-> retention/privacy proof -> measured cost and performance improvement

Weights are study-budget signals, not form guarantees. Python/SQL development, transformation and debugging/deployment total 42%, but professional answers must connect those to security, governance and production evidence.

About related items: A Related item: callout adds prerequisite, architectural, migration, security, operational, or adjacent context that makes an objective easier to understand. It is useful supporting knowledge, not a claim that the item appears verbatim in Databricks' published exam objectives.

Objective map¶

Published domain	Weight	Professional decision or artifact
Developing Code for Data Processing using Python and SQL	22%	Modular project, dependency boundary, batch/stream pipeline, CDC/control flow and automated tests.
Data Ingestion & Acquisition	7%	Format/source contract, append behavior, schema/state and replay.
Data Transformation, Cleansing, and Quality	10%	Efficient transformation with explicit quality/quarantine evidence.
Data Sharing and Federation	5%	Governed D2D/open sharing or federated access with ownership and failure boundaries.
Monitoring and Alerting	10%	System/query/Spark/event/API evidence plus actionable notification or data-quality alert.
Cost & Performance Optimisation	13%	Measured layout, plan, compute, maintenance and incremental-processing decision.
Ensuring Data Security and Compliance	10%	Least privilege, sensitive-data transformation and provable retention/purge behavior.
Data Governance	7%	Discoverable metadata and correct Unity Catalog inheritance/ownership.
Debugging and Deploying	10%	Evidence-led repair and environment-aware bundle/Git CI/CD.
Data Modelling	6%	Scalable Delta and dimensional design with deliberate clustering/partition choices.

1. Developing code for data processing using Python and SQL (22%)¶

Design a deployable Python project¶

A professional project separates pure business transformations from I/O, orchestration and environment configuration. A useful shape is:

src/project/
  contracts.py       # schemas and configuration models
  transforms.py      # DataFrame -> DataFrame functions
  quality.py         # expectations and quarantine rules
  pipelines.py       # read/write composition
tests/unit/
tests/integration/
resources/           # job and pipeline definitions
databricks.yml       # bundle identity, includes, variables and targets
pyproject.toml       # package/dependency/test configuration

Declarative Automation Bundles package resource configuration, files and environment targets for validate/deploy/run workflows. Keep environment-specific workspace paths, catalogs, schemas, service principals and permissions in target configuration—not hard-coded inside transformations. The official guide uses “Declarative Automation Bundles” and notes the former Databricks Asset Bundles/DABs name.

Control third-party dependencies¶

Choose PyPI packages, wheels or source archives from repeatability and support requirements. Pin direct dependencies deliberately, build an immutable wheel, scan provenance/license/vulnerabilities, test against the target runtime, and install at a supported scope. The library documentation distinguishes compute, job-task and notebook approaches. Diagnose installation failures through repository reachability, authentication, Python/runtime/ABI compatibility, transitive conflicts, init order and driver/executor availability.

Do not use a notebook %pip install latest-package as the production definition. Capture resolution in pyproject.toml/lock or build metadata, publish to a controlled artifact repository, and prove a clean-environment install.

Choose native expressions before UDFs¶

Prefer built-in Spark SQL/DataFrame expressions because the optimizer can understand them. A scalar Python UDF crosses serialization/runtime boundaries; a Pandas UDF uses Arrow/vectorized batches and can be a better fit for supported vectorized logic, but still adds type, memory and dependency constraints. Review Pandas UDF behavior and declare input/output schemas.

Related item: A UDF decision is not only about syntax. It affects optimizer visibility, serialization, executor memory, library deployment, error handling and whether the logic can be expressed portably in SQL.

Build batch and streaming pipelines with explicit state¶

Lakeflow Spark Declarative Pipelines manages declarative batch/stream data flows, dependencies, updates, event logs and data-quality expectations. Structured Streaming offers lower-level control over source, sink, trigger, checkpoint, watermark and foreachBatch behavior. Choose declarative pipelines for managed data-flow semantics and expectations; choose Structured Streaming when custom state/control/integration requirements justify the responsibility.

Auto Loader (cloudFiles) incrementally discovers cloud files and stores discovery/schema progress. Checkpoint and schema state are production assets. Deleting or reusing them can cause replay, duplicates or data loss depending on sink idempotency.

Make custom streaming writes recoverable¶

The foreachBatch contract permits a micro-batch to run again. External effects need a durable idempotency key; returning successfully after swallowing a failed write can lose data. Handle empty input, and consume the complete batch when upstream stateful operators require it.

For append writes to Delta, pair a stable txnAppId with txnVersion=batchId. Each destination commits separately: a failure between two writes can leave only the first visible. Retrying with the same identity lets each destination recognize its own completed write. A new checkpoint starts batch numbering again, so use a new application identity and reconcile any replayed business records. This does not create a transaction spanning tables. See Delta streaming writes.

Implement CDC and append flows deliberately¶

AUTO CDC replaces the older APPLY CHANGES name in current Lakeflow documentation. It sequences change events into a target and supports SCD patterns. Define keys, sequence column, deletion/truncation semantics, ignored columns and out-of-order behavior; prove duplicates and late events. Review AUTO CDC APIs.

CDC sequencing must establish a unique logical order per business key. Null sequence values and two different updates with the same key/sequence violate that contract. A source sequence or composite order can disambiguate timestamp ties. Separately, AUTO CDC FROM SNAPSHOT compares complete snapshots in version order; it is not an arbitrary late-event reader.

For historical backfill, define a cutover for each key: its seed must precede its first live change. Verify overlap and gaps explicitly; a once flow can run again during a full refresh. Keep sufficient historical input to support the chosen recovery policy.

A streaming table is incrementally updated from a stream; a materialized view maintains query results. Streaming tables strongly fit append/change streams, while materialized views fit declarative derived results with managed refresh. Confirm source compatibility, latency, update mode and full-refresh consequences.

Orchestrate control flow without assuming transactionality¶

Lakeflow Jobs supports task dependencies, run-if outcomes, if/else, for each, retries and parameters. Successful upstream tasks are not automatically rolled back if a later task fails. Make task effects idempotent or compensatable, persist commit/evidence boundaries and use job control flow intentionally.

Choose compute/configuration per task: high-memory needs do not justify oversizing every task; retries are unsafe for non-idempotent side effects; an optimization or automatic setting needs a measured purpose. Jobs can be created through UI, CLI or REST APIs, but the deployed definition should have one controlled source of truth.

VERIFY CURRENT — serverless retry controls: serverless auto-optimization can add retries beyond the task retry policy. The serverless Jobs guide explains how to disable it for non-idempotent work and how notebook environment dependencies and preview high-memory settings differ from classic compute. Do not confuse this setting with Delta file optimization.

The following is a task-property fragment, not a complete bundle. Confirm the current bundle task schema before deployment:

max_retries: 0
retry_on_timeout: false
disable_auto_optimization: true

These settings limit automatic retries; they cannot prevent an operator from repairing the job or an external caller from starting another run. Preserve a business operation key even when retries are disabled.

Test transformations and systems¶

Unit-test pure DataFrame transforms with small, adversarial fixtures. assertDataFrameEqual compares data; assertSchemaEqual protects structure.
Test nulls, duplicates, empty input, bad types, late/out-of-order events, timezone boundaries and skewed keys.
Integration-test source, checkpoint, target, identity, dependency and bundle wiring in an isolated catalog/schema.
Verify exactly-once claims at the application boundary by replaying the same input and comparing target state.
Use a debugger locally or in supported Databricks tooling for code paths; use event/run/Spark evidence for distributed behavior.

The PySpark testing guidance documents the built-in equality helpers.

2. Data ingestion and acquisition (7%)¶

Model source behavior before selecting a reader¶

Source/format	Important contract
Delta	table version, schema, change data, concurrent writers and retention
Parquet/ORC/Avro	schema, partitions/blocks, compression and evolution
JSON/XML	record boundary, nested fields, malformed data and schema drift
CSV/text	delimiter, quoting, header, encoding, line endings and types
Binary	metadata and content ownership; parsing occurs downstream
Message bus	offsets, event time, ordering, retention and replay
Cloud files	discovery mode, arrival order, checkpoint, schema state and archive/cleanup

The Spark data-source documentation is the engine reference; Databricks source/connector support and runtime behavior remain the operational authority.

An append-only pipeline promises it will not mutate earlier source events, not that every upstream system is append-only. Land immutable raw records with source identity, event/ingest time and provenance; deduplicate or apply CDC in a later controlled layer. For a single design that accepts historical batch plus continuing stream, ensure both paths use compatible schema, identifiers and target semantics.

Related item: Event-time watermarking bounds state for late data; it is not an ingestion completeness guarantee. Record the accepted lateness and what happens to events beyond it.

3. Transformation, cleansing and quality (10%)¶

Write efficient transformations with known semantics¶

Use projections and selective filters early, avoid unnecessary Python boundaries, and make join cardinality explicit. Broadcast only a genuinely bounded side. A window has partition and ordering semantics; an aggregate collapses grain. Validate before optimizing:

The example below receives already validated line items. Route bad quantities, prices and missing keys to quarantine first. Require one product row per product_id; a duplicate dimension key would multiply revenue. Preserve unmatched, otherwise valid facts under an explicit unknown category.

from pyspark.sql import functions as F

def build_daily_sales(lines, products):
    valid = lines.filter("quantity > 0 AND net_unit_price >= 0")
    return (valid.join(products.select("product_id", "category"), "product_id", "left")
                 .withColumn("category", F.coalesce(F.col("category"), F.lit("UNKNOWN")))
                 .groupBy("order_date", "category")
                 .agg(F.sum(F.col("quantity") * F.col("net_unit_price")).alias("revenue")))

Test unmatched product keys, duplicates, nulls and decimal overflow/precision. Reserve the unknown label or use a dedicated dimension member to avoid confusing it with a real category. Avoid collect() for production-scale data and arbitrary repartitioning without plan evidence.

Quarantine rather than silently coerce¶

Lakeflow expectations can warn, drop or fail an update depending on policy. Quarantine patterns route invalid records with original payload, rule ID, reason, source position, detected timestamp and reprocessing status. With Auto Loader in classic jobs, implement equivalent valid/invalid branches and preserve checkpoint semantics.

Define which defects block publication versus permit degraded operation. Measure total, valid, invalid and duplicate counts; reconcile input = accepted + quarantined when the policy supports it. Protect quarantine access because invalid rows often contain sensitive fields.

Related item: A dead-letter location without an owner, remediation SLA and replay path is a data graveyard—not a quality system.

Delta Sharing supports Databricks-to-Databricks sharing and an open protocol for external recipients. A provider controls share contents and recipient access; the recipient queries live shared data without owning the provider's files. D2D can use Databricks identities/objects; open sharing uses recipient credentials/tokens that require secure rotation and revocation.

Lakehouse Federation uses governed connections and foreign catalogs to query supported external systems without first ingesting them. Choose it when freshness and no-copy access outweigh remote latency, pushdown limitations and source-capacity dependency.

Requirement	Starting point
Share live governed assets outward	Delta Sharing
Databricks recipient with native collaboration	D2D sharing
Non-Databricks recipient	Open sharing protocol
Query an operational system in place	Federation
Repeated heavy transformation, isolation or strict SLA	Ingest/materialize with an owned freshness contract

Record data owner, recipient, objects/columns, update/freshness, revoke path, audit evidence and egress/cost. Do not confuse a share with copying or federation with replication.

5. Monitoring and alerting (10%)¶

Build observability by layer¶

System tables expose supported account/workspace operational data for billing, audit, compute, jobs, pipelines and query history. Scope, region availability, retention and schema vary; query them through Unity Catalog permissions.
Query Profile explains SQL operators, scans, joins, shuffle and skew.
Spark UI explains jobs/stages/tasks, executor work, shuffle, spill, skew and memory pressure.
Jobs/Pipelines UI and REST API expose run/update state and lifecycle.
The pipeline event log supplies update, flow, expectation, progress and error evidence.

Correlate by run/update/query identifiers and time window. A billing spike without workload attribution, or a failed task without source/checkpoint state, is incomplete evidence.

Alert on an actionable condition¶

A SQL alert evaluates a query result against a threshold. For data quality, write a stable query such as invalid ratio, freshness lag or reconciliation difference; define schedule, threshold, empty/error behavior, notification destination, owner, deduplication and runbook. Job notifications cover lifecycle/performance events; they do not replace business-data alerts.

Avoid alerting on every transient symptom. Route warnings and failures differently, include environment/resource/run links, and test delivery plus recovery. Review job notifications because destinations and system-destination behavior are volatile.

Related item: Monitoring tells you what happened; an SLO says what acceptable service means; an alert should identify an actionable breach before the user discovers it.

6. Cost and performance optimisation (13%)¶

Diagnose the bottleneck before choosing a lever¶

Use a controlled loop: reproduce, capture plan/profile/Spark evidence, identify the dominant scan/shuffle/skew/spill/remote/queue/driver condition, change one factor, compare correctness and cost. Bigger compute can reduce elapsed time while increasing total cost and cannot correct a Cartesian join or single hot key.

Understand table maintenance and layout¶

Unity Catalog managed tables reduce manual lifecycle and maintenance burden and enable managed features such as predictive optimization where eligible. External tables retain external data-lifecycle responsibility.

Deletion vectors can mark changed/deleted rows without rewriting whole data files for supported operations; later maintenance materializes changes.
Data skipping uses file statistics to avoid reading irrelevant files; selective predicates and useful layout make it effective.
File pruning avoids irrelevant partitions/files. High-cardinality or tiny partitioning creates metadata and small-file costs.
Liquid clustering incrementally clusters by chosen keys and is generally more flexible than fixed partitions or Z-order for eligible Delta tables.
Predictive optimization can automate maintenance on eligible managed tables; verify account/runtime/region and cost behavior.

Use liquid clustering, deletion vectors, and predictive optimization as current sources.

Apply CDF and incremental processing from a contract¶

Change Data Feed records row-level changes after enablement within table-history retention. It can drive incremental downstream work and address some full-read latency limitations. It does not contain changes from before enablement and is not an indefinite archive. Persist downstream progress and define behavior after the starting version ages out.

A direct Delta stream normally expects append-only changes. skipChangeCommits bypasses modifying commits; it does not propagate their updates or deletes. For mutable sources, read CDF and handle the operation types deliberately. Its update_preimage is the old row, while update_postimage is the new row; do not turn both into competing current-state updates. See the CDF reference.

Related item: Checkpoint state tracks a streaming query; CDF exposes table changes; AUTO CDC applies ordered changes to a target. They are related but not interchangeable state mechanisms.

7. Ensuring data security and compliance (10%)¶

Layer least privilege¶

Workspace ACLs protect notebooks, folders, jobs, pipelines and other workspace objects.
Unity Catalog privileges protect catalogs, schemas, tables, views, volumes, functions and other securables.
Compute policies and CAN USE-style permissions constrain execution resources.
Secrets should be referenced through supported secret management and never logged, embedded in bundles or returned as data.
Row filters and column masks enforce supported table-level dynamic policies; ABAC can centralize tag-driven policy where eligible.

The Unity Catalog access-control overview maps privileges, ownership, ABAC, filters/masks and workspace bindings. Grant to account groups/workload identities and separate deployer, runtime owner and data consumer.

Distinguish privacy transformations¶

Technique	Effect	Boundary
Hashing	One-way digest; vulnerable to dictionary/linkage depending on input/salt	often pseudonymization, not guaranteed anonymity
Tokenization	Replaces value with controlled token; mapping may permit reversal	mapping store becomes highly sensitive
Suppression	Removes/redacts fields or rows	can reduce utility and leave linkage through other fields
Generalization	Coarsens values such as age band or region	assess re-identification across combined quasi-identifiers
Masking	Changes value shown to a principal/context	source may remain present and accessible to privileged users

Build PII detection/classification before masking, cover batch and streaming paths, quarantine detection uncertainty, and test unauthorized identities. Privacy is not achieved by printing “REDACTED” in one notebook.

Prove retention and purge¶

Define data classes, legal/business retention, authoritative locations, replicas/shares/caches/checkpoints/backups, delete trigger, execution SLA and evidence. Delete or anonymize eligible records in tables; propagate to downstream materializations; revoke shared access; account for time travel and file cleanup. Aggressive VACUUM can break readers and recovery, while deleting a catalog entry alone may not delete external data.

A deletion vector can make a row invisible without removing its bytes from existing files. In a disposable exercise, inspect deletion evidence, run REORG TABLE catalog.schema.table APPLY (PURGE) where supported, then retain evidence of eligible old-file cleanup through VACUUM after the applicable retention period. REORG rewrites affected files; old copies can still exist until cleanup. Do not shorten retention to make a test look successful. See REORG TABLE. This is table-level technical evidence, not a determination that every copy or legal obligation has been addressed.

VERIFY CURRENT: retention, predictive maintenance, fine-grained controls, serverless identity and regional compliance behavior change. Validate in the actual cloud/account and involve legal/privacy owners.

8. Data governance (7%)¶

Make data discoverable and govern inheritance¶

Add meaningful catalog/schema/table/column descriptions, ownership, tags, domain, sensitivity, quality/freshness expectation and certification only through a controlled stewardship process. Use lineage for impact and discovery, but treat it as supported captured evidence rather than an exhaustive dependency database.

Unity Catalog privileges inherit downward from catalog/schema grants; metastore grants and ownership have broader effects. To select a table, a principal normally requires USE CATALOG, USE SCHEMA and SELECT, whether directly or inherited. Avoid redundant grants and group sprawl; record the intended effective access, then test it. Review the current privileges reference.

Related item: Metadata tags support discovery and ABAC; certification communicates steward-reviewed trust; neither automatically grants access or guarantees a metric's business definition.

9. Debugging and deploying (10%)¶

Triage failures from symptom to state¶

Identify environment, resource, run/update/task/query ID, code/bundle revision and principal.
Read the first causal error, not only the final wrapper.
Check job/pipeline events, cluster/driver/executor logs, system tables, Query Profile and Spark UI as appropriate.
Determine the commit boundary: which tasks/tables/external side effects succeeded?
Preserve evidence, correct the smallest root cause, repair from a safe task boundary or rerun idempotently, then reconcile outputs.

Job repair can rerun failed/skipped tasks and optionally dependent tasks with parameter overrides. It does not undo successful side effects. Parameters used for recovery must be auditable and must not bypass quality/security controls.

For pipeline failures, inspect event-log error context, flow progress, expectations, source offsets/files, schema/checkpoint state and Spark evidence. A full refresh can replace state/data and is not a generic retry.

Deploy from version control¶

Use Git folders for collaborative development and source synchronization, but deploy production resources through reviewed CI/CD and a bundle or equivalent declarative contract. A robust flow runs formatting/static checks, unit tests, bundle validation, security/dependency checks, isolated integration deployment/tests, approval and production deployment with a workload identity.

Keep secrets out of Git and CI logs. Use environment targets and least-privilege service principals. Tag deployed resources with revision/environment and retain validation/deployment evidence. Review current CI/CD guidance.

Related item: CI validates and packages change; CD applies it to an environment. Git synchronization alone is neither a release approval nor a reproducible infrastructure deployment.

10. Data modelling (6%)¶

Design Delta tables from workload and lifecycle¶

Declare grain, business keys, surrogate keys, change semantics, access predicates, retention and ownership. Use Delta for transaction/history/schema/table features, then select managed/external lifecycle. Avoid encoding every consumer into one wide mutable table.

Partitioning works best for low-cardinality columns commonly used for pruning and with enough data per partition; overpartitioning creates tiny files/directories. Liquid clustering supports evolving multi-column access patterns without fixed directory partitions and can replace many partition/Z-order designs for eligible tables. Measure rather than copying a column choice from an unrelated workload.

Build dimensional models for analytics¶

A fact table has a declared event/snapshot grain and foreign keys to conformed dimensions. Measures may be additive, semi-additive or non-additive. Dimensions carry descriptive context and can preserve history with slowly changing patterns. Use unknown members or an explicit unmatched-key workflow to prevent silent fact loss.

Medallion layers and dimensional modeling answer different questions: bronze/silver/gold describe progressive quality/serving boundaries; star schemas describe consumer relationships and grain. A gold layer can contain multiple dimensional marts.

Related item: Physical layout (partitioning/clustering), logical model (facts/dimensions) and incremental semantics (append/merge/CDC) are three separate decisions that must agree with the workload.

Integrated decision scenarios¶

Scenario A — regulated customer-event platform¶

Cloud files and a message bus supply historical and live customer events. Define immutable identifiers, schema/checkpoint/watermark state and Auto Loader/stream contracts. Land replayable bronze, quarantine malformed/PII classification failures, apply AUTO CDC into conformed customer history, and publish dimensional marts. Enforce least privilege, masks and privacy transformations; design purge across tables, CDF consumers, shares and history. Deploy via bundle and prove replay, denied access, late data and purge evidence.

Scenario B — low-latency shared supply data¶

Internal sources use Delta while a partner database is federated and an external consumer needs open sharing. Compare federation versus ingestion based on latency/source load/SLA, create a governed connection, validate pushdown, and publish only approved objects via Delta Sharing. Monitor query/source cost and share access. Materialize repeated results only with an owned freshness process and revoke/test recipient credentials.

Scenario C — failing and expensive production pipeline¶

An hourly pipeline exceeds SLA and fails after a downstream task side effect. Correlate job run, pipeline event log, query profile, Spark stages and billing/system tables. Identify skew/shuffle and a non-idempotent external write. Repair from a proven boundary, add a commit ledger/idempotency key, fix join/layout after a controlled comparison, add SLO/data-quality alerts and deploy through tested CI/CD. Prove totals and cost, not just faster completion.

Worked failure cases and diagnostic answers¶

These are original learning exercises, separate from the vendor's sample questions.

Failure case	Expected reasoning and evidence
Batch 12 reaches the revenue table; the quarantine write fails	Preserve the checkpoint and application identity. On retry, the completed revenue write is recognized and quarantine can finish. Verify each table and reconcile input counts; there was a period of partial visibility.
Customer 8 has two different states at sequence 40	Reject the ambiguous source contract or obtain an authoritative secondary order. Arrival order cannot establish the correct history.
Historical seed sequence 80 overlaps live sequence 79 for one key	The per-key cutover is invalid. Resolve the source boundary before loading; one global timestamp does not prove every key's ordering.
A dashboard no longer shows a deleted row	That proves query visibility only. Check file rewrite/cleanup, history and downstream copies before concluding physical purge.
A valid order references an absent product dimension	Preserve the fact as unknown or route it to an owned reconciliation process. An inner join would silently reduce totals.

Does disabling task retries stop every repeat? No. Serverless auto-optimization is a separate control, and manual repairs or repeated submissions still need idempotent effects.
Does one checkpoint make two destinations atomic? No. Inspect separate commits and retry the incomplete work safely.
Can an empty micro-batch occur? Yes; handle it without fabricating a business event.
Does a latest-timestamp sort resolve all CDC ordering? No; different changes can tie and need an authoritative order.
Can a backfill be replayed after a full refresh? Yes; retain its source and plan for the reset.
Does skipping source change commits synchronize deletions? No; use an explicit change propagation strategy.
What does an update preimage represent? The prior row, not another new state to append blindly.
Does REORG ... PURGE alone prove old files disappeared? No; old-file retention and cleanup remain separate evidence.
Why test product-key uniqueness before aggregation? A duplicate match multiplies fact rows and overstates revenue.
What belongs in a performance comparison? Equal data and semantics, controlled compute/cache conditions, correctness totals and total cost alongside duration.

Review evidence — September 28, 2026: a local SQLite exercise checked portable join and aggregate behavior, including missing and duplicate dimension keys. An in-memory model checked two-destination failure/retry and checkpoint-identity reasoning. The Python example and YAML fragment were parsed. These checks did not run Spark, Delta, Databricks Jobs, AUTO CDC, or physical purge. All eight workspace labs below remain proposed.

Hands-on lab sequence¶

Project and tests: Package two DataFrame transforms in a wheel; pin dependencies and add null/duplicate/empty/skew fixtures using DataFrame/schema equality helpers.
Batch plus stream acquisition: Process historical files then new arrivals through Auto Loader/Structured Streaming; record schema/checkpoint state and prove replay behavior.
Declarative CDC and quality: Build a Lakeflow pipeline with expectations, quarantine and AUTO CDC/SCD behavior; test duplicates, deletes and out-of-order events.
Sharing/federation: Create a written or real connection/foreign catalog and share/recipient design; test pushdown/failure and revoke access.
Observability: Correlate system table, job/pipeline event, Query Profile and Spark UI evidence; implement and test one quality alert and one run notification.
Cost/performance: Benchmark a representative query before/after one layout, join or compute change under controlled cache/data conditions; record cost and correctness.
Privacy/governance: Build group grants, row/column protection, discoverability metadata and a retention/purge evidence matrix; test authorized and denied paths.
Deployment and repair: Define dev/test/prod bundle targets and CI checks, deploy to isolation, inject a downstream failure, repair safely and reconcile side effects.

Readiness checks¶

Code, acquisition and quality¶

Security, governance, deployment and modeling¶

Places to learn¶

This is not a complete list, and it is not a recommendation to consume everything. Pick the route that closes measured gaps, then spend substantial time building, breaking, observing, repairing and redeploying a system. Public resource availability was checked September 28, 2026. Durations are editorial planning estimates, not confirmed paid-course runtimes. Signed-in Academy lessons were not inspected; Udemy and O'Reilly access was blocked, and Whizlabs returned no useful page body.

Resource	Access	Estimated time
Official certification page and July 3, 2026 exam guide	Free	2–3 hours to map every objective and inspect the vendor sample format; do not reproduce questions
Databricks Academy — Advanced Data Engineering with Databricks plus four named self-paced courses	Free account, customer/partner entitlement varies	30–50 hours with labs; verify course visibility and runtime after sign-in
Databricks Free Edition or an authorized workspace	Free account/organizational	25–45 hours for the eight labs and failure experiments; some enterprise features require another environment
Databricks documentation	Free	10–18 hours selected reproduction across pipelines, Jobs, governance, performance and deployment
CarlosR, Databricks employee: AUTO CDC engineering article (April 24, 2026)	Free technical blog	About 45–75 minutes to read and sketch replay/ordering tests; estimate includes the worksheet. Corroborate against current CDC docs; its simplified examples do not establish tie handling or replace a source contract.
Databricks YouTube	Free	4–8 hours selected recent Data + AI Summit, engineering, Lakeflow, Unity Catalog and performance sessions
Whizlabs: Databricks Data Engineer Professional	Paid; training/practice product	Public stable totals were not exposed; budget 8–18 hours and verify July 2026 alignment after sign-in
Udemy search: Data Engineer Professional	Paid marketplace	10–25 hours if a course demonstrably maps to the July 2026 guide; verify instructor, update, outline and avoid dump-focused listings
O'Reilly search: Databricks data engineering	Paid/trial	8–20 hours selected current book/video/live material; map exact chapters to the blueprint and verify publication date

Use vendor sample questions only through the official guide, and use ethical practice tests to identify weak domains rather than memorize recalled content. Recheck the live weights, linked PDF, renamed products, course catalog and all VERIFY CURRENT controls near the appointment.