Pass the Databricks Databricks Certification Databricks-Certified-Professional-Data-Engineer Questions and answers with CertsForce

Exact extract: “dropDuplicates with watermark performs stateful deduplication on the keys within the watermark delay.”

Exact extract: “Records older than the event-time watermark are considered late and may be dropped.”

When adding new fields to a Delta table ' s schema, these fields will not be retrospectively applied to historical records that were ingested before the schema change. Consequently, while the team can use the new metadata fields to investigate transient processing delays moving forward, they will be unable to apply this diagnostic approach to past data that lacks these fields.

Lakeflow Declarative Pipelines (DLT) supports data quality enforcement using @dlt.expect, @dlt.expect_or_drop, and @dlt.expect_all.

@dlt.expect applies a rule and records whether rows pass or fail the condition but does not drop failing rows . Instead, failing rows can be written to a quarantine table.

@dlt.expect_or_drop enforces that only rows passing the condition flow downstream, dropping bad records automatically.

In this case, the requirement is:

Good rows (reading < 120) go to the silver table .

Bad rows (reading > = 120) go to a quarantine table .

Bad rows should not be included in silver.

The correct implementation is Option A , where:

The silver table uses @dlt.expect to validate reading < 120. These rows flow normally.

The quarantine table applies an expectation for reading > = 120, ensuring bad records are captured separately.

Other options are incorrect:

Option B/D: These either use expect_or_drop incorrectly or apply wrong conditions, leading to dropped rows without quarantining properly.

Option C: Uses expect_or_drop for both tables, which would discard bad rows instead of persisting them into a quarantine table.

Thus, Option A meets the business requirement to split good and bad data streams while ensuring both are captured for auditing and processing.

https://docs.databricks.com/en/lakehouse/data-objects.html

 This is the correct answer because it addresses the situation while minimally interrupting other teams in the organization without increasing the number of tables that need to be managed. The situation is that an aggregate table used by numerous teams across the organization will need to have a number of fields renamed, and additional fields will also be added, due to new requirements from a customer-facing application. By configuring a new table with all the requisite fields and new names and using this as the source for the customer-facing application, the data engineering team can meet the new requirements without affecting other teams that rely on the existing table schema and name. By creating a view that maintains the original data schema and table name by aliasing select fields from the new table, the data engineering team can also avoid duplicating data or creating additional tables that need to be managed. Verified References: [Databricks Certified Data Engineer Professional], under “Lakehouse” section; Databricks Documentation, under “CREATE VIEW” section.

Databricks recommends applying data masking or anonymization before persisting PII to ensure compliance with privacy regulations such as GDPR and HIPAA. In a Lakeflow Declarative Pipeline , developers can define custom Python or SQL-based masking functions to standardize PII handling across both batch and streaming inputs.

This approach ensures that data entering the Delta Lake is already anonymized, guaranteeing consistent and auditable behavior. By applying masking during ingestion (in the Bronze layer), audit trails are preserved through pipeline event logs.

While Unity Catalog column masks (option C) can enforce dynamic masking at query time, they do not prevent PII storage. Thus, option D aligns with the best practice of securing PII before storage , while still supporting reproducibility and analytics usability.

Exact extract: “A shallow clone creates a copy of the metadata that references the source data files; it is fast and inexpensive.”

Exact extract: “A deep clone copies the data.”

Delta Lake introduced Delta Universal Format (Delta UniForm) , which allows seamless interoperability between Delta Lake and Apache Iceberg . This means a Delta table can be converted into an Iceberg table while maintaining Delta capabilities .

Explanation of Each Option:

(A) Require the analytics team to use a tool that supports Delta table

Incorrect: While Delta Lake is widely used, requiring the team to change tools is not a flexible or scalable solution .

(B) Enable uniform on the transactions table to ' iceberg ' so that the table can be read as an Iceberg table

Incorrect:

The uniform feature must be enabled after conversion .

You cannot directly enable uniform without first converting the table .

(C) Create an Iceberg copy of the transactions Delta table which can be used by the analytics team

Incorrect:

Creating a separate Iceberg copy would duplicate storage and increase maintenance complexity .

This is not necessary when Delta UniForm allows direct compatibility with Iceberg .

(D) Convert the transactions Delta table to Iceberg and enable uniform so that the table can be read as a Delta table

Correct:

The best approach is to convert the existing Delta table to Iceberg using the Databricks Delta to Iceberg migration tools .

After conversion, enabling uniform ensures the table remains accessible in both Delta and Iceberg formats .

Conclusion:

The best practice for interoperability between Delta and Iceberg is to convert the Delta table to Iceberg and enable uniform , ensuring cross-compatibility without data duplication.

Thus, Option (D) is the correct answer .

When a Databricks job runs multiple tasks with dependencies, the tasks are executed in a dependency graph. If a task fails, the downstream tasks that depend on it are skipped and marked as Upstream failed. However, the failed task may have already committed some changes to the Lakehouse before the failure occurred, and those changes are not rolled back automatically. Therefore, the job run may result in a partial update of the Lakehouse. To avoid this, you can use the  transactional writes  feature of Delta Lake to ensure that the changes are only committed when the entire job run succeeds. Alternatively, you can use the  Run if  condition to configure tasks to run even when some or all of their dependencies have failed, allowing your job to recover from failures and continue running. References:

transactional writes : https://docs.databricks.com/delta/delta-intro.html#transactional-writes

Run if : https://docs.databricks.com/en/workflows/jobs/conditional-tasks.html

The Databricks CLI provides a way to automate interactions with Databricks services. When dealing with pipelines, you can use the databricks pipelines get --pipeline-id command to capture the settings of an existing pipeline in JSON format. This JSON can then be modified by removing the pipeline_id to prevent conflicts and renaming the pipeline to create a new pipeline. The modified JSON file can then be used with the databricks pipelines create command to create a new pipeline with those settings.