Pre-Summer Special Limited Time 70% Discount Offer - Ends in 0d 00h 00m 00s - Coupon code: force70

Databricks Certified Data Engineer Professional Exam Databricks-Certified-Professional-Data-Engineer Question # 41 Topic 5 Discussion

Databricks Certified Data Engineer Professional Exam Databricks-Certified-Professional-Data-Engineer Question # 41 Topic 5 Discussion

Databricks-Certified-Professional-Data-Engineer Exam Topic 5 Question 41 Discussion:
Question #: 41
Topic #: 5

A task orchestrator has been configured to run two hourly tasks. First, an outside system writes Parquet data to a directory mounted at /mnt/raw_orders/. After this data is written, a Databricks job containing the following code is executed:

(spark.readStream

.format( " parquet " )

.load( " /mnt/raw_orders/ " )

.withWatermark( " time " , " 2 hours " )

.dropDuplicates([ " customer_id " , " order_id " ])

.writeStream

.trigger(once=True)

.table( " orders " )

)

Assume that the fields customer_id and order_id serve as a composite key to uniquely identify each order, and that the time field indicates when the record was queued in the source system. If the upstream system is known to occasionally enqueue duplicate entries for a single order hours apart, which statement is correct?


A.

The orders table will not contain duplicates, but records arriving more than 2 hours late will be ignored and missing from the table.


B.

The orders table will contain only the most recent 2 hours of records and no duplicates will be present.


C.

All records will be held in the state store for 2 hours before being deduplicated and committed to the orders table.


D.

Duplicate records enqueued more than 2 hours apart may be retained and the orders table may contain duplicate records with the same customer_id and order_id.


Get Premium Databricks-Certified-Professional-Data-Engineer Questions

Contribute your Thoughts:


Chosen Answer:
This is a voting comment (?). It is better to Upvote an existing comment if you don't have anything to add.