Pass the Databricks Databricks Certification Databricks-Certified-Associate-Developer-for-Apache-Spark-3.5 Questions and answers with CertsForce

Comprehensive and Detailed Explanation From Exact Extract:

The correct way to load specific columns from an ORC file is to first load the file using.load()and then apply.select()on the resulting DataFrame. This is valid with.read.format("orc")or the shortcut.read.orc().

df = spark.read.format("orc").load("/file/test_data.orc").select("col1","col2")

Why others are incorrect:

Aperforms selection after filtering, but doesn’t match the intention to minimize memory at load.

Bincorrectly tries to use.select()before.load(), which is invalid.

Cuses a non-existent.selected()method.

Dcorrectly loads and then selects.

Comprehensive and Detailed Explanation From Exact Extract:

The.coalesce(numPartitions)function is used to reduce the number of partitions in a DataFrame. It does not increase the number of partitions. If the specified number of partitions is greater than the current number, it will not have any effect.

From the official Spark documentation:

“coalesce() results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim one or more of the current partitions.”

However, if you try to increase partitions using coalesce (e.g., from 10 to 20), the number of partitions remains unchanged.

Hence,df.coalesce(20)will still return a DataFrame with 10 partitions.

The most efficient and correct approach is to define a schema using StructType and pass it tospark.read.schema(...).

This avoids schema inference overhead and ensures proper data types are enforced during read.

Example:

frompyspark.sql.typesimportStructType, StructField, StringType, DoubleType

schema = StructType([

StructField("order_id", StringType(),True),

StructField("amount", DoubleType(),True),

])

df = spark.read.schema(schema).json("path/to/json")

— Source:Databricks Guide – Read JSON with predefined schema

Comprehensive and Detailed Explanation From Exact Extract:

According to the PySpark API documentation:

"getOrCreate(): Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder."

This means Spark maintains a global singleton session within a JVM process. Repeated calls togetOrCreate()return the same session, unless explicitly stopped.

Option A is incorrect: the method does not destroy any session.

Option B incorrectly ties uniqueness toappName, which does not influence session reusability.

Option D is incorrect: it contradicts the fundamental behavior ofgetOrCreate().

(Source:PySpark SparkSession API Docs)

Comprehensive and Detailed Explanation From Exact Extract:

In Apache Spark's cluster mode:

"The driver program runs on the cluster’s worker node instead of the client’s local machine. This allows the driver to be close to the data and other executors, reducing network overhead and improving fault tolerance for production jobs."

(Source: Apache Spark documentation –Cluster Mode Overview)

This deployment is ideal for production environments where the job is submitted from a gateway node, and Spark manages the driver lifecycle on the cluster itself.

Option A is partially true but less specific than D.

Option B is incorrect: the driver never executes all tasks; executors handle distributed tasks.

Option C describes client mode, not cluster mode.

Pandas API on Spark (formerly Koalas) offers:

Familiar Pandas-like syntax

Distributed execution using Spark under the hood

Scalability for large datasets across the cluster

It provides the power of Spark while retaining the productivity of Pandas.

Comprehensive and Detailed Explanation From Exact Extract:

Apache Spark employs a lazy evaluation model for transformations. This means that when transformations (e.g.,map(),filter()) are applied to a DataFrame, Spark does not execute them immediately. Instead, it builds a logical plan (lineage) of transformations to be applied.

Execution is deferred until an action (e.g.,collect(),count(),save()) is called. At that point, Spark's Catalyst optimizer analyzes the logical plan, optimizes it, and then executes the physical plan to produce the result.

This lazy evaluation strategy allows Spark to optimize the execution plan, minimize data shuffling, and improve overall performance by reducing unnecessary computations.

na.drop(how='any')drops any row that has at least one null value.

This is exactly what's needed when the goal is to retain only fully complete records.

Usage:CopyEdit

filtered_df = users_raw_df.na.drop(how='any')

Explanation of incorrect options:

A: thresh=0 is invalid — thresh must be ≥ 1.

B: how='all' drops only rows where all columns are null (too lenient).

D: spark.na.drop doesn't support mixing how and thresh in that way; it's incorrect syntax.

Comprehensive and Detailed Explanation From Exact Extract:

Spark's best practice is to estimate partition count based on data volume and a reasonable partition size — typically 128 MB to 256 MB per partition.

With 1 TB of data: 1 TB / 128 MB ≈ ~8000 partitions

This ensures that tasks are distributed across available CPUs for parallelism and that each task processes an optimal volume of data.

Option A (equal to cores) may result in partitions that are too large.

Option B (fixed 200) is arbitrary and may underutilize the cluster.

Option C (nodes) gives too few partitions (10), limiting parallelism.