Download the Latest Associate-Developer-Apache-Spark-3.5 Dumps – 2025 Associate-Developer-Apache-Spark-3.5 Exam Questions [Q62-Q84]

4/5 - (1 vote)

Download the Latest Associate-Developer-Apache-Spark-3.5 Dumps – 2025 Associate-Developer-Apache-Spark-3.5 Exam Questions

Latest Databricks Associate-Developer-Apache-Spark-3.5 Certification Practice Test Questions

NO.62 A data scientist wants each record in the DataFrame to contain:
The first attempt at the code does read the text files but each record contains a single line. This code is shown below:

The entire contents of a file
The full file path
The issue: reading line-by-line rather than full text per file.
Code:
corpus = spark.read.text(“/datasets/raw_txt/*”)
.select(‘*’, ‘_metadata.file_path’)
Which change will ensure one record per file?
Options:

Add the option wholetext=True to the text() function

Add the option lineSep=’n’ to the text() function

Add the option wholetext=False to the text() function

Add the option lineSep=”, ” to the text() function

NO.63 18 of 55.
An engineer has two DataFrames – df1 (small) and df2 (large). To optimize the join, the engineer uses a broadcast join:
from pyspark.sql.functions import broadcast
df_result = df2.join(broadcast(df1), on=”id”, how=”inner”)
What is the purpose of using broadcast() in this scenario?

It increases the partition size for df1 and df2.

It ensures that the join happens only when the id values are identical.

It reduces the number of shuffle operations by replicating the smaller DataFrame to all nodes.

It filters the id values before performing the join.

NO.64 An engineer has a large ORC file located at /file/test_data.orc and wants to read only specific columns to reduce memory usage.
Which code fragment will select the columns, i.e., col1, col2, during the reading process?

spark.read.orc(“/file/test_data.orc”).filter(“col1 = ‘value’ “).select(“col2”)

spark.read.format(“orc”).select(“col1”, “col2”).load(“/file/test_data.orc”)

spark.read.orc(“/file/test_data.orc”).selected(“col1”, “col2”)

spark.read.format(“orc”).load(“/file/test_data.orc”).select(“col1”, “col2”)

NO.65 How can a Spark developer ensure optimal resource utilization when running Spark jobs in Local Mode for testing?
Options:

Configure the application to run in cluster mode instead of local mode.

Increase the number of local threads based on the number of CPU cores.

Use the spark.dynamicAllocation.enabled property to scale resources dynamically.

Set the spark.executor.memory property to a large value.

NO.66 A developer notices that all the post-shuffle partitions in a dataset are smaller than the value set forspark.sql.
adaptive.maxShuffledHashJoinLocalMapThreshold.
Which type of join will Adaptive Query Execution (AQE) choose in this case?

A Cartesian join

A shuffled hash join

A broadcast nested loop join

A sort-merge join

NO.67 Given the following code snippet in my_spark_app.py:

What is the role of the driver node?

The driver node orchestrates the execution by transforming actions into tasks and distributing them to worker nodes

The driver node only provides the user interface for monitoring the application

The driver node holds the DataFrame data and performs all computations locally

The driver node stores the final result after computations are completed by worker nodes

NO.68 Given the schema:

event_ts TIMESTAMP,
sensor_id STRING,
metric_value LONG,
ingest_ts TIMESTAMP,
source_file_path STRING
The goal is to deduplicate based on: event_ts, sensor_id, and metric_value.
Options:

dropDuplicates on all columns (wrong criteria)

dropDuplicates with no arguments (removes based on all columns)

groupBy without aggregation (invalid use)

dropDuplicates on the exact matching fields

NO.69 What is the relationship between jobs, stages, and tasks during execution in Apache Spark?
Options:

A job contains multiple stages, and each stage contains multiple tasks.

A job contains multiple tasks, and each task contains multiple stages.

A stage contains multiple jobs, and each job contains multiple tasks.

A stage contains multiple tasks, and each task contains multiple jobs.

NO.70 27 of 55.
A data engineer needs to add all the rows from one table to all the rows from another, but not all the columns in the first table exist in the second table.
The error message is:
AnalysisException: UNION can only be performed on tables with the same number of columns.
The existing code is:
au_df.union(nz_df)
The DataFrame au_df has one extra column that does not exist in the DataFrame nz_df, but otherwise both DataFrames have the same column names and data types.
What should the data engineer fix in the code to ensure the combined DataFrame can be produced as expected?

df = au_df.unionByName(nz_df, allowMissingColumns=True)

df = au_df.unionAll(nz_df)

df = au_df.unionByName(nz_df, allowMissingColumns=False)

df = au_df.union(nz_df, allowMissingColumns=True)

NO.71 45 of 55.
Which feature of Spark Connect should be considered when designing an application that plans to enable remote interaction with a Spark cluster?

It is primarily used for data ingestion into Spark from external sources.

It provides a way to run Spark applications remotely in any programming language.

It can be used to interact with any remote cluster using the REST API.

It allows for remote execution of Spark jobs.

NO.72 A data scientist at a financial services company is working with a Spark DataFrame containing transaction records. The DataFrame has millions of rows and includes columns fortransaction_id,account_number, transaction_amount, andtimestamp. Due to an issue with the source system, some transactions were accidentally recorded multiple times with identical information across all fields. The data scientist needs to remove rows with duplicates across all fields to ensure accurate financial reporting.
Which approach should the data scientist use to deduplicate the orders using PySpark?

df = df.dropDuplicates()

df = df.groupBy(“transaction_id”).agg(F.first(“account_number”), F.first(“transaction_amount”), F.first (“timestamp”))

df = df.filter(F.col(“transaction_id”).isNotNull())

df = df.dropDuplicates([“transaction_amount”])

NO.73 A data engineer is working on the DataFrame:

(Referring to the table image: it has columns Id, Name, count, and timestamp.) Which code fragment should the engineer use to extract the unique values in the Name column into an alphabetically ordered list?

df.select(“Name”).orderBy(df[“Name”].asc())

df.select(“Name”).distinct().orderBy(df[“Name”])

df.select(“Name”).distinct()

df.select(“Name”).distinct().orderBy(df[“Name”].desc())

NO.74 A data engineer wants to create an external table from a JSON file located at/data/input.jsonwith the following requirements:
Create an external table namedusers
Automatically infer schema
Merge records with differing schemas
Which code snippet should the engineer use?
Options:

CREATE TABLE users USING json OPTIONS (path ‘/data/input.json’)

CREATE EXTERNAL TABLE users USING json OPTIONS (path ‘/data/input.json’)

CREATE EXTERNAL TABLE users USING json OPTIONS (path ‘/data/input.json’, mergeSchema
‘true’)

CREATE EXTERNAL TABLE users USING json OPTIONS (path ‘/data/input.json’, schemaMerge
‘true’)

NO.75 A data engineer wants to create a Streaming DataFrame that reads from a Kafka topic called feed.

Which code fragment should be inserted in line 5 to meet the requirement?
Code context:
spark
.readStream
.format(“kafka”)
.option(“kafka.bootstrap.servers”, “host1:port1,host2:port2”)
.[LINE 5]
.load()
Options:

.option(“subscribe”, “feed”)

.option(“subscribe.topic”, “feed”)

.option(“kafka.topic”, “feed”)

.option(“topic”, “feed”)

NO.76 A developer is running Spark SQL queries and notices underutilization of resources. Executors are idle, and the number of tasks per stage is low.
What should the developer do to improve cluster utilization?

Increase the value of spark.sql.shuffle.partitions

Reduce the value of spark.sql.shuffle.partitions

Increase the size of the dataset to create more partitions

Enable dynamic resource allocation to scale resources as needed

NO.77 48 of 55.
A data engineer needs to join multiple DataFrames and has written the following code:
from pyspark.sql.functions import broadcast
data1 = [(1, “A”), (2, “B”)]
data2 = [(1, “X”), (2, “Y”)]
data3 = [(1, “M”), (2, “N”)]
df1 = spark.createDataFrame(data1, [“id”, “val1”])
df2 = spark.createDataFrame(data2, [“id”, “val2”])
df3 = spark.createDataFrame(data3, [“id”, “val3”])
df_joined = df1.join(broadcast(df2), “id”, “inner”)
.join(broadcast(df3), “id”, “inner”)
What will be the output of this code?

The code will work correctly and perform two broadcast joins simultaneously to join df1 with df2, and then the result with df3.

The code will fail because only one broadcast join can be performed at a time.

The code will fail because the second join condition (df2.id == df3.id) is incorrect.

The code will result in an error because broadcast() must be called before the joins, not inline.

NO.78 11 of 55.
Which Spark configuration controls the number of tasks that can run in parallel on an executor?

spark.executor.cores

spark.task.maxFailures

spark.executor.memory

spark.sql.shuffle.partitions

NO.79 Given the schema:

event_ts TIMESTAMP,
sensor_id STRING,
metric_value LONG,
ingest_ts TIMESTAMP,
source_file_path STRING
The goal is to deduplicate based on: event_ts, sensor_id, and metric_value.
Options:

dropDuplicates on all columns (wrong criteria)

dropDuplicates with no arguments (removes based on all columns)

groupBy without aggregation (invalid use)

dropDuplicates on the exact matching fields

NO.80 25 of 55.
A Data Analyst is working on employees_df and needs to add a new column where a 10% tax is calculated on the salary.
Additionally, the DataFrame contains the column age, which is not needed.
Which code fragment adds the tax column and removes the age column?

employees_df = employees_df.withColumn(“tax”, col(“salary”) * 0.1).drop(“age”)

employees_df = employees_df.withColumn(“tax”, lit(0.1)).drop(“age”)

employees_df = employees_df.dropField(“age”).withColumn(“tax”, col(“salary”) * 0.1)

employees_df = employees_df.withColumn(“tax”, col(“salary”) + 0.1).drop(“age”)

NO.81 A data scientist has identified that some records in the user profile table contain null values in any of the fields, and such records should be removed from the dataset before processing. The schema includes fields like user_id, username, date_of_birth, created_ts, etc.
The schema of the user profile table looks like this:

Which block of Spark code can be used to achieve this requirement?
Options:

filtered_df = users_raw_df.na.drop(thresh=0)

filtered_df = users_raw_df.na.drop(how=’all’)

filtered_df = users_raw_df.na.drop(how=’any’)

filtered_df = users_raw_df.na.drop(how=’all’, thresh=None)

NO.82 An engineer wants to join two DataFramesdf1anddf2on the respectiveemployee_idandemp_idcolumns:
df1:employee_id INT,name STRING
df2:emp_id INT,department STRING
The engineer uses:
result = df1.join(df2, df1.employee_id == df2.emp_id, how=’inner’)
What is the behaviour of the code snippet?

The code fails to execute because the column names employee_id and emp_id do not match automatically

The code fails to execute because it must use on=’employee_id’ to specify the join column explicitly

The code fails to execute because PySpark does not support joining DataFrames with a different structure

The code works as expected because the join condition explicitly matches employee_id from df1 with emp_id from df2

NO.83 30 of 55.
A data engineer is working on a num_df DataFrame and has a Python UDF defined as:
def cube_func(val):
return val * val * val
Which code fragment registers and uses this UDF as a Spark SQL function to work with the DataFrame num_df?

spark.udf.register(“cube_func”, cube_func)
num_df.selectExpr(“cube_func(num)”).show()

num_df.select(cube_func(“num”)).show()

spark.createDataFrame(cube_func(“num”)).show()

num_df.register(“cube_func”).select(“num”).show()

NO.84 A Spark application developer wants to identify which operations cause shuffling, leading to a new stage in the Spark execution plan.
Which operation results in a shuffle and a new stage?

DataFrame.groupBy().agg()

DataFrame.filter()

DataFrame.withColumn()

DataFrame.select()