Pass the Amazon Web Services AWS Certified Data Engineer Data-Engineer-Associate Questions and answers with CertsForce

Option C is correct because the required access pattern is: find all completed rides for a specific driver. In DynamoDB, when you need to query data efficiently by attributes that are not part of the base table primary key, you typically create a global secondary index (GSI). AWS documentation states that a GSI can have a key schema that is different from the base table and can use top-level attributes such as DriverID and RideStatus as its partition and sort keys. That makes it possible to query directly for a given driver and then narrow the results to completed rides, without scanning the full table.

Option A is incorrect because a local secondary index (LSI) must use the same partition key as the base table. Since the table partition key is RideID, an LSI cannot be created with DriverID as the partitioning access path.

Option B uses RiderID, which does not satisfy the requirement to retrieve rides by driver. Option D is also wrong because filter expressions are applied after items are read and therefore do not avoid scanning large amounts of data. The study guide emphasizes choosing the correct data model and access pattern for the workload, which places this question in the Data Storage and Management domain.

The correct answer is to configure AWS Glue triggers to run the ETL jobs every hour and use AWS Glue connections to establish connectivity between the data sources and Amazon Redshift. AWS Glue triggers are a way to schedule and orchestrate ETL jobs with the least operational overhead. AWS Glue connections are a way to securely connect to data sources and targets using JDBC or MongoDB drivers. AWS Glue DataBrew is a visual data preparation tool that does not support MongoDB as a data source. AWS Lambda functions are a serverless option to schedule and run ETL jobs, but they have a limit of 15 minutes for execution time, which may not be enough for complex transformations. The Redshift Data API is a way to run SQL commands on Amazon Redshift clusters without needing a persistent connection, but it does not support loading data from AWS Glue ETL jobs. References:

AWS Glue triggers

AWS Glue connections

AWS Glue DataBrew

[AWS Lambda functions]

[Redshift Data API]

The requirement to retain immutable object versions using a WORM model mandates the use of S3 Object Lock, not just S3 Versioning. Object Lock enforces immutability by preventing deletion or modification of objects for a specified retention period, which is required for compliance scenarios such as regulatory data retention.

To meet the access requirement of retrieving data within 5 hours, the appropriate archival storage class is S3 Glacier Flexible Retrieval, which supports expedited and standard retrieval within minutes to hours. S3 Glacier Deep Archive does not meet this requirement because retrieval times are typically 12 hours or longer.

Using S3 Intelligent-Tiering for active data optimizes storage costs automatically before transitioning older objects. A lifecycle policy can then transition objects older than 1 year to Glacier Flexible Retrieval and delete them after the 7-year retention period expires.

Options that rely only on S3 Versioning do not provide immutability guarantees. Options using Glacier Deep Archive fail the access-time requirement. Therefore, Option C is the only solution that satisfies WORM compliance, retrieval-time constraints, and cost optimization.

To make the S3 data accessible daily in the AWS Glue Data Catalog, the data engineer needs to create a crawler that can crawl the S3 data and write the metadata to the Data Catalog. The crawler also needs to run on a daily schedule to keep the Data Catalog updated with the latest data. Therefore, the solution must include the following steps:

Create an IAM role that has the necessary permissions to access the S3 data and the Data Catalog. The AWSGlueServiceRole policy is a managed policy that grants these permissions1.

Associate the role with the crawler.

Specify the S3 bucket path of the source data as the crawler’s data store. The crawler will scan the data and infer the schema and format2.

Create a daily schedule to run the crawler. The crawler will run at the specified time every day and update the Data Catalog with any changes in the data3.

Specify a database name for the output. The crawler will create or update a table in the Data Catalog under the specified database. The table will contain the metadata about the data in the S3 bucket, such as the location, schema, and classification.

Option B is the only solution that includes all these steps. Therefore, option B is the correct answer.

Option A is incorrect because it configures the output destination to a new path in the existing S3 bucket. This is unnecessary and may cause confusion, as the crawler does not write any data to the S3 bucket, only metadata to the Data Catalog.

Option C is incorrect because it allocates data processing units (DPUs) to run the crawler every day. This is also unnecessary, as DPUs are only used for AWS Glue ETL jobs, not crawlers.

Option D is incorrect because it combines the errors of option A and C. It configures the output destination to a new path in the existing S3 bucket and allocates DPUs to run the crawler every day, both of which are irrelevant for the crawler.

1: AWS managed (predefined) policies for AWS Glue - AWS Glue

2: Data Catalog and crawlers in AWS Glue - AWS Glue

3: Scheduling an AWS Glue crawler - AWS Glue

[4]: Parameters set on Data Catalog tables by crawler - AWS Glue

[5]: AWS Glue pricing - Amazon Web Services (AWS)

 AWS Glue is a fully managed serverless ETL service that can handle petabytes of data in seconds. AWS Glue can run Apache Spark and Apache Flink jobs without requiring any infrastructure provisioning or management. AWS Glue can also integrate with Apache Pig, Apache Oozie, and Apache Hbase using AWS Glue Data Catalog and AWS Glue workflows. AWS Glue can reduce the overall operational overhead by automating the data discovery, data preparation, and data loading processes. AWS Glue can also optimize the cost and performance of ETL jobs by using AWS Glue Job Bookmarking, AWS Glue Crawlers, and AWS Glue Schema Registry. References:

AWS Glue

AWS Glue Data Catalog

AWS Glue Workflows

[AWS Glue Job Bookmarking]

[AWS Glue Crawlers]

[AWS Glue Schema Registry]

[AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide]

To achieve the highest throughput and efficiently use cluster resources while loading data into an Amazon Redshift cluster, the optimal approach is to use a single COPY command that ingests data in parallel.

Option D: Use a single COPY command to load the data into the Redshift cluster.The COPY command is designed to load data from multiple files in parallel into a Redshift table, using all the cluster nodes to optimize the load process. Redshift is optimized for parallel processing, and a single COPY command can load multiple files at once, maximizing throughput.

Options A, B, and C either involve unnecessary complexity or inefficient approaches, such as using multiple COPY commands or INSERT statements, which are not optimized for bulk loading.

Amazon EC2 instances can use two types of storage volumes: instance store volumes and Amazon EBS volumes. Instance store volumes are ephemeral, meaning they are only attached to the instance for the duration of its life cycle. If the instance is stopped, terminated, or fails, the data on the instance store volume is lost. Amazon EBS volumes are persistent, meaning they can be detached from the instance and attached to another instance, and the data on the volume is preserved. To meet the requirement of persisting the data even if the EC2 instances are terminated, the data engineer must use Amazon EBS volumes to store the application data. The solution is to launch new EC2 instances by using an AMI that is backed by an EC2 instance store volume, which is the default option for most AMIs. Then, the data engineer must attach an Amazon EBS volume to each instance and configure the application to write the data to the EBS volume. This way, the data will be saved on the EBS volume and can be accessed by another instance if needed. The data engineer can apply the default settings to the EC2 instances, as there is no need to modify the instance type, security group, or IAM role for this solution. The other options are either not feasible or not optimal. Launching new EC2 instances by using an AMI that is backed by an EC2 instance store volume that contains the application data (option A) or by using an AMI that is backed by a root Amazon EBS volume that contains the application data (option B) would not work, as the data on the AMI would be outdated and overwritten by the new instances. Attaching an additional EC2 instance store volume to contain the application data (option D) would not work, as the data on the instance store volume would be lost if the instance is terminated. References:

Amazon EC2 Instance Store

Amazon EBS Volumes

AWS Certified Data Engineer - Associate DEA-C01 Complete Study Guide, Chapter 2: Data Store Management, Section 2.1: Amazon EC2

Options B and D are correct. AWS Prescriptive Guidance states that shuffle is a major cause of Spark performance issues and can exhaust local disk space on executors, leading to “No space left on device” failures. AWS specifically recommends assessing shuffle performance in CloudWatch metrics and in the Spark UI, and for data skew, it says to examine metrics in the Spark UI, including task duration and spill behavior across executors. That makes B the correct low-cost way to identify the source of the problem.

For remediation, AWS Glue documentation states that Spark can throw “No space left on device” when there is insufficient local disk on the executor, and that you can use Amazon S3 to store Spark shuffle data by enabling the AWS Glue Spark shuffle plugin. The S3 shuffle approach is specifically presented as a way to run shuffle-intensive jobs more reliably when they are bound by local disk capacity. In skew scenarios, using a skew-mitigation technique such as salting is also a standard way to distribute hot keys more evenly. That makes D the most cost-effective corrective step.

Options A and C increase cost by adding capacity before diagnosing or directly fixing the shuffle bottleneck. Option E is less precise than the Spark UI and Glue metrics for identifying skew.