You orchestrate ETL pipelines by using Cloud Composer One of the tasks in the Apache Airflow directed acyclic graph (DAG) relies on a third-party service. You want to be notified when the task does not succeed. What should you do?
A.
Configure a Cloud Monitoring alert on the sla_missed metric associated with the task at risk to trigger a notification.
B.
Assign a function with notification logic to the sla_miss_callback parameter for the operator responsible for the task at risk.
C.
Assign a function with notification logic to the on_retry_callback parameter for the operator responsible for the task at risk.
D.
Assign a function with notification logic to the on_failure_callback parameter for the operator responsible for the task at risk.
By assigning a function with notification logic to the on_failure_callback parameter, you can customize the action that is taken when a task fails in your DAG1. For example, you can send anemail, a Slack message, or a PagerDuty alert to notify yourself or your team about the task failure2. This option is more flexible and reliable than configuring a Cloud Monitoring alert onthe sla_missed metric, which only triggers when a task misses its scheduled deadline3. The sla_miss_callback parameter is also related to the sla_missed metric, and it is executed when the task instance has not succeeded and the time is past the task’s scheduled execution date plus its sla4. The on_retry_callback parameter is executed before a task is retried4. These options are not suitable for notifying when a task does not succeed, as they depend on the task’s schedule and retry settings, which may not reflect the actual task completion status. References:
1: Callbacks | Cloud Composer | Google Cloud
2: How to Send an Email on Task Failure in Airflow - Astronomer
3: Monitoring SLA misses | Cloud Composer | Google Cloud
4: BaseOperator | Apache Airflow Documentation
Contribute your Thoughts:
Chosen Answer:
This is a voting comment (?). You can switch to a simple comment. It is better to Upvote an existing comment if you don't have anything to add.
Submit