You could also use the Data Factory SDK to roll your own monitoring solution.
Sometimes I need to set more specific alerts where I choose only certain activities to monitor for failure. For others, I really only care if the pipeline fails. You would need to look at the status of the activities within the pipeline execution to see the failure of which you were alerted.įor many of my implementations, just setting an alert to notify me when any activity failure occurs is fine. If there is a retry of an activity and the second attempt is successful, there would be an activity failure but no pipeline failure.Ĭonversely, if we set an alert to notify us of activity failures, and we have a pipeline designed with the try-catch pattern, we might get an alert about an activity failure, but the pipeline would still show success. If we are using Azure Monitor alerts, we need to understand that setting an alert for pipeline failures doesn’t catch all activity failures. This difference between pipeline and activity status has a few implications of which we should be aware as we monitor our data factories. Activity1 fails, Activity2 is skipped, and Activit圓 succeeds. The presence of the success path alongside the failure path changes the outcome reported by the pipeline, even though the activity executions from the pipeline are the same as the previous scenario. If Activity1 fails and Activit圓 succeeds, the pipeline will fail. Now let’s say we have a pipeline with 3 activities, where Activity1 has a success path to Activity2 and a failure path to Activit圓. The failure dependency means this pipeline reports success. This scenario is treated as a try-catch block by Data Factory. If Activity1 fails and Activity2 succeeds, the pipeline will succeed. Because Activity1 failed, Activity2 is not executed and the pipeline fails.īut if we have a pipeline with two activities where Activity2 has a failure dependency on Activity1, the pipeline will not fail just because Activity1 failed. In this scenario, if Activity1 fails, the pipeline will fail. If we have a pipeline containing Activity1 and Activity2, and Activity2 has a success dependency on Activity1, it will only execute if Activity1 is successful. A dependency has a condition of one of the following: Succeeded, Failed, Skipped, or Completed. Dependency with a Failure ConditionĪctivities are linked together via dependencies. If nothing else in the pipeline failed, the pipeline would report success. Since it was set to have 1 retry, it executed again and succeeded. Output of a Data Factory activity that was executed and initially failed. I often set retries to a non-zero number in copy activities, lookups, and data flows in case there are transient issues that would cause a failure that might not be present if we waited 30 seconds and tried the activity again. Data Factory Activity General settings showing the Retry Property If we execute a pipeline containing one activity with the default Retry setting, the failure of the activity would cause the pipeline to fail. This is the number of times Data Factory can try to execute the activity again if the initial execution fails. In the General settings of any activity is a property called Retry. The failed activity has a failure path or a completion path to a subsequent activity and no success path.
The maximum number of retry attempts is greater than 0, and the initial activity execution fails but the second attempt succeeds.There are two main scenarios where an activity would report failure, but the pipeline would report success: In many instances, when an activity fails during a pipeline run, the pipeline run will report failure as well.
Activities can be categorized as data movement, data transformation, or control activities. Activities in a pipeline define actions to perform on your data. It is the unit of execution – you schedule and execute a pipeline. In Azure Data Factory, a pipeline is a logical grouping of activities that together perform a task. Question: When an activity in a Data Factory pipeline fails, does the entire pipeline fail?
0 kommentar(er)
