Event-driven architecture (EDA) is a common data integration pattern that involves production, detection, consumption, and reaction to events. Data integration scenarios often require Azure Data Factory customers to trigger ETL or ELT pipelines when certain events occur.
Azure Data Factory native integration with Azure Event Grid now covers custom topics: the application and third-party topics. Custom topic, created by the event publisher, provides an endpoint where source sends events. Azure Data Factory subscribes to the topic and triggers a pipeline in response to incoming events.
Additionally, Azure Data Factory now supports Advanced Filtering that gives users fine grain control over which events to respond to based on the values in the event payload. For instance, you can express logic such as, a ETL pipeline should run if and only if data.date field in list [1, 2, 3, 4, 5], data.month field not in list [11, 12], data.tag field contains any of ['Fiscal Year 2021', 'FiscalYear2021', 'FY2021'].
You can also implement Metadata Driven + Event Driven architecture, by passing event payload values to pipeline parameters, using prefix @triggerBody().event.data.<key_name>, and consumes these values in your pipeline definition. For information, please refer to Retrieve and Reference Trigger Metadata in Pipeline.
To create a Custom Event Trigger, event producer needs to define a custom topic in Event Grid (see Azure Event Grid Custom Event Topic tutorials). And by creating a Custom Event Trigger, Azure Data Factory will create a subscription against the aforementioned topic.
Click + New Trigger, specify the Event Grid Topic Name, Subject Begins With, Subject Ends With, and Event Types
Once published and activated, you can double check successful creation by browsing to the aforementioned event topic in Azure Portal and see a new subscription created by Azure Data Factory.