10 Monitoring and Alerts using Fabric Events
The Fabric Accelerator leverages Fabric events [1][2][3] to monitor and alert significant events in real-time, enhancing the observability of your data platform. This observability encompasses all activities within OneLake and Fabric workspaces, as well as the execution of data pipelines and Spark notebooks. Currently, the following monitoring and alerts are available:
- Frequently used Fabric workspaces, item types, items, users, and user actions.
- Frequently run data pipelines and Spark notebooks, including elapsed duration, execution status, trigger types, job types, and schedules.
- Frequently used OneLake actions by users.
- Alerts for job execution anomalies.
- Alerts for jobs showing regression trends compared to the last 60 days.
- Alerts for OneLake usage anomalies.
- Alerts for new users, among others.
- The Eventstream es_fabricEvents connects to events generated by items in your data platform.
- The following data sources are connected to the event stream. Data is sent to event stream when a user or system generated event occurs.
| Connector | Data source |
|---|---|
| Fabric Workspace Item Events | The default workspace for this accelerator |
| Fabric Job Events | Master ETL ASQL data pipeline Optimize DeltaLake Tables Spark notebook |
| Fabric OneLake Events | lh_bronze lakehouse lh_silver lakehouse dw_gold datawarehouse |
-
The Eventstream then filters data based on the schema of data source type (workspace/job/OneLake) and lands them as-is in kdb_fabricEvents KQL database in the Eventhouse eh_fabricAccelerator. This is the bronze layer. Each data source type has a dedicated Kusto table named workspaceEvents, jobEvents, and storageEvents respectively.
-
Data in bronze Kusto tables embed key information in JSON columns and may contain duplicates. Update Policies are used to extract data from JSON columns and de-duplicate it in real-time from bronze tables. Each Events table has a corresponding Update Policy that calls a Kusto function expand...() to create Expanded... Kusto tables as the silver layer. At the end of this stage, workspaceEventsExpanded, jobEventsExpanded, and storageEventsExpanded are created.
-
Finally, Kusto Materialized Views are used to obtain daily snapshots of events data from the silver tables. These views — dailyAggWorkspaceEvents, dailyAggJobEvents and dailyAggStorageEvents form the gold layer for reporting and alerts.
In summary, the mapping from source to destination looks like this
| Connector | Data source | Bronze (Kusto table) | Update Policy Function | Silver(Kusto table) | Gold (kusto materialized view) |
|---|---|---|---|---|---|
| Fabric Workspace Item Events | The default workspace for this accelerator | workspaceEvent | expandWorkspaceEvents() | workspaceEventsExpanded | dailyAggWorkspaceEvents |
| Fabric Job Events | Master ETL ASQL data pipeline Optimize DeltaLake Tables Spark notebook |
jobEvent | expandJobEvents() | jobEventsExpanded | dailyAggJobEvents |
| Fabric OneLake Events | lh_bronze lakehouse lh_silver lakehouse dw_gold datawarehouse |
storageEvent | expandStorageEvents() | storageEventsExpanded | dailyAggStorageEvents |
- The Real-Time Dashboard fabricEventsDashboard is created from gold layer materialized views, with each tile representing a KQL query.
- Data Activator alerts is created from Real-Time Dashboard tiles.
Frequently used Fabric workspaces, item types, items, users, and user actions.
Frequently run data pipelines and Spark notebooks, including elapsed duration, execution status, trigger types, job types, and schedules.
Frequently used OneLake actions by users.
Data pipeline and Spark jobs execution anomalies
Unusual usage patterns in OneLake usage
- Alerts for jobs showing regression trends compared to the last 60 days.
- Alerts for job execution anomalies
- Alerts for OneLake usage anomalies
- Alerts for new users
If you don't see data in the alert tiles, it means there are no issues to report, which is a positive outcome. Use the tiles in the alert page of Real-Time dashboard to configure your alerts. The setup of Data Activator alerts is not part of this accelerator because the selection of alert channels and notification groups may vary based on your specific needs.
