Monitor Data Anomalies

Alation Cloud Service Applies to Alation Cloud Service instances of Alation

Anomaly detection in Alation Data Quality uses machine learning to establish baseline trends and identify unusual deviations in your data.

Anomaly detection metrics enable automated identification of unusual patterns and deviations in your data. By establishing baseline trends through machine learning, Alation Data Quality can alert you to unexpected changes that may indicate data quality issues, system problems, or significant business events.

The anomaly detection follows a lifecycle:

• Assign Anomaly Metric: When you create a monitor, you can assign anomaly metrics to a table or column for analyzing data patterns.

• Warmup Period: When an anomaly metric is assigned, it enters a 30-day warmup period. During this time, the model analyzes historical patterns to learn your data’s normal behavior, seasonality, and expected variation. No alerts are generated during this phase.

• Baseline Establishment: After the warmup, the model has an established baseline. It continues to run at its scheduled interval, comparing the latest metric value against the learned baseline to detect true anomalies.

• Feedback Loop: The model’s accuracy improves over time through user feedback. When you review anomalies, you can use the actions Confirm Anomaly or Mark as Expected. This feedback teaches the model to recognize legitimate business events (like planned data loads) and reduce false positives.

Anomaly Metrics

Table-Level Anomaly Metrics

Metric Type	Description
Row Count	Detects unusual changes in the number of rows in a dataset. This metric helps identify unexpected data volume fluctuations that may indicate missing data loads, duplicate ingestion, or data deletion issues.
Freshness	Detects unusual delays in data updates based on timestamp columns. Use this metric to monitor whether data is being refreshed according to expected schedules and identify pipeline delays or failures.
Schema Drift	Detects unexpected changes to table structure, including added, removed, or modified columns. Schema drift detection helps prevent downstream application failures and ensures data contracts remain intact. Schema drift detects the following changes: If a new column is added (Column Addition) If an existing column is deleted (Column Removal) If the column data type got changed.

Column-Level Anomaly Metrics

Metric Type	Description
Duplicate Count	Detects unusual changes in the number of duplicate values in a column. This metric identifies potential data quality degradation when columns expected to contain unique values show increasing duplication.
Missing Count	Detects unusual changes in the number of null or missing values in a column. Monitor completeness trends to catch upstream data quality issues before they impact downstream consumers.
Maximum	Detects unusual changes in the highest value within a numeric column. Useful for identifying outliers, data entry errors, or legitimate business events that push values beyond typical ranges.
Minimum	Detects unusual changes in the lowest value within a numeric column. Helps catch data errors, such as negative values in columns that should only contain positive numbers, or unexpected zero values.
Average	Detects unusual changes in the average value of a numeric column. This metric reveals shifts in central tendency that may indicate changing business conditions or data quality issues.
Standard Deviation	Detects unusual changes in data spread or variability within a numeric column. Increasing or decreasing variability can signal changes in data collection processes or underlying business dynamics.

Add Anomaly Metrics

When you create a monitor and apply checks, you can add anomaly metrics. This is an optional feature during monitor creation. For more information, see Add a Monitor.

Manage Anomalies

You can add or remove more tables and anomaly metrics for a configured monitor. This is done when you already created a monitor and want to manage the previously added anomaly metrics. For more information, see Manage a Monitor.

1. Navigate to the Alation Data Quality application and click the Monitors tab.

2. Click on a monitor name to view its details.

3. Navigate to Manage Checks and Metrics > Anomaly Metrics.

4. For existing tables, add one or more table-level anomaly metrics.

5. For existing columns, add one or more column-level anomaly metrics.

6. To remove a previously added anomaly, expand the row and click Delete.

Note

You cannot edit a previously added anomaly. You can only add or delete it.

Analyze Anomalies

When you run the monitor and anomalies are detected, after a warmup period of 30 days, anomaly metrics start to appear in the Monitors > Monitor Name > Anomaly Metric Results tab.

The detected anomalies are represented in the following ways:

• Time-Series Graphs: Most metrics are displayed as time-series visualizations showing the metric value over time with anomaly indicators highlighting unusual data points.

• Heat Map: Schema drift anomalies are displayed using a heat map visualization that highlights structural changes in table schemas over time.

To analyze anomalies:

1. Navigate to the Alation Data Quality application and click the Monitors tab.

2. Click on a monitor name to view its details.

3. Open Anomaly Metric Results that provides an overview of the detected anomalies at the table and column-level.

4. View the time-series graph or a heat map for detected anomalies.

   • For table-level metrics, click on metrics to view details.

   • For column-level metrics, click View Details.

5. View the anomaly representations marked in red and hover over the detected anomalies or healthy ones for further details and actions.

6. Provide feedback to the large language model (LLM) to improve anomaly detection accuracy over time in the following ways:

   

   When you hover over a data point, you see an anomaly card pop up where you can access feedback actions:

   • If the detected anomaly is as expected, click Mark as Expected.

   • To confirm the detected anomaly, click Confirm Anomaly.

   • To link the detected anomaly to an incident management system, click Create New Incident or Link to Existing Incident (if you already have one).

   • To mark a healthy pattern as anomaly, click Mark as Anomaly.