Sudden Change refers to a type of condition available in an alerting policy which can be applied to a time-series metric. It’s used to analyze trends based on historical data to predict a single future value for the time-series data, and then alerts on large increases or decreases of the actual value compared to the predicted value.
The only configurable parameter for this type of condition is the precentage increase or decrease of the actual value compared to the predicted value.
Configuring The Sudden Change Condition
Complete the following steps to configure a new policy using the Sudden Change condition:
- Navigate to Alerts > + Add New Policy.
- Under the Conditions tab, select Add Condition > Add Metric Condition.
- In the Deviation(s) section, check Sudden Change.
- Select Increase/Decrease by More Than.
- Enter a percentage value.
Active Policy Example
The following graph depicts a policy with a sudden change deviation condition on a certain metric. In this example, an alert triggers due to a sudden change event (in red) on the 25th data point.
- Black Dots: Each black point represents 1 of the 24 previous 5-minute values (PT5M) of a metric
- Black Line: A best-fit regression line through the black points
- Blue Line: The projected trend of the 25th data point, based off of the historical data shown
- Blue Dot: The predicted value of the 25th data point
- Red Line: Breadth of sudden change, deviating from the expected projection in blue
- Red Dot: 25th data point’s actual observed value
How a Percent Drop is Measured
A percent drop, or step change, is computed as:
| (projected value) - (observed value) | / | (projected value) |
The sudden change algorithm returns this value for use by a condition in a policy. If the value exceeds the threshold in the policy condition, then the condition is true. If all the other conditions in the policy (if any) are also true, then an event is emitted.
Before reporting back the above value as a potential change, the algorithm performs several checks.
One of these checks is designed to determine if the regression model is a good enough fit for us to have any confidence in it’s projected value. Another check is used to add confidence that the observed value is sufficiently different from the projected value to be truly “anomalous”. Additional checks deal with detecting the trend in values leading up to the observed value. For example, if the trend was already negative and the actual observed value is just continuation of that trend, then no drop will be reported. The algorithm also requires data points to be consistently available and not be sparse.
In the above example, it is possible that some of these checks may have failed. In that case, the algorithm reports back that there was no drop.
When configuring a condition for sudden change deviation, we recommend setting a duration of no longer than 5 minutes. This is due to the nature of the event you are trying to capture: a single, sudden change in activity. Expecting a secondary sudden change in a longer duration of time may cause your policy to never activate, meaning you could miss otherwise genuine alerts.