How Unomaly detects anomalies
Unomaly receives a stream of log events into a pipeline that analyzes the structure and frequency patterns to quickly determine whether incoming log events have been seen before, giving you a curated overview of new and changed log events in your IT environment. The new and changed log events are classified as structural, parameter or frequency anomalies.
Learning the event profile
Data sources (systems) generate large volumes of log data. Log events are individual lines in log data. Event profiles are records of each log event's structure with information about the log event's origins, counts, arrival times, and frequency patterns.
As log events stream in, Unomaly identifies their structure in a process we refer to as tokenization. Tokenization breaks down the message content into a sequence of words and punctuation and computes a hash that is unique for each log event's structure. This structure hash is used to determine if incoming log events match previously seen profiles on that system.
Each new structure hash is compared against the existing structures in the profile database for each system to find if a match exists. Unomaly creates new profiles or updates existing profiles depending on the result of this matching algorithm.
- If the new hash does not match an existing hash, Unomaly creates a new profile in the database and flags this event as a structural anomaly. (Never before seen)
- If a similar hash exists and it's not an exact match, Unomaly creates a new profile in the database and flags this event as one of the relevant anomaly types or as a known. (New in system)
- If an exact match exists, Unomaly analyzes the parameters in the sequence and updates the existing profile's information about the parameter and frequency patterns of the event. (Parameter or frequency anomalies)
During the training period for each system, every new log event is detected as a structural anomaly. Training ends when Unomaly does not detect "Never before seen" events from a system in the last 6 hours. However, learning doesn't end with the training period because Unomaly continues to update the event profiles as it receives more data.
The majority of these log events have the same or similar structure. Another part of the tokenization process identifies the dynamic parameters of the structure and merges similar profiles into a single aggregated profile with dynamic parameters. The result of this merge may reduce the amount of data to analyze by at least 99%. The diagram above gives an example of the average data reduction and breakdown of anomalies as seen in our Licensing server over 30 days.
Recognizing anomalies
The profiles database makes it possible to quickly determine when incoming log events are new or different from previously seen events. Log events that don't match existing structures are immediately flagged as a structural anomaly. Meanwhile, log events that match are analyzed further to determine whether they are parameter anomalies or frequency anomalies.
Structural anomalies
Structural anomalies are new or significantly changed log events that do not fit into the established pattern of log event structures that Unomaly has learned. These may be events with profiles that have not been detected in the system or in the entire IT infrastructure.
| Anomaly Type | Description |
|---|---|
| Never before seen | Events that are new in the entire IT environment that Unomaly is monitoring. |
| New in system | Events that are new in a system but have occurred in other systems. |
| Parameter change | Events that match an existing profile but has a parameter value that has not been seen before. Unomaly tracks up to 100 values for each dynamic parameter. |
| Anomaly Type | Description |
|---|---|
| Parameter change | Events that match an existing profile but has a parameter value that has not been seen before. Unomaly tracks up to 100 values for each dynamic parameter. |
Frequency anomalies
Each profile includes information about the log event's arrival times and intervals between occurrences of matching profiles. Unomaly tracks the frequency of a log event's occurrence and a standard deviation between occurrences to determine when an increase or decrease in the events rate occurs.
| Anomaly Type | Description |
|---|---|
| System away | Anomalies indicating that Unomaly has not received events from the system for a certain amount of time. |
| Frequency spike | Anomalies where an event is produced at a significantly greater rate than previously seen, within a time window of 59*85 seconds. Spikes are detected using the z-score algorithm. The severity of the spike (medium or large) is determined by the amplitude of the spike, relative to the surrounding events. |
| Event stop | Anomalies where a periodic log event (that is an event that was seen regularly) is not detected when it is expected. For each event, an expected timestamp is calculated for the next instance of that profile based on previously seen intervals of occurrence. When the event doesn't happen at the expected time (within a margin of error), Unomaly detects it as a stopped event. |
Influencing the algorithm
Unomaly creates profiles and determines the types of anomalies it detects algorithmically. If needed, you may tweak a few settings to improve or correct the results.
Transforms
Tokenization for certain log events may produce too many tokens in the structure. When this happens, you can define transforms, which are rules for merging selected tokens in a log line into one token. These merges can improve how Unomaly recognizes parameters as the algorithm learns your systems. In some cases, it may also reduce the number of structural anomalies Unomaly falsely detects.
Sensitivity
The Sensitivity setting dictates how aggressively Unomaly merges similar structures into a profile during tokenization. A lower sensitivity means that more profiles are merged and fewer parameter anomalies are detected. A higher sensitivity will merge fewer profiles, creating more unique profiles, which may lead to lower performance.
Knowns
Create a Known to track an event after Unomaly has learned it. When an incoming log event matches a defined known, Unomaly flags the event based on the classification you assign to it. For example, you can define a known to highlight a critical event each time it occurs or to ignore an event.
Use knowns to add relevant contextual information, such as descriptions and tags to explain what the event means or remediation steps if the event can lead to an issue. You can also use knowns to specify how you want Unomaly to handle the event, such as whether or not to add it to a situation and assign it a score.