Anomaly Detection
Anomaly detection plays a vital role in managing modern, large-scale distributed systems. Within these systems, analyzing data aggregations can be a significant pain point, and sometimes potential issues and threats slip into these systems undetected.
Anomaly Detection is a Security tool that utilizes your Security Data Lake environment powered by Illuminate. (See Illuminate documentation on the Anomaly Detection add-on technology pack for more information.) This tool's primary purpose is to help you detect outliers in a dataset by running Artificial Intelligence / Machine Learning (AI/ML) behavioral analysis, allowing you to receive alerts whenever something deviates from its usual behavior or operates outside normal levels based on an understanding of your unique log data. As a result, you and your team become empowered to navigate cyber threats proactively, identify unusual activities, and take steps toward mitigating anomalies within your IT environment.
The process begins when Security Data Lake Illuminate receives your log data, then normalizes and enriches it. Security Data Lake then feeds the enriched data into the Anomaly Detection tool, which breaks the data up into time slices and looks for data points outside of the expected range based on your historical data. When anomalous data points are detected, these messages are logged into a special anomaly index in your Security Data Lake instance. You can then create alerts regarding these anomalies based on your configuration settings. Additionally, these anomalies are represented on Security dashboards with various customizable widgets, offering interactive and actionable analytics.
This user guide helps you prepare for and set up your Anomaly Detection tool.
Enable Anomaly Detectors
Using Anomaly Detection requires no prior experience in machine learning as the user interface is intuitive and provides numerous descriptors. Configuring the tool begins by switching the toggles to enable any of the available detectors offered.
After you have purchased the Security Data Lake Security license and installed or updated your Illuminate product, you can then install the Anomaly Detection technology packs.
Navigate to Enterprise > Illuminate then search for "Anomaly Detection."
Select and enable both the Anomaly Detection Add-on and the Anomaly Detection Spotlight content packs.
Now, select Anomalies > Anomaly Detectors to see available anomaly detectors. Depending on your organization's needs and preferences, enable the required detectors by clicking on the toggle so that running state is shown.
Note
When multiple detectors are turned on, initialization is queued and performed sequentially.
A full list of all the available detectors and their functions may be reviewed in the anomaly detector user guide.
Note that starting multiple detectors simultaneously can be resource intensive on your OpenSearch nodes. The number of detectors allowed to be initializing at a time can be configured with the Security Data Lake config value opensearch_anomaly_max_concurrent_initializations=X, with the default value being 1. Hence, we recommend that you change the concurrent initializations value with caution and be sure that your clusters are adequately resourced to handle it gracefully.
Interpret the Data
When the anomaly tool detects anomalies in the log data, it pulls that data into the Anomaly Detection processor. The data is then deconstructed into human-readable and decluttered messages known as Security Data Lake anomaly event messages, which will flow through the standard event message processing flow. These anomaly event messages summarily give you precise details of detected anomalous events.
Note
For a complete list of these event message types and their definitions, review the anomaly event message index
All anomaly event messages have common fields and additional fields depending on which detector the messages originate from. These additional fields tell you which entity (user, host, etc.) demonstrated anomalous behavior. Different detectors, like anomaly_detector_name, will add different fields.
Use Case
Let's look at an example.
Below is an example of an anomaly event message with both standard fields and anomaly fields populated. Think of an anomaly event message simply as how the Security Data Lake anomaly detection tool represents data ingested behind the scenes.
In this example, an anomaly was detected by the Windows Event Log File Permissions Change Spike detector with an anomaly confidence value of 0.99, meaning that the detector is quite certain that the detected event is anomalous. The full event message provides further details:
anomaly score: 3.99 (Indicates relative severity of an anomaly. The higher the score, the more anomalous a data point is. With an anomaly score of 3.99, this represents a relatively high anomalous data point) anomaly grade: 1 (This is a severity level scale, ranging from 0 to 1. 0 is ‘not anomalous’ and a non-zero value represents the relative severity of the anomaly. With a value of 1 in this case, this is a high severity anomaly. anomaly_file_perm_change_count: 3 (This is an additional field value which is specific to the Windows Event Log File Permissions Change Spike detector, and this value represents the number of times a file was permanently changed. user_name: (This is an additional field value that is specific to the Windows Event Log File Permissions Change Spike detector and this value provides information on the username where the detected anomaly originated from.
Here, the tool provides filtered data in the anomaly event message, further empowering you to locate the needle in the haystack within your IT infrastructure.
Create Anomaly Events and Alerts
Now that you know what Security Data Lake Anomaly Detection is and how to enable it, the next step is to utilize the anomaly event messages to create events and alerts.
Using the example of the Windows Event Log File Permissions Change Spike above, let's assume you want to be alerted any time there are changes to file permissions in a given environment’s Windows hosts. You could begin by creating the search query:
anomaly_detector_name:wineventlog_file_permissions_change AND anomaly_file_perm_change_count:>0 AND anomaly_confidence:>.01
This query would then search log data within the set parameters. Now, you can create an alert that triggers a specific alert, like an email or Slack message, to a specific group of users. For full instructions on this process, see the events and alerts user guide.
Generate Events from Anomaly Detectors
By default when you enable an anomaly detector, Security Data Lake will generate an event definition. You have the option to disable event definitions for enabled anomaly detectors by toggling the Enable event definition capsule button found in the Anomaly Detectors tab on the Anomalies page.
Use Case
In this example, the anomaly detector for the Fortigate - Unusual Data Transfer is enabled, while its corresponding event definition is disabled.
When a system-generated event definition is created by enabling an anomaly detector, this appears on the Events Definitions page. You can then add alerts to event definitions for when an anomalous event definition is triggered:
Click the More button for a selected event definition.
Select Edit from the drop-down options.
Navigate to the Notifications section in the resulting screen and click the Add Notification button.
Select the required alert type and configure the applicable parameters.
Anomaly Detectors
Note
The following article has been moved to the official Illuminate documentation. See below for details.
For a full list of all available detectors included in the Security Data Lake Anomaly Detection tool, see the related Illuminate documentation.
For a complete index of all the common message fields populated in each event log message generated by Anomaly Detection, see the corresponding guide.
Anomaly Event Message Fields
Note
The following article has been moved to the official Illuminate documentation. See below for details.
All anomaly event messages generated by Security Data Lake's Anomaly Detection tool have common fields and additional, detector-specific fields, depending on which detector the messages originate from. These anomaly fields are described in the Anomaly Detectors index depending on which detectors are enabled. For a full list of all the common message fields that are populated in all anomaly event messages, see the event message field index in the Illuminate documentation.