Anomaly Detection Content

The Anomaly Detection Add-on technology pack, and the associated Anomaly Detection spotlight, provide useful components that are required when taking advantage of the anomaly detection feature in Security Data Lake Security. The Anomaly Detection spotlight provides predefined anomaly detection rules that can be used in conjunction with different Illuminate packs.

Anomaly Detection Add-on

The anomaly detection add-on provides the stream and index set for anomaly detection rules and is required to utilize the anomaly detection rules that are provided in the Anomaly Detection spotlight. The add-on should be enabled if you are utilizing the anomaly detection feature, even if you are not using the anomaly detection rules provided by the Illuminate Anomaly Detection spotlight.

Anomaly Feature Fields

The Anomaly Detection Add-on includes pipelines that add feature fields used by the anomaly detection rules included in the Anomaly Detection spotlight. These are usually product-specific fields that indicate a specific type of event has occurred, such as a failed logon, with field names that begin with the prefix anomdet_.

Stream Configuration

This technology pack includes one stream:

"Security Data Lake Anomaly Detection Messages"

Index Set Configuration

This technology pack includes one index set definition:

"Security Data Lake Anomaly Detection Messages"

Anomaly Detection Spotlight Content Pack

Anomaly Detection Rules

The Anomaly Detection spotlight provides pre-defined anomaly detection rules designed to work with the existing Illuminate processing packs. For a detailed list of included rules, see the relevant documentation.

Anomaly Detection Rule Updates

Warning

Read the following instructions in full prior to updating the Anomaly Detection rules!

Illuminate 5.2.0 and later include a new Anomaly Detection spotlight pack that contains updated versions of the rules previously included in Illuminate versions prior to Illuminate 5.2.0.

The previous Anomaly Detection Spotlight pack has been renamed to "Core:Anomaly Detection Spotlight (Legacy)" and the updated rules are in the Illuminate pack titled "Core:Anomaly Detection Spotlight."

For a full list of the previously implemented rules and their updated counterparts, see the updated rules matrix.

Updating the Anomaly Detection Rules

If you have enabled and are running any of the anomaly detection rules from versions of Illuminate prior to Illuminate 5.2.0, complete the following sequence to enable the newly released rules included in this content pack:

Navigate to the Illuminate installer page under the Enterprise menu of your Security Data Lake web interface.
Download and install the most recent version of Illuminate if you have not already.
Enable the Illuminate pack "Core:Anomaly Detection Spotlight" but do not disable the "Core:Anomaly Detection Spotlight (Legacy)" pack.
Navigate to the Anomalies page under the Security menu of your Security Data Lake web interface. Note here that a new set of anomaly detection rules are listed.
Record which anomaly detection rules are currently running.
Then, refer to the updated rules list to determine which updated rule corresponds with each legacy rules.
One rule at a time, for each currently running legacy rule:
1. Enable the updated version of the corresponding rule.
2. Wait for the newly enabled rule to complete training, which may take at least 24 hours.
3. Disable the legacy rule(s). (Some updated single rules replace multiple legacy rules.)
Repeat this process until all of the legacy anomaly detection rules have been updated and disabled
After all of the updated legacy rules have been disabled and the corresponding updated rules enabled, return to the Illuminate installer page and disable the pack "Core:Anomaly Detection Spotlight (Legacy)."

Anomaly Detectors

This article contains a full list of all available detectors included in the anomaly detection feature.

For a complete index of all the common message fields populated in each event log message generated by anomaly detection, see the corresponding guide.

Detector Index

Detector Name	Description	Index Pattern	Requires	Anomaly-Specific Fields
Linux Auditbeat - File Deletion V2	This detection monitors for unusual file deletion activity in your environment’s Linux hosts.	`gl_linux_auditbeat_deflector`	1. Linux Auditbeat configured with the system module (not part of Beats OSS) and Sockets Dataset enabled and sending logs to the Security Data Lake server Beats input(s) 2. Illuminate 2.2.2 or greater with the Linux Auditbeat technology pack and the Anomaly Detection add-on pack enabled	Monitored field: `anomdet_file_deleted` Added field: `files_deleted_count`
Windows Security Event Log - Failed Authentication v2	This detection rule looks for anomalies in the number of failed authentication attempts by user.	`gl_windows_security_deflector`	1. A supported agent configured and sending logs to Security Data Lake: ( a.) Winlogbeat sending logs to the Security Data Lake server Beats input(s), ( b.) NXlog sending logs to the Security Data Lake Server GELF input(s) 2. Illuminate 2.2.2 or greater with the Windows technology pack and the Anomaly Detection add-on pack enabled 3. Windows systems configured to audit user authentication failures	Monitored field: `anomdet_windows_auth_failure`. Added field: `anomaly_user_authentication_failed`
Windows Security Event Log - User File Activity	This detection monitors for anomalous numbers file activity (read, write, delete) in your environment’s Windows hosts by monitoring Windows Event ID 4663.	`gl_windows_security_deflector`	1. A supported agent configured and sending logs to Security Data Lake: ( a.) Winlogbeat sending logs to the Security Data Lake server Beats input(s), ( b.) NXlog sending logs to the Security Data Lake Server GELF input(s) 2. Illuminate 2.2.2 or greater with the Windows technology pack and the Anomaly Detection add-on pack enabled 3. Windows systems configured to audit object access	Monitored field: `anomdet_windows_file_write` Added field: `file_write_count` Monitored field: `anomdet_windows_file_delete` Added field: `file_delete_count` Monitored field: `anomdet_windows_file_read` Added field: `file_read_count`
Windows Security Event Log - Object Permissions Change V2	This detection monitors for object permissions changes activity in your environment’s Windows hosts by monitoring Windows Event ID 4670.	`gl_windows_security_deflector`	1. A supported agent configured and sending logs to Security Data Lake: ( a.) Winlogbeat sending logs to the Security Data Lake server Beats input(s), ( b.) NXlog sending logs to the Security Data Lake Server GELF input(s) 2. Illuminate 2.2.2 or greater with the Windows technology pack and the Anomaly Detection add-on pack enabled 3. Windows systems configured to object permissions changes	Feature: `permissions_change_count` Field Name: `anomdet_windows_permissions_change` Aggregation Type: `sum()` Monitored field: `anomdet_windows_permissions_change` Added field: `permissions_change_count`
Linux Auditbeat - Failed Authentication V2	This detection monitors for unusual logon activity in your environment’s Linux hosts by monitoring failed logons.	`gl_linux_auditbeat_deflector`	1. Linux Auditbeat configured with the system module (not part of Beats OSS) and Sockets Dataset enabled and sending logs to the Security Data Lake server Beats input(s) 2. Illuminate 2.2.2 or greater with the Linux Auditbeat technology pack and the Anomaly Detection add-on pack enabled	Feature: `authentication_failure_count` Field Name: `anomdet_linux_auditbeat_logon_failed` Aggregation Type: `sum()` Monitored field: `anomdet_linux_auditbeat_logon_failed` Added field: `authentication_failure_count`
Linux Auditbeat - Unusual Data Transfer V2	This detection monitors for unusual network activity in your environment’s Linux hosts by monitoring the total volume of network traffic.	`gl_linux_auditbeat_deflector`	1. Linux Auditbeat configured with the system module (not part of Beats OSS) and Sockets Dataset enabled and sending logs to the Security Data Lake server Beats input(s) 2. Illuminate 2.2.2 or greater with the Linux Auditbeat technology pack and the Anomaly Detection add-on pack enabled	Monitored field: `network_bytes` Added field: `total_network_bytes`
Office 365 - Authentication Activity V2	This detection monitors for unusual authentication or authentication failure patterns by Microsoft 365 user.	`gl_o365_deflector`	1. Security Data Lake is configured to collect logs from the Microsoft 365 service 2. Illuminate 2.2.2 with the O365 processing pack and the Anomaly Detection add-on pack enabled	Monitored field: `anomdet_o365_authentication` Added field: `anomaly_user_authentication` Monitored field: `anomdet_o365_logon_failed` Added field: `anomaly_user_authentication_failed`
Okta - Failed Authentication V2	This detection monitors for unusual user authentication failure patterns in Okta events.	`gl_okta_deflector`	1. A supported agent configured and sending logs to Security Data Lake: ( a.) Winlogbeat sending logs to the Security Data Lake Server Beats input(s), ( b.) NXlog sending logs to the Security Data Lake server GELF input(s) 2. Illuminate 2.2.2 or greater with the Windows technology pack and the Anomaly Detection add-on pack enabled 3. Windows systems configured to audit file write activity	Monitored field: `anomdet_okta_logon_failed` Added field: `anomaly_user_authentication_failed`
Symantec ProxySG - Data Exfiltration V2	This detection monitors Bluecoat ProxySG logs for any unusual data transfers between hosts. Anomaly analysis is performed per user; events are aggregated by the categorical fields `user_name` and `destination_ip`.	`gl_symantec_deflector`	1. Bluecoat ProxySG sending logs to the Security Data Lake server 2. Illuminate 2.2.2 or greater with the Symantec ProxySG technology pack and the Anomaly Detection add-on pack enabled	Monitored field: `source_bytes_sent` Added field: `total_source_types_sent`
Fortigate - Unusual Data Transfer V2	This detection monitors the amount of traffic associated with Fortinet Fortigate firewalls. Anomaly analysis is performed per host; events are aggregated by the categorical field `source_ip`.	`gl_fortinet_deflector`	1. Fortinet Fortigate configured and sending logs to the Security Data Lake server 2. Illuminate 2.2.2 or greater installed with the Fortinet Fortigate technology pack and the Anomaly Detection add-on pack enabled	Monitored field: `network_bytes` Added field: `total_network_bytes`
Cisco ASA - Unusual Data Transfer V2	This detection monitors the amount of traffic reported by Cisco ASA devices. Anomaly analysis is performed per network connection; events are aggregated by the fields `source_ip`.	`gl_cisco_deflector`	1. Cisco ASA devices configured and enabled and sending logs to the Security Data Lake server 2. Illuminate 2.2.2 or greater with the Cisco ASA technology pack and the Anomaly Detection add-on pack enabled	Monitored field: `network_bytes` Added field: `network_bytes`
Palo Alto - Data Exfiltration V2	This detection monitors Palo Alto logs for any unusual data transfers between hosts. Anomaly analysis is performed per host; events are aggregated by the fields `source_ip` and `destination_ip`.	`gl_paloalto_deflector`	1. Palo Alto devices running 9.1.x or greater sending logs to the Security Data Lake server 2. Illuminate 2.2.2 with the Palo Alto processing pack and the Anomaly Detection add-on pack enabled	Monitored field: `source_bytes_sent` Added field: `total_source_bytes_sent`
Palo Alto - Failed Authentication V2	This detection monitors the amount of authentication activity for failed logon attempts associated with Palo Alto GlobalProtect clients. Anomaly analysis is performed per user; events are aggregated by the field `user_name`.	`gl_paloalto_deflector`	1. A supported agent configured and sending logs to Security Data Lake: ( a.) Winlogbeat sending logs to the Security Data Lake server Beats input(s), ( b.) NXlog sending logs to the Security Data Lake Server GELF input(s) 2. Illuminate 2.2.2 or greater with the Windows technology pack and the Anomaly Detection add-on pack enabled 3. Windows systems configured to audit file write activity	Monitored field: `anomdet_paloalto_logon_failed` Added field: `anomaly_user_authentication_failed`

Anomaly Event Message Fields

All anomaly event messages generated by anomaly detection feature have common fields and additional, detector-specific fields, depending on which detector the messages originate from. These anomaly fields are described in the anomaly detectors index depending on which detectors are enabled. The common message fields that are populated in all anomaly event messages are described in the following index.

Common Fields

Field Name	Values	Notes	Example
`anomaly_approximate_start_time`	Timestamp	The approximate time when the anomaly happened. If customers are using OpenSearch 1.2 or later, this value comes directly from OpenSearch. On OpenSearch 1.1, this will be the halfway point between `anomaly_data_start_time` and `anomaly_data_end_time`.	2022-03-09 15:23:28
`anomaly_confidence`	Numeric 0.00 - 1.00	The probability of the accuracy of the `anomaly_score`. The closer this number is to 1, the higher the accuracy.	0.5
`anomaly_data_end_time`	Timestamp	The end of the detection range of the aggregated data.	2022-03-09 15:25:28
`anomaly_data_start_time`	Timestamp	The start of the detection range of the aggregated data.	2022-03-09 15:21:20
`anomaly_detector_id`	String	The unique id of the anomaly detector. Non-human-readable and really only useful if someone is trying to use the OpenSearch APIs directly.	xBcAin8BhVrcFRn8vhUX
`anomaly_detector_name`	String	The name of the anomaly detector. These names are controlled by Security Data Lake and is a limited and well-defined set.	`windows_brute_force_logon`
`anomaly_execution_end_time`	Timestamp	The actual end time of the detector for a specific run that produces the anomaly result.	2022-03-09 15:26:28
`anomaly_execution_start_time`	Timestamp	The actual start time of the detector for a specific run that produces the anomaly result.	2022-03-09 15:23:28
`anomaly_feature_name`	String	Name of the field or fields that the anomaly detection engine was analyzing for anomalous values (may not be useful for users).	`anomdet_windows_logon_failed`
`anomaly_grade`	Numeric 0.00 - 1.00	Number between 0 and 1 that indicates how anomalous a data point is. An anomaly grade of 0 represents “not an anomaly,” and a non-zero value represents the relative severity of the anomaly. This is a normalized version of `anomaly_score`.	0.5
`anomaly_score`	Numeric	Indicates relative severity of an anomaly. The higher the score, the more anomalous a data point is.	3.875
`event_source_product`	String	Identifies anomaly msgs for identification and pipeline routing purposes. (Same value as field source)	cell
`message`	String	Standard message field in all Security Data Lake messages. Default format is [<detector_name>] Anomaly - - <anomaly_score>	`graylog_anomaly`
`source`	String	Standard source field in all Security Data Lake messages. Will always be set to `graylog_anomaly`. (Same value as `event_source_product`.)	`graylog_anomaly`
`timestamp`	Timestamp	Standard timestamp field in all Security Data Lake messages. Will be set to the time when Security Data Lake pulled the anomaly data from OpenSearch, not directly related to when the anomaly happened.	2022-03-09 15:31:52

In this section: