Event Definitions
An event refers to specific circumstances in your environment that would cause a change in normal behavior. For example an event could be changes made to firewall policies or a failed login attempt by a blacklisted IP address. Security Data Lake helps you manage events by allowing you to define the specific parameters of an event and alerts you when your log data matches these parameters. In this article, we review how to create and manage event definitions, including how to attach an alert to the event.
There are two main ways to create a new event definition:
Create an Event Definition Through the Wizard
To create an event definition through the wizard in the Security Data Lake interface:
Navigate to Alerts.
Click the Event Definitions tab.
Select Create event definition in the upper right corner.
The first screen in the wizard presents fields where you set the event title, description, and priority.
Define a Priority
The priority of an event is a user-specified classification. Events can be prioritized from 1 to 3 (1 = low, 2 = normal, and 3 = high) according to their importance. This assessment can help you triage events, which is a necessary practice in security investigations. The priority of an event is displayed as a thermometer icon in the overview and is written into the alert.
An example of a low prioritized event (priority level 1) might be one failed login in 10 minutes. A normal prioritization (priority level 2) might be 2 or 3 failed logins in 10 minutes. More than 15 failed logins in 10 minutes could be considered high priority (priority level 3) because this occurrence could mean that a person or bot is trying to break into a system.
Set Event Type
Additionally in the wizard you can define the type of event. Select the event type from the Condition Type drop-down menu:
Filter & Aggregation: This type is an event based on search and filtering of log data.
Event Correlation: This type is an event based on the occurrence of multiple other defined events in a sequence.
After you make your selection, additional fields appear on this page to define the event.
The remainder of this article focuses on Filter & Aggregation events. For information about event correlation, see Correlation Engine.
Note
Sigma rules events are displayed on the Event Definitions page, but you define sigma events on the Sigma Rules page in Security Data Lake.
Define Event Criteria
By combining a search query and an aggregation, you can specifically describe the criteria that would constitute a Filter & Aggregation event. In the Filter section, set your search query and other details:
Define a search query that your messages should match. The query should use the same syntax as any search from the Search page.
Apply a search filter to modify the query results. See below for information about search filters. (optional)
Select a stream in which the message can be found.
Define the window of time that the filter searches backward to match messages.
The search is executed at the given interval. If the filter matches, an event is created.
An example of using a filter to define an event could be filtering against a search for failed user logins by further refining it to include only certain users. You might also define the window of time to be the last 24 hours or the last 3 days.
If the defined filter matches messages currently on the Security Data Lake server, the messages are displayed in the Filter Preview panel on the right.
Create an Aggregation
An aggregation is the combination of two or more entities. The new entity processes specific and meaningful results. Aggregations can run a mathematical operation on either a numeric field value or the raw count of messages generated that match the filter. Aggregations can group matches by a selected field before making the comparison.
For instance, if the field username is defined, then it is possible to alert on five successive failed logins by a particular username. In this example, you would create an event definition that triggers if there are five or more consecutive failed login messages that pertain to the targeted user.
Create a Custom Field
You can also create a custom field as part of an event definition in the Fields menu. These fields allow an event generated from this definition to populate data from the original log into the Security Data Lake events index. This prevents you from having to run subsequent searches to get vital information. These fields can also be used to limit the amount of data sent to an alert target, and you can run aggregations that include custom fields.
These fields can be accessed within an alert and can be used as part of the Enterprise event correlation feature.
Note
The event is recorded to the All Events stream and contains the custom field as well as the result of the aggregation that triggered the event.
Attach an Alert
In the Notifications menu, you can attach an alert to your event definition. See the Alerts article for information on how to set up an alert and the alert types available.
Create an Event Definition Directly From Search Results
You can select any value in your search results to create an event definition. This event definition generates tailored alerts that include only the specific part of the query that you want to be alerted on. To do so:
Go to your search results.
Click any value in an aggregation widget, log view, or message widget.
Select Create event definition from the drop down menu.
Pick one of the Strategy by options in the dialog box that appears. You can select any of these options:
Exactly this value: Displays parameters related to your current search. You may add or remove any of these.
Any in widget: Displays parameters related to the selected value.
Custom: Allows you to include any part of the search query.
Click Show strategy details to select or deselect any parameters you would like to add to the event definition. The parameters you select here populate in your event definition under Filter & Aggregation.
Note
In addition to the three options displayed above, you might be presented with other options depending on the value you select. For example, if you select an aggregation widget metric value, you are presented with additional Any in row and Any in column options.
Click Continue Configuration. You are redirected to the Event Definitions page. Start by giving your event definition a unique title and filling in other details in Event Details. The selections you made in steps 4 and 5 are populated in Filter & Aggregation. You can add search filters, custom fields, and alerts in this menu.
After reviewing the summary of your new event definition, click Create event definition. A new event definition is created, and you will receive alerts for the given condition.
Manage Defined Events
All defined events are available on the Alerts & Events page. You can find details about each entity, such as the priority, status, and scheduling, on the Event Definitions page. Click the information icon in the Scheduling column to view information about status, last execution, next execution, next time range, and queued notifications.
The Event Definitions page includes the Bulk actions menu, which lets you delete, enable, and disable multiple entities simultaneously. Under More , you can edit, duplicate, enable or disable, and delete individual definitions.
Replay a Search
You can replay the specific search that first triggered an event. Select an entity on the Alerts & Events page to access the replay search option, which can be found under Actions. You may review the search results and messages to gather important details in investigating the event. Note that this page can also be bookmarked for future reference during investigations.
Schedule an Event
Cron scheduling automates the process of running an event, making it possible to run events periodically at a fixed time, date, or other interval.
With cron scheduling, you can define a more precise window of time that makes sense to monitor. For example, you can schedule your event to alert you on logins that happen at an unusual time, like outside work hours. Or you could choose to run a costly event only during nighttime hours, when CPU usage is low.
Cron scheduling is determined by cron expressions. A cron expression is a string of fields separated by white space. Graylog uses Quartz cron syntax, for example:
<second> <minute> <hour> <day of month> <month> <day of week> <year>
Note
Quartz cron expressions may have six to seven fields. The seventh field <year> is optional.
For example, let's say you want to run an event every Monday at 8:05 AM. You enter 8 in the hour field, 5 in the minute field, and MON as the day of the week. So the expression is:
0 5 8 ? * MON *
The <day of month> field in this example is not relevant and is therefore represented with a question mark (?). Other fields that are marked with an asterisks can take any value.
There are several special characters used in cron expressions. Here are some of the most common ones:
*: An asterisks can be used for fields that can take any value. For example, if you want to run an event that starts at 12 AM, enter:0 0 0 * * * *. The first three zeros mark thesecond,minute, andhourthat the event will run. The asterisks in the remaining fields mean that these fields accept all values.?: Denotes that the field value is irrelevant. Mainly used for days of the week. If you want to run an event on any day of the week, you may use an expression such as:* 5 8 * * ? *./: A forward slash is used to specify ranges. For example,M/Fdenotes Monday through Friday
You can look into resources such as the Cron Trigger Tutorial for more information on special characters and creating cron expressions.
Cron Expression Syntax
The following are some common examples of cron expressions:
Cron Expression | Description |
|---|---|
| At 2:00 AM. |
| Every hour. |
| The first 7 days of every month at 9 AM. |
| Every 10 minutes, starting at 5 minutes past the hour. |
| Every 5 minutes, starting at 1 minute past the hour, on day 7 of the month. |
| Every first day of January, starting at 7 AM. |
Add Cron Scheduling to an Event Definition
To add cron scheduling to an event definition:
Check the Use cron scheduling box under Filter & Aggregation.
Enter the desired cron expression in the Cron expression box.
Select a relevant timezone.
Your cron expression will be added to the event definition and can be viewed under Event Details.
See the image below for an example of a cron expression that will schedule an event to run15 minutes past the hour, every thirty minutes:
Note
As seen in image, a description of the expression you enter will appear below the text box. If the input is incorrect, you will be given a warning.
Filter with Dynamic Lists
Dynamic lists allow you to define a filter where some of the search arguments are parameterized. Every time an event definition is being checked, these parameters are replaced with the result of a dynamic list.
Dynamic lists (such as a lookup tables) can be used to create event definitions:
Go to the Alerts page.
Navigate to Event Definitions and click Create Event Definition.
Enter the required information for Event Details.
Select Filter & Aggregation as the Condition Type.
Enter your search query using the same syntax as used on the Search page.
Click the undeclared parameter that shows up in the Query Parameters box. Enter the required information in the menu that appears and selectSave.
Check the Filter Preview section to validate the outcome before you proceed.
Click Create event definition on the Summary tab. Now you will receive alerts based on this event definition.
Dynamic Lists Use Case
In this scenario, the user wants to monitor a list of former employees for safety reasons. They want to receive an alert if anyone on the list tries to log in to the company system. Generally, this type of query is usually difficult to maintain because of the immense number of values to compare. Using a lookup table allows them to compare a log value to any value within the list. The lookup table will be updated to include all former employees, including ones that have recently left the company.
The parameter $former_employee$ is backed by a lookup table that returns a current list of former employees. After creating the event definition, the user will be alerted on any login attempts from anybody on the list.
Event Definitions Use Case
After reviewing how to create an event definition, this article will provide you with an in-depth example on how you might set up a new event definition from the wizard and attach an alert to this event.
Scenario
The following use case scenario describes an instance where Security Data Lake receives log messages for a service you need to authenticate in order to log in (e.g. SSH, web app). Let's say you want to receive an email from Security Data Lake for possible brute force attacks being run on the authentication of the service. If one user fails to log in to the system 10 times in one minute, you want to get an email from Security Data Lake.
Enter Event Details
After clicking on Create Event Definition, you are presented with the event definition wizard. Here, enter the title and description of the event definition and define a priority.
Since you got a message about a brute force attack, which should be treated urgently, set the priority to High and proceed to the next step.
Specify the Condition
Choose Filter & Aggregation for the Condition Type because you only want to see messages that indicate a failed authentication. Enter specific information about the streams and messages you would like to include under Filter.
Define a Filter
First, select the stream in which your log files are routed. If no stream was created for the web application, we highly recommend visiting Streams for more information. The query result will be limited to web application logs and no other logs will influence the filter process.
Now, you need to filter the incoming messages, so you can later count the messages that match the filter. A log message that indicates a failed authentication, will look like this:
Login failed for user admin from ip 240.210.133.39
Add Login failed for user to the Search Query field. Now, a preview table appears on the right side of the screen that shows the messages matching the query. The preview uses Search within the last as the time range. If you provoke a log entry now, you should see this after updating the query.
Set Search within the last to 10 seconds and Execute search every to 10 seconds. The event engine will execute the query every 10 seconds for a time range of 10 seconds.
Note
If Search within the last is greater than Execute search every (which would be called a hopping window), the event engine will generate multiple events for the same log line. For example, if you set Search within the last to one minute and Execute search every to 10 seconds, the engine will find the same log entry 6 times because the log entry is visible for one minute and we are searching every 10 seconds. If Search within the last is the same as the Execute search every period, you will create something known as a tumbling window, which is recommended for most situations.
Create an Aggregation
Since you want to aggregate on your events (to see if you have more than 10 messages in 10 seconds), select Aggregation of results reaches a threshold under Create Events for Definition if....
To be able to find how many failed login attempts there were per user, you need to add a pipeline to incoming messages. The pipeline will extract the user name and store it in the User field. Every message with a failed login has a User field. Enter that field in the selection Group by Field(s).
Finally, add an aggregation rule: If count() is >=10.
Review Previous Steps
To summarize what you have done so far:
Added a stream to minimize the messages you need to filter on.
Inserted a query to filter the logs down to only failed logins.
Grouped your logs so the aggregation is applied per user.
Added a rule: only raise an alert if the count is more than, or equals 10.
Now, click Next to reach the Fields tab.
Add a Custom Field
Add a custom field to the event by clicking on Add custom field. Since you have an aggregated event definition, the only accessible fields are the ones you configured in Group by Fields. When aggregating multiple messages to one event, you lose all fields you do not group by. Add the user key, (which you put in Group by Fields) to the alert. Then, fill in the required information:
Give the key a name.
Check the box under Use Field as Event Key. This means that alerts will be grouped by this key. This is important in the next step because the group key is considered to be in a grace period. The number that is entered here, reflects the order in the grouping. If we were to add another key (like the IP address from which the user connects), then we would first group by user and then group by IP, entering 2 for the IP address here.
Select a template for Set Value From. This will extract the field from the resulting aggregation or filtered log message. Enter
${source.user}. The resulting aggregation will be the source containing a field named User since we entered it in Group By Fields.
If you only want events which set the key (important if you have filtered log messages), then you could check the Require all template values to be set box. But since this is an aggregation, this step is not needed, and you can move on to the next page.
Add an Alert
You want to receive an email when an event is raised. Configuring an event will elevate it to an alert. You can read more about setting up an email alert under alert types.
Select your predefined email alert and set the Grace Period to 5 minutes. If you are targeted by a brute force attack, you do not want to get an email every 10 seconds reminding you that you are being attacked. This grace period will only be respected per the event key you selected in custom fields. So you will only get an email for usernames the attackers are targeting.
Since this is an aggregated event, setting a number in Message Backlog might not be helpful. You can leave it unchecked. The backlog will show all messages within the time range of Search within the last and it will use the query you entered. Selecting a number here will limit the amount of messages in the backlog.
Save the Event Definition
Take a final look at the event definition on the Summary page. After checking that all configurations are correct, click on Create event definitionto save the event definition.
Sample Alert
If an attacker tries to login 10 times in under 10 seconds, you will receive an email like the one below:
Notice that the user was set as key.
Create a Widget
Navigate to the search page and create a widget following these steps:
Select the All events stream to narrow down all messages to events.
Add
event_definition_idto the query in order to only display the events related to the newly created event definition.Create a new aggregation widget and set Direction as Row and Field as timestamp.
Type
keyin the Columns field and select count() under Metrics.
The Brute force events per user widget can be added to a report.