Search Configuration
Security Data Lake allows you to customize search query options, such as limiting the time range you can select or configuring the list of displayed relative time ranges.
All search configuration settings can be customized using the web interface on the System > Configurations page in the Search section.
Query Time Range Limit
Sometimes the amount of data stored in Security Data Lake is quite big and spans a wide time range (e. g. multiple years). To prevent accidentally running search queries that could use too many resources, you can limit the time range that searches can run in.
Using this feature, the time range of a search query exceeding the configured query time range limit will automatically be adapted to the given limit.
The query time range limit is a duration formatted according to ISO 8601 following the basic format P<date>T<time> with the following rules:
Designator | Description |
|---|---|
| Duration designator (for period) placed at the start of the duration representation |
| Year designator that follows the value for the number of years |
| Month designator that follows the value for the number of months |
| Week designator that follows the value for the number of weeks |
| Day designator that follows the value for the number of days |
| Time designator that precedes the time components of the representation |
| Hour designator that follows the value for the number of hours |
| Minute designator that follows the value for the number of minutes |
| Second designator that follows the value for the number of seconds |
Examples:
ISO 8601 duration | Description |
|---|---|
| 30 days |
| 1 hour |
| 1 day and 12 hours |
More details about the format of ISO 8601 durations can be found here.
Time Range Presets
The list of time ranges displayed in the Relative Time Frame Selector can be configured, too. It consists of a list of ISO 8601 durations that you can select on the search page.
Search Result Highlighting
Security Data Lake supports search result highlighting:
Enabling/Disabling Search Result Highlighting
Using search result highlighting will result in slightly higher resource consumption of searches. You can enable and disable it using a configuration parameter in the graylog.conf of your Security Data Lake nodes:
allow_highlighting = true
View Query String History
Security Data Lake enables you to search through your recent query string history to retain queries you have used in other event replays and dashboards. The search bar supports auto completion and will display relevant search queries you have entered in the past. When clicked on, these queries will replace the current query string.
The search query history button is found at the end of the search bar, to the right of the light bulb icon. All queries are saved to the database, making it possible to search through past queries via the drop-down menu that appears when you click on the Search History button. Previous searches are listed in descending order from most recent to oldest.
Tip
The shortcut, alt-space shows suggestions for a query input. When the input is empty this will show query history suggestions. If you already have an input use alt-ctrl-h.
The query string history feature enables you to filter through previous searches and reuse one of them. This functionality is also present in dashboards and widgets. Queries are scoped by user, so no one else sees your queries, but they are shared between dashboards. Please refer to Saved Searches for more details.
Time Frame Selector
The time frame selector allows you to pull specific time ranges from your Security Data Lake data and analyze issues that affect your environment. Most importantly, this tool offers multiple ways to filter time ranges. It is found in the upper left corner of the Search page.
This tool helps you build queries that can perform actions such as:
Understanding and responding to data breaches, broken processes, and other security incidents
Troubleshooting systems and networks
Understanding the behaviors of your users
Conducting forensics activities
Time Frame Options
To access the window, click the clock icon. A dialog offers the following ranges:
Relative
Absolute
Keyword
Relative Time Frame Selector
The Relative time frame selector lets you search for messages within time ranges relative to Now or another date of your choosing. This selector offers a wide set of relative time frames that fit most of your search needs, including an All Time option.
Consider how this filter works:
The From field allows you to type in values and select units for time via a drop-down menu. You can choose from seconds, minutes, hours, and days. For your convenience, you can click the Preset Times button to access pre-determined times, interpreted in minutes, hours, and days. If you decide to select all messages instead, your dashboard would display data from the date of first ingestion.
The Until date allows relative time ranges to end at a specific period instead of default to the current time/date.
Absolute Time Frame Selector
Use the absolute time frame selector when you precisely know the boundaries of your search. This option displays an accordion containing two options:
Calendar
Timestamp
In the Calendar option, use the hourglass icon to jump from the very beginning of the day (00:00:00.000) to the very end of the day (23:59:59.99).
To understand Calendar in more detail, consider the functions of Until and From:
Until defaults to disabling all dates previous to the selected From date.
From date will disable all previous dates if you configure a Query Time Range Limit (on the System > Configurations page).
You can use the magic wand icon for both Calendar and Timestamp.
In Calendar, the icon updates the Time to the current time but does not modify the date in the calendar.
In Timestamp, the icon updates the entire Timestamp to the current date and time.
Keyword Time Frame Selector
Security Data Lake offers a keyword time frame selector that allows you to specify the time frame for the search in natural language like last hour or last 90 days. The web interface shows a preview of the two actual timestamps that will be used for the search.
Here are a few examples for possible values.
"last month" searches in between the 1st day of last month to the last day of the current month
“4 hours ago” searches between four hours ago and now
“1st of April to 2 days ago” searches between 1st of April and 2 days ago
“yesterday midnight +0200 to today midnight +0200” searches between yesterday midnight and today midnight in
timezone +0200 - will be 22:00 in UTC
The time frame is parsed using the natty natural language parser. Please consult its documentation for details.
Note
Natty in version 4.2+: From 4.2 on, some errors/irregularities with natty have been addressed. When natty interferes the time part of a query string (e.g. "last Monday"), it uses the reference time. This creates timestamps in the mid of the day which is counter-intuitive and not really expected. Instead, from now on, when natty interferes the time part in a query, this time part gets aligned to the start and end of the day.
Adding Customized Time Range Presets
You can customize keyword time ranges and add them to existing selections. There are two ways to do this.
From the Time Range Selector Menu
In the Time Range Selector menu, select either Relative, Absolute, or Keyword for the preset type of your choice.
Enter the desired configuration and click Update time range.
From the Configuration Menu
In the configuration interface, click Configure Presets found at the bottom of the Time Range Selector drop-down menu. Optionally, you can go to System > Configurations and select Edit configuration.
Click Add option at the bottom of the Search Time Range Presets list. Enter a description and click Update Configuration.
To add more time ranges, click Add option and edit the new time range. Then click Update configuration.
Managing Customized Time Range Presets
You can rearrange the entries in the list according to priority. Select the dots found at the beginning of the row and drag up or down.
You can also access your customized time range preset in the Time Range Selector drop down menu. There you will see the description that you entered during customization.
Frequently used time ranges can be saved and added to the Search Time Range Presets list. To do so, click the Save as Preset button in the top right corner of the menu. Enter a description and click Save preset. You will be notified if you enter a preexisting time range. In the Time Range Selector, click Load Preset to retrieve saved presets.