Field Types
Log messages that Security Data Lake receives typically consist of multiple fields. You can see the field types a message contains in search results or some dashboard widgets.
Field type refers to the data type of each field in a log message. A field type might be a string, boolean, or number, for instance. The field type then determines how the data is displayed, like in a dashboard. For example, you might choose to display a URL as a string.
Manage Field Mappings
The field type is assigned when Security Data Lake receives a message. However, for a given field, the received field type might not match the way you want to work with the data.
Field mapping allows you to change the field type on specific fields for incoming log messages. You may pair a field with a field type. For example the field host_ip could be mapped to the field type IP.
Field types can be changed by selecting Change field type in the drop-down menu presented when you click any field name in your search results. The current field type is displayed at the top of this menu. To create a new field mapping:
Click Change field type and select a new field type in the dialog box that appears.
Select the index sets you want to include. By default, field type changes are applied to all index sets of the current message or search.
Select the Rotate affected indices after change option to ensure that the selected indices are rotated when the field type is changed. In this case, the rotation is done immediately without waiting for the end of the rotation cycle.
Click Change field type.
Warning
Changing the field type can have a significant impact on log ingestion. Selecting a field type that is incompatible with the logs you are ingesting could lead to ingestion errors. Make sure that you enable Failure Processing and that the Processing and Indexing Failures stream is watched closely afterward.
Use Case
A user has ingested log messages that contain an IP address in the client_ip field. This field is indexed as keyword by default. The user can select this field and change its type to ip in the user interface. After performing the change and completing the index rotation cycle, the index mapping template is modified by Security Data Lake so that the client_ip field is indexed as ip.
Override Field Mappings
The origins of indexes and profiles are listed on the Configurations page. You can override origin values with custom mappings as follows:
Select Edit for the field you want to override. This action is available for both profiles and index values.
Choose the new field type from the drop-down menu in the Change Field Type dialog box.
Select the Rotate affected indices after change option to ensure that the selected indices are rotated when the field type is changed. If you clear this check box, you must manually rotate indices for the change to take effect.
Click Change field type.
This procedure does not remove or delete fields. You can override an index or profile temporarily and then the field value may be reset. An index or profile that is overridden is displayed on the Configurations page as an overridden index or overridden profile. You can also filter based on whether an index or profile is the origin or overridden.
You may use the Bulk actions button to remove multiple field mappings from an index or profile that is overridden.
Tip
If you are setting up a new Security Data Lake cluster and know what field types the cluster uses, you can benefit by creating your custom field types as part of your cluster set up.
Warning
Changing the field type can have a significant impact on log ingestion. Selecting a field type that is incompatible with the logs you are ingesting could lead to ingestion errors. It is recommended that Failure Processing is enabled and that the Processing and Indexing Failures stream is watched closely afterward.
Note
Assigning or removing a profile requires you to rotate the indices for the change to take effect. The
Rotate affected indices after changecheck box is selected by default so this change happens automatically. If you clear the check box, you must manually rotate the indices, as described
above.
Manage Field Type Profiles
The Field Type Profiles tab is where you can create, view, and manage index set field type profiles. The list view shows all your profiles. Use the filter to limit the list or find specific profiles by keyword.
The Custom Field Mappings column shows the number of custom field mappings that are present. You can click the number to see the mappings. The list view also shows any index sets the profile is assigned to.
As with other entity lists in Security Data Lake, you can show or hide columns and modify the way they are sorted. As described above, you can create a new profile by clicking Create profile.
Edit a Profile
Warning
Be careful when updating or removing profiles that are currently assigned to index sets to avoid unintended results.M
You can edit profiles in the Field Type Profiles tab. Click Edit on a profile, then update the information. You can change existing field type mappings as well as add or delete mappings.
You can also remove any profile on this page. Select More > Delete from the list view to permanently remove a profile.
Stream-Aware Field Types
Stream-aware field types allow you to map your streams to related field types. This provides you with precise field-type suggestions based on your stream selection.
Note
This feature is disabled by default to reduce performance impact. You can enable this configuration property in your Security Data Lake configuration file if needed by setting the stream_aware_field_types configuration property to stream_aware_field_types = true.
When the configuration property stream_aware_field_types is set to true in the Security Data Lake server configuration file, Security Data Lake will periodically collect information on stream-field relations from your search backendand use it to provide only those fields that are present in the streams used in the query.
If all of your streams go to dedicated, separate index sets, it is advised to keep the default value of stream_aware_field_types property as false. This will decrease the load on the search backend, and stream separation across index sets helps with showing proper fields for a query. On the other hand, if multiple streams go to the same index sets and you want precise field types and suggestions, you can set it to true.
For example, in the image below, we have selected the Illuminate:O365 Messages stream, and the Fields list to the left of the screen displays suggested field types based on this stream.
Important
It is a best practice to monitor your data node (ES or OS) load after you enable this feature, especially when using large numbers of streams and fields.