Skip to main content

Index Model

Security Data Lake transparently manages one or more sets of indices to optimize search and analysis operations for speed and low resource consumption.

To enable managing indices with different mappings, analyzers, and replication settings, Security Data Lake uses index sets, which are an abstraction of all these settings. For information on creating index default configuration templates, see Index Set Templates.

Index Sets

Each index set contains the necessary settings for Security Data Lake to create, manage, and fill search backend indices and handle index rotation and data retention depending on your specific requirements or needs.

Security Data Lake is maintaining an index alias per index set that is always pointing to the current write-active index from that index set. There is always exactly one index to which new messages are written until the configured rotation criterion (number of documents, index size, or index age) has been met.

A background task continuously checks if the rotation criterion of an index set has been met, and a new index is created and prepared when that happens. Once the index is ready, the index alias is automatically switched to it. That means that all Security Data Lake nodes can write messages into the alias without even knowing what the current write-active index of the index set is.

Almost every read operation is performed with a given time range. Because Security Data Lake writes messages sequentially into search backend, it can keep information about the time range each index covers. It selects a list of indices to query when having a time range is provided. If no time range was provided, it will search in all known indices.

Eviction of Indices and Messages

There are configuration settings for the maximum number of indices Security Data Lake is managing in a given index set.

Depending on the configured retention strategy, the oldest indices of an index set will automatically be closed, deleted, or exported when the configured maximum number of indices has been reached.

The deletion is performed by the Security Data Lake leader node in a background thread, which is continuously comparing the number of indices against the configured maximum:

INFO : org.graylog2.indexer.rotation.strategies.AbstractRotationStrategy - Deflector index <graylog_95> should be rotated, Pointing deflector to new index now!
INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <graylog_95> to <graylog_96>.
INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_96>.
INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <graylog_96>.
INFO : org.graylog2.indexer.MongoIndexSet - Index <graylog_96> has been successfully allocated.
INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <graylog_deflector> to new index <graylog_96>.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <f1018ae0-dcaa-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <graylog_deflector> to index <graylog_96>.

Index Set Configuration

There are a variety of options for index sets related to how Security Data Lake will store messages into the search backendcluster. These sets can be managed on the Indices & Index Sets page by navigating to System > Indices.

To create a new set, select the Create index set button at the top right of the page. From here complete the following fields to create the index set:

  • Title: A descriptive name of the index set.

  • Description: A description of the index set for human consumption.

  • Index prefix: A unique prefix used for search backendindices managed by the index set. The prefix must start with a letter or number, and can only contain letters, numbers, _, -and +. The index alias will be named accordingly, e. g. bitdefender_deflector if the index prefix was bitdefender.

  • Analyzer: (default: standard) The search backend analyzer for the index set.

  • Index shards: (default: 1) The number of search backend shards used per index.

  • Index replicas: (default: 0) The number of search backend replicas used per index.

  • Max. number of segments: (default: 1) The maximum number of segments per search backend index after index optimization (force merge). (See Segment Merging for details.)

  • Disable index optimization after rotation: Disable search backend index optimization (force merge) after index rotation. Only activate this if you have serious problems with the performance of your search backend cluster during the optimization process.

  • Field type refresh interval: Determines how often the field type information for the active write index will be updated in increments of seconds or minutes.

Index Rotation Configuration

You must also determine the index rotation strategy for this index set. By default, Data Tiering is selected under Rotation & Retention, and this is the recommended option. See Data Tiering for complete information about this method and the different configuration options.

Index Retention Configuration

Select your strategy for index set retention:

  • Archive: Automatically archive an index before closing or deleting it.

  • Delete: Delete indices in search backend to minimize resource consumption.

Maintenance

Keeping the Index Ranges in Sync

Security Data Lake will take care of calculating index ranges automatically as soon as a new index has been created.

In case the stored metadata about index time ranges has run out of sync, Security Data Lake will notify you in the web interface. This can happen if an index was deleted manually or messages from already “closed” indices were removed.

The system will offer to re-generate all time range information.

You can easily re-build the information yourself after manually deleting indices or doing other changes that might cause synchronization problems:

$ curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild

This will trigger a system job:

INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog2_*): 5 indices affected.
INFO : org.graylog2.indexer.ranges.MongoIndexRangeService - Calculated range of [graylog_96] in [7ms].
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Created ranges for index graylog_96: MongoIndexRange{id=null, indexName=graylog_96, begin=2017-01-17T11:49:02.529Z, end=2017-01-17T12:00:01.492Z, calculatedAt=2017-01-17T12:00:58.097Z, calculationDuration=7, streamIds=[000000000000000000000001]}[...]
INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 5 indices. Took 44ms.
INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <9b64a9d0-dcac-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 46ms.

Manually Rotating the Active Write Index

Sometimes you might want to rotate the active write index manually and not wait until the configured rotation criterion for the latest index has been met, if for example, you’ve changed the index mapping or the number of shards per index.

You can do this either via an HTTP request against the REST API of the Security Data Lake leader node or via the web interface:

$ curl -XPOST https://***.gravityzone.bitdefender.com/api/system/deflector/cycle
Default Index Set.png

A manual rotation produces logs similar to the following:

INFO : org.graylog2.rest.resources.system.DeflectorResource - Cycling deflector for index set <58501f0b4a133077ecd134d9>. Reason: REST request.
INFO : org.graylog2.indexer.MongoIndexSet - Cycling from <graylog_97> to <graylog_98>.
INFO : org.graylog2.indexer.MongoIndexSet - Creating target index <graylog_98>.
INFO : org.graylog2.indexer.indices.Indices - Created Graylog index template "graylog-internal" in Elasticsearch.
INFO : org.graylog2.indexer.MongoIndexSet - Waiting for allocation of index <graylog_98>.
INFO : org.graylog2.indexer.MongoIndexSet - Index <graylog_98> has been successfully allocated.
INFO : org.graylog2.indexer.MongoIndexSet - Pointing index alias <graylog_deflector> to new index <graylog_98>.
INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <024aac80-dcad-11e6-97c3-6c4008b8fc28> [org.graylog2.indexer.indices.jobs.SetIndexReadOnlyAndCalculateRangeJob]
INFO : org.graylog2.indexer.MongoIndexSet - Successfully pointed index alias <graylog_deflector> to index <graylog_98>.
INFO : org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy - Number of indices (5) higher than limit (4). Running retention for 1 index.
INFO : org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy - Running retention strategy [org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy] for index <graylog_94>
INFO : org.graylog2.indexer.retention.strategies.DeletionRetentionStrategy - Finished index retention strategy [delete] for index <graylog_94> in 23ms.

Data Tiering

Security Data Lake gives you the option to store and manage your index set data in tiers. When you set up index sets, you establish tier properties based on your data retention requirements. This method allows you to utilize storage effectively for both performance and cost-effectiveness.

This article explains how Security Data Lake implements Data Tiering and what you need to know to establish effective policies for your index sets.

Why Use Data Tiering?

With Security Data Lake Data Tiering, each tier serves as a storage repository for data that needs to be handled in a similar way. Data tiers can be thought of as levels of data with the same storage and management specifications. Data is grouped based on how often it is used and the level of search performance required. Each tier has its own storage and accessibility criteria, meaning that you can move your data to a tier that best aligns with your needs.

Data Tiering can help lower storage costs because less frequently accessed data can be stored in a lower cost tier. For example, data that only needs to be retained for compliance checks can be stored in a tier that does not provide high performance storage and is therefore less expensive. You can reserve the costlier, high performance tier for more recent and more frequently searched data.

Data can be classified into tiers based on:

  • performance requirements

  • frequency of use

  • cost efficiency

We recommended Data Tiering as a cost-effective way of storing data for self-managed installations. Data Tiering provides three tiers for data storage: the hot tier, the warm tier, and the archive. Each of these tiers is described in detail below.

The Hot Tier

New indices that are part of a data stream are automatically allocated to the hot tier, which in the case of tiering refers to the search backend cluster you use for storage and acts as the default for all incoming data. Data in the hot tier is easy to access and search, but operating costs are generally higher because of the resources that must be allocated to maintain it.

The Warm Tier

Data in the warm tier is searchable, but search performance is lower compared to the hot tier. Warm tier data is stored in searchable snapshots and not directly in index sets. When a search is triggered in the warm tier, the system loads the data from this restored layer into the search backend cluster. The warm tier is suitable for storing data that does not require frequent access, such as logs from recent weeks.

A searchable snapshot index reads from the repository and does not download all data to the cluster at restore time. This method makes the warm tier a cost-effective storage solution. Snapshots are stored in a warm storage repository, which can be either an Amazon S3 bucket, or a local file system. Searchable snapshots remain in the repository in snapshot format and are read-only.

Archiving

Security Data Lake offers archiving for less critical data, making it a lower cost option for storing compliance and historical data.

The archive stores messages until you need to re-process them into Security Data Lake for analysis. You can instruct Security Data Lake to automatically archive log messages to compressed flat files on the local file system or an S3-compatible object storage. Messages are stored before retention cleaning begins, and they are not deleted from the search backend.

Note

Currently, you can utilize both a Data Lake and archives to preserve your log data long term. Both features perform similar functions. However, there are benefits to using a Data Lake for less immediately valuable data. Retrieving logs from a Data Lake is a faster process because log retrieval is granular. Additionally, the data in a Data Lake is compressed, so it is generally a lower-cost option for data storage.

Set Up Data Tiering

You can establish Data Tiering when you create or update index sets. In fact, you can set different tier policies for each index set, based on the requirements of the data they contain.

After your initial set up of the warm tier, we recommend that you monitor the system resource utilization of the Security Data Lake warm tier nodes to determine the optimal amount of disk space for their file caches. In particular, the active vs. used percentage metrics (along with the total, active, used, and evicted bytes of the file cache) should be closely observed.

The percentage of used file cache should be less than the percentage of active file cache, whose values are ideally 70% or less. Importantly, the number of active bytes in the file cache should be less than the used bytes in the file cache, and the used bytes should be less than the total bytes of the file cache, so:

Active bytes < used bytes < total bytes

If you enable Data Tiering with the Security Data Lake Data Node, then you must first issue a certificate authority for the third-party tool you use to query OpenSearch Node's API. Details on issuing a certificate authority may be found in the Data Node documentation.

If you are using self-managed OpenSearch, proceed to the following section.

Retrieve File Cache Metrics

The OpenSearch node stats API allows you to retrieve statistics about your cluster. Here is an example cURL command that can be used to retrieve the file cache metrics discussed in the previous section. (This may be useful in cases where OpenSearch metrics are not being captured and stored within a time-series datastore such as InfluxDB or Prometheus.) 

$ curl -s -XGET "http://admin:[email protected]:9200/_nodes/stats/file_cache?pretty"

The following is an example snippet of output from the response of the above command.

"jW4Q6SuXQASt8lM796CBHg" : {
      "timestamp" : 1712857680055,
      "name" : "10.0.1.229",
      "transport_address" : "10.0.1.229:9300",
      "host" : "10.0.1.229",
      "ip" : "10.0.1.229:9300",
      "roles" : [
        "search"
      ],
      "attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "file_cache" : {
        "timestamp" : 1712857680055,
        "active_in_bytes" : 50066073022,
        "total_in_bytes" : 128849018880,
        "used_in_bytes" : 71497646618,
        "evictions_in_bytes" : 0,
        "active_percent" : 70,
        "used_percent" : 55,
        "hit_count" : 83077,
        "miss_count" : 877
      }
    }

For your reference, the following are the available OpenSearch metrics that may be retrieved via the node stats API:

  • nodes.stats.file_cache.active_in_bytes

  • nodes.stats.file_cache.total_in_bytes

  • nodes.stats.file_cache.used_in_bytes

  • nodes.stats.file_cache.evictions_in_bytes

  • nodes.stats.file_cache.active_percent

  • nodes.stats.file_cache.used_percent

  • nodes.stats.file_cache.hit_count

  • nodes.stats.file_cache.miss_count

Archiving

Storing extensive amounts of data in your search backend can be costly. Security Data Lake allows you to store inactive data in the Graylog archive to help lower storage costs and maximize retention as needed. When archiving index sets, you can set a retention period based on a variety of factors. Archived indices are then deleted after the retention period is complete.

In Security Data Lake, you can archive index sets to compressed flat files on the local file system, to as S3-compatible object storage, or to Google Cloud Storage (GCS). Note that archived index sets are stored before retention cleaning begins so that no data is lost.

Note

Archived data is inactive and cannot be searched unless restored. If necessary, archived indices can be re-imported through the user interface. After the data is restored, you can search through and analyze this data via the web interface. For more information, refer to Restore an Archive.

Select a Max Segment Size

When you archive an index, the archive job writes the data into segments. The Max Segment Size setting sets the size limit for each of these data segments to control the size of the segment files and process them with tools with a file-size limit.

Once the size limit is reached, a new segment file is started. For example:

/path/to/archive/
  graylog_201/
    archive-metadata.json
    archive-segment-0.gz
    archive-segment-1.gz
    archive-segment-2.gz

Configure Index Retention with Data Tiering

Note

Our model for rotation and retention of indices is referred to as data tiering. Data tiering is offered as an option to store and manage data in tiers for specific purposes. See Data Tiering for more information.

You can determine how long you want to retain the data in a new or existing index set. To do so:

  1. Navigate to the Rotation and Retention section on the index set configuration page.

  2. Toggle to the Data Tiering option.

  3. Select the minimum and maximum amount of days you want to store your data.

Data is deleted once the maximum amount of days are reached. If you wish to keep this data, you may select Archive before deletion.

You may also select Enable Warm Tier. You can then enter the minimum amount of days you want the data to stay in the hot tier before moving it to warm. You can also designate a warm storage repository to store the index set in. If none exist, you can create a new warm storage repository in this menu.

Select Streams To Archive

The Streams to Archive setting is included in the archive and allows you to archive only important data as determined by your streams, rather than everything that is brought into Security Data Lake.

Note

New streams are archived automatically. If you create a new stream and don’t want it to be archived, disable it in this configuration dialog.

Archive Data in an Index Set

Now that you have completed setup for Security Data Lake archiving, you may begin to archive your data as desired. You can choose to archive an index set directly or define a specific timeframe to set up automatic archiving.

Create a New Archive for an Index Set

To manually archive a specific index set:

  1. Navigate to Enterprise > Archives.

  2. Select the desired index from the drop down menu.

  3. Click Archive Index.

The archived index set can be viewed in the Archive Catalog found on the same page in the interface.

Set Up Automatic Archiving for an Index Set

Alternatively, you can determine indices to be archived at specific times. This may be done via the Security Data Lake interface or with the REST API.

Automate Archiving via The Security Data Lake Interface

  1. Navigate to System > Indices.

  2. Locate the index set you wish to archive.

  3. Click the Edit button.

  4. Scroll down to the Rotation and Retention section.

  5. Now, you will need to choose your rotation and retention strategy. Select between Data Tiering or Legacy. Note that our recommended model for rotation and retention of indices is data tiering (see Data Tiering for more information). Legacy rotation and retention strategies are supported currently but will be deprecated in future.

    • If you select Data Tiering, check the Archive before deletion box.

    You may also set the maximum amount of indices allowed before deletion begins. The oldest indices are deleted first.

  6. Click Update index set.

Automate Archiving via The REST API

The Security Data Lake REST API can also automate archive creation if you need a more flexible approach.

rest API Plugins.png

In the example below, a curl command is used to archive an index. This command starts a system job in the Security Data Lake server to create an archive for index graylog_386.

system_job.id is used to check the progress of the job.

$ curl -s -u admin -H 'X-Requested-By: cli' -X POST http://127.0.0.1:9000/api/plugins/org.graylog.plugins.archive/archives/graylog_386
Enter host password for user 'admin': ***************
{
   "archive_job_config" : {
     "archive_path" : "/tmp/graylog-archive",
     "max_segment_size" : 524288000,
     "segment_filename_prefix" : "archive-segment",
     "metadata_filename" : "archive-metadata.json",
     "source_histogram_bucket_size" : 86400000,
     "restore_index_batch_size" : 1001,
     "segment_compression_type": "SNAPPY"
   },
   "system_job" : {
     "id" : "cd7ebfa0-079b-11e6-9e1b-fa163e6e9b8a",
     "description" : "Archives indices and deletes them",
     "name" : "org.graylog.plugins.archive.job.ArchiveCreateSystemJob",
     "info" : "Archiving documents in index: graylog_386",
     "node_id" : "c5df7bff-cafd-4546-ac0a-5ccd2ba4c847",
     "started_at" : "2016-04-21T08:34:03.034Z",
     "percent_complete" : 0,
     "provides_progress" : true,
     "is_cancelable" : true
   }
 }

The REST API can automate other archive-related tasks, such as:

  • Restoring and deleting archives

  • Updating your archive configuration

We encourage you to explore the REST API browser, which can be found under System > Nodes.

Set a Retention Period for Legacy Strategies

You may need to set a variety of retention periods for different indices. Some indices may only require a two month retention period whereas others may require longer periods of storage. For example, firewall logs are often kept for a period of two months, but other log messages may need to be kept for longer periods to fulfill compliance regulations.

After the retention period is complete, Security Data Lake automatically deletes older messages.

Next Steps

Now that you have created your first archive, start the process of restoring archived data.

Restore an Archive

Once you have set up Security Data Lake archiving and archived your first set of data, you can now restore your archives as required, like when you may need to search and analyze the data.

Restore Archived Data

Note

The restore process adds loads to your OpenSearch cluster because all messages are effectively re-indexed; we advise that you first test small archives to see how the cluster behaves before restoring large sets of data. Also, use the restore index batch size setting to control the OpenSearch batch size on re-index.

You may restore archived indices via the:

Note that Security Data Lake restores all indices in the Restored Archives index set to avoid conflicts with the original indices (if they still exist).

restored archives.png

Restored indices are also marked as reopened, so they are ignored by index-retention jobs and are not closed or deleted. Therefore, you must manually delete restored indices when you no longer need them.

Restore Indices via the Security Data Lake Interface

To restore an archive in the interface:

  1. Navigate to Enterprise > Archives.

  2. Select an index set from the Archive Catalog section.

  3. Click the Restore Index button.

You may also restore or delete multiple index sets via the Bulk Actions drop down found next to the search box.

Restore Indices via the REST API

As with archive creation, you can also use the REST API to restore an archived index into the OpenSearch cluster:

$ curl -s -u admin -H 'X-Requested-By: cli' -X POST http://127.0.0.1:9000/api/plugins/org.graylog.plugins.archive/archives/graylog_386/restore
Enter host password for user 'admin': ***************
{
   "archive_metadata": {
     "archive_id": "graylog_307",
     "index_name": "graylog_307",
     "document_count": 491906,
     "created_at": "2016-04-14T14:31:50.787Z",
     "creation_duration": 142663,
     "timestamp_min": "2016-04-14T14:00:01.008Z",
     "timestamp_max": "2016-04-14T14:29:27.639Z",
     "id_mappings": {
       "streams": {
         "56fbafe0fb121a5309cef297": "nginx requests"
       },
       "inputs": {
         "56fbafe0fb121a5309cef290": "nginx error_log",
         "56fbafe0fb121a5309cef28d": "nginx access_log"
       },
       "nodes": {
         "c5df7bff-cafd-4546-ac0a-5ccd2ba4c847": "graylog.example.org"
       }
     },
     "histogram_bucket_size": 86400000,
     "source_histogram": {
       "2016-04-14T00:00:00.000Z": {
         "example.org": 227567
       }
     },
     "segments": [
       {
         "path": "archive-segment-0.gz",
         "size": 21653755,
         "raw_size": 2359745839,
         "compression_type": "SNAPPY"
         "checksum": "751e6e76",
         "checksum_type": "CRC32"
       }
     ],
     "index_size": 12509063,
     "index_shard_count": 4
   },
   "system_job": {
     "id": "e680dcc0-07a2-11e6-9e1b-fa163e6e9b8a",
     "description": "Restores an index from the archive",
     "name": "org.graylog.plugins.archive.job.ArchiveRestoreSystemJob",
     "info": "Restoring documents from archived index: graylog_307",
     "node_id": "c5df7bff-cafd-4546-ac0a-5ccd2ba4c847",
     "started_at": "2016-04-21T09:24:51.468Z",
     "percent_complete": 0,
     "provides_progress": true,
     "is_cancelable": true
   }
 }

The returned JSON payload contains the archive metadata and the system job description that runs the index-restore process.

Restore into a Separate Cluster

The added load from restored indices slows down your indexing speed. To avoid adding more load to your primary OpenSearch cluster, you can also restore the archived indices on a different cluster:

  1. Transfer the archived indices to a different machine.

  2. Place them in a configured backend.

Each index archive is in a separate directory, so if you only want to transfer one index to a different machine, copy the corresponding directory into the backend. For example:

$ tree /tmp/graylog-archive
  /tmp/graylog-archive
  ├── graylog_171
  │   ├── archive-metadata.json
  │   └── archive-segment-0.gz
  ├── graylog_201
  │   ├── archive-metadata.json
  │   └── archive-segment-0.gz
  ├── graylog_268
  │   ├── archive-metadata.json
  │   └── archive-segment-0.gz
  ├── graylog_293
  │   ├── archive-metadata.json
  │   └── archive-segment-0.gz
  ├── graylog_307
  │   ├── archive-metadata.json
  │   └── archive-segment-0.gz
  ├── graylog_386
  │   ├── archive-metadata.json
  │   └── archive-segment-0.gz
  └── graylog_81
      ├── archive-metadata.json
      └── archive-segment-0.gz
  7 directories, 14 files

Indexer Failures

Security Data Lake nodes constantly keep track of every indexing operation they perform. This helps to make sure they don't unintentionally lose any messages. The web interface can show you a number of write operations that failed and also a list of failed operations. Like any other information in the web interface, this is also available via the REST APIs so you can hook it into your own monitoring systems.

Indexer Failures Zero.png

Information about indexing failures is stored in a capped MongoDB collection that is limited in size. A lot (many tens of thousands) of failure messages should fit in there but it should not be considered a complete collection of all errors ever thrown.

Common indexer failure reasons

There are some common failures that can occur under certain circumstances. These are explained here:

MapperParsingException

An error message would look like this:

MapperParsingException[failed to parse [failure]]; nested: NumberFormatException[For input string: "some string value"];

You try to write a string into a numeric field of the index. The indexer tries to convert it to a number, but fails because the string does contain characters that could not be converted.

This could be triggered by sending GELF messages with different field types or extractors trying to write strings without converting them to numeric values first. The recommended solution is to actively decide on field types. If you sent in a field like https_response_code with a numeric value, then you should never change that type in the future.

The same can happen with all other field types like booleans, for example.

Note that index cycling is something to keep in mind here. The first type written to a field per index wins. If the Security Data Lake index cycles, that means the field types are starting from scratch for that index. If the first message written to that index has the https_response_code set as string, then it will be a string until the index cycles the next time. Take a look at Index model for more information.

Indexer and Processing Failures Index

Security Data Lake allows you to store indexing and processing failure notifications in a dedicated failure index.

Failure messages are logged and aggregated on a dashboard and used to set up alert notifications, ensuring that you can analyze the messages and understand why the error occurred.

Configure Failure Processing

The individual failure types are enable by default for new installations of Security Data Lake. To to adjust your settings:

  1. Navigate to System > Configurations.

  2. Select Plugins, then Failure Processing.

  3. Click Edit configuration.

    Indexer Edit Configuration.png

  4. Select or clear the check box for each failure processing feature.

    • Log Indexing Failures: Stores indexer failure notifications and logs them in a dedicated Security Data Lake stream.

    • Log Processing Failures: Processes failure notifications to be stored in your search backend and logs them in a dedicated Security Data Lake stream.

    • Include Failed Messages: Displays a full log message in the failure notification for investigation. Enable either Log Indexing Failures or Log Processing Failures to activate this selection.

    • Continue Processing on Error: Stores the original message alongside a new field (gl2_processing_error) with specific error details. Meanwhile, a failure message with the error details is stored in the dedicated Security Data Lake stream. Enable Log Processing Failures to activate this selection.

  5. Click Update configuration to save your selections.

When failure processing is enabled, the widget in System Overview displays a failed-message counter.

indexer_failures1.png

Common Indexer Failure Reasons

The most common indexer failure is classified as a “MapperParsingException.”

For additional information on this type of failure, review Common Indexer Failure Reasons.

Common Processing Failure Reasons

A processing failure, which can occur within the Security Data Lake processing stack, may have multiple causes. The following is a list of the most common reasons:

  • RuleStatementEvaluationError

    • Occurs when there is an error in the statement between the “then” and “end” values of the pipeline rule.

  • RuleConditionEvaluationError

    • Occurs when there is an error in the statement between the “when” and “then” values of the pipeline rule.

  • ExtractorException

    • Occurs when an extractor or converter incorrectly reads or extrapolates a message.

  • MessageFilterException

    • Occurs when there is a backend system failure involving the Security Data Lake application; further troubleshooting with Security Data Lake support may be required.

  • InvalidTimestampException

    • Occurs when there is a failure during an attempt to set or extract a value in the timestamp field. For example, a pipeline rule failed while attempting to extract a timestamp from a string and attempted to assign this null timestamp to a message.

  • UNKNOWN

    • The reason for this error is unknown and will require further investigation into the log data.

In this section: