Managing log data

Once logs have been ingested into Security Data Lake, they are filtered, enriched, and routed through a process known as Data Routing.

Routing occurs primarily at the stream level, where rules and filters determine how data moves through the system and where it is stored.

Streams are the mechanism Security Data Lake uses to organize and route log data. Each message is assigned to one or more streams based on defined rules.

You can create pipeline rules to determine which messages are routed to specific streams, allowing you to apply different filters, transformations, or retention policies to various data sets.

A single message can belong to multiple streams, making it possible to view and process the same data in different ways.

Pipelines provide a flexible way to transform and enrich messages after they have been routed into streams.

Each pipeline is composed of a series of stages, and each stage can include one or more pipeline rules. These rules apply specific functions, such as filtering, transforming, tagging, or rerouting messages, enabling deeper data enrichment and control over how logs are processed.

You can create and manage pipeline rules using the rule builder in the Security Data Lake interface.

After logs are routed through streams and pipelines, they can be directed to one or more destinations, depending on how you want the data to be stored and managed.

The primary destinations are:

Routing rules allow you to send log data to one or multiple destinations simultaneously, based on filters and defined criteria.

Data stored in a Data Lake is compressed and optimized for long-term retention and can be restored later for search and analysis when needed.

This approach is ideal for organizations that need to balance storage costs with long-term data availability.

Log data from a stream can be written directly into one or more index sets. An index set defines how data is stored and managed within the search backend, including rotation schedules, retention limits, and storage preferences.

This process allows you to manage the lifecycle of your index sets, including rotation, retention, storage backend selection, and optional archiving.

You can also apply index set templates with predefined settings that align with your performance, retention, and cost requirements.

Security Data Lake supports multiple data tiers, enabling you to balance speed and efficiency by assigning data to different storage tiers based on access frequency.