Data Routing
Data Routing refers to the process of filtering, enriching, and directing log data after it has been ingested into Graylog. This is achieved by applying stream rules and pipeline rules to determine where and how data is processed and stored. Routing is configured at the stream level, so the process begins by identifying which stream you want to manage.
Configure data routing for a stream
To apply Data Routing on a stream:
Open the Streams tab from the top-level menu.
Locate the stream you want to configure.
Select Data Routing for that stream.
Security Data Lake guides you through three main stages: Intake, Processing, and Destinations.
1. Intake
At this stage, you can create or modify stream rules to define which data moves from an input into the selected stream.
To add a new rule, select Create Rule. For more information on creating rule, refer to stream rule creation.
2. Processing
This step displays the pipelines currently attached to the stream and allows you to apply additional ones.
To connect a new pipeline, follow these steps:
Select Edit pipeline connection.
Choose the pipeline you want to add.
Click Update connections.
If the stream was created as part of an Illuminate content pack, note that some data processing rules might already be applied before your new pipeline rules run.
3. Destinations
In the final step, you define where your processed log data will be stored or archived.
You can enable one or more destinations based on your data-management needs:
Data Lakes – Store large volumes of log data in long-term, lower-cost storage such as Amazon S3. Ideal for logs that you want to retain but not actively search or analyze. (Enterprise feature)
Index Sets – Route data into Security Data Lake’s searchable storage, optimized for analysis, alerting, and event management.
Note
When setting up destinations for a stream, consider the following:
What is the purpose of routing or storing this data, and which destination best meets that goal?
What specific data should be routed to each destination?
What filters or rules can be applied to include or exclude certain data?
To make these decisions, it’s important to understand the types of destinations available and their use cases.
To enable a destination, toggle the corresponding switch. If a Data Lake or index set is not available, you’ll receive a message explaining why it cannot be selected.
You can route all data to multiple destinations or apply filters to send only specific subsets of data to each one.
Create a Filter Rule
You can apply filter rules to control which data is excluded from a destination.
To create a new filter rule, follow these steps:
Expand the destination type using the arrow on the right.
In the Filter Rules section, select Create Rule to open the Filter Rule wizard.
Note
Filter rules are exclusionary: by default, all stream data is routed to a destination unless you explicitly exclude certain messages through these rules.
Filter rules are defined using conditions based on message fields or attributes. For example, a specific input source, field value, or whether a field is null.
Each rule must include at least one condition, and you can add multiple conditions to refine the filter.