Streams
Streams use stream rules and pipeline rules to filter messages based on attributes such as message source, severity, or content.
This allows you to organize, process, and store log data in a structured way for targeted analysis and alerting.
Key Concepts
Default stream
Security Data Lake includes a built-in stream called All messages, which automatically receives any log data that doesn’t match another stream’s rules.
This default stream cannot be deleted or paused and serves as a catch-all for all incoming messages.
When you create custom streams with specific rules, only messages that match those rules are routed to those streams; all others remain in the default stream.
Stream Rules
Note
Stream rules are considered a legacy feature and will be deprecated in a future Security Data Lake release. We recommend using pipelines instead, as they provide greater flexibility for filtering, enriching, and routing messages.
Stream rules rules define the conditions that determine which messages are routed into a stream. Each stream can have multiple rules that evaluate fields, values, or keywords in real time.
Rules can be combined using logical operators such as AND or OR. For example, a stream rule might filter messages from a specific IP address or route logs based on severity levels such as error” or warning.
Pipelines and streams
Pipelines can be attached to streams to perform further processing.
A pipeline consists of sequential stages containing pipeline rules, which act on messages that meet specific conditions. Pipeline rules can enrich, transform, or reroute messages — for example, by extracting fields, renaming keys, or normalizing values.
To learn how to attach pipelines to streams, see Data Routing.
Streams vs. Saved Searches
The key difference between streams and saved searches is timing:
Streams process messages in real time, enabling immediate alerting and forwarding.
Saved searches are static queries against stored data.
Because streams tag each message as it is processed, filtering by stream ID later is much faster than rebuilding complex queries each time you search.
Manage streams
You can manage your existing streams on the Streams page from the top-level menu.
Each listed stream includes key details such as:
Archiving – Whether archiving is enabled for index sets linked to the stream.
Data Lake – Indicates if a Data Lake destination is configured for this stream.
Rules – The number of stream rules attached.
Pipelines – The number of pipelines applied.
Throughput – The rate at which messages are processed.
Status – Whether the stream is Running (active) or Paused (inactive).
Data Routing – Opens the configuration wizard for stream rules, pipelines, and destinations. For more information, refer to this KB article.
Share – Share this stream’s configuration with other users or teams.
More Actions – Access additional options such as managing stream rules, cloning the stream, or setting alerts.
Create a new stream
To create a new stream to route your log data, follow these steps:
Go to the Streams tab at the top of the page.
Select Create stream.
Enter a unique title and description, such as
All error messagesorNetwork logs by source.Select the index set where messages from this stream will be written.
Note
You can later modify this destination using Data Routing, such as routing messages to a Data Lake.
(Optional) Enable Remove matches from 'Default Stream' to exclude these messages from the default stream.
Select Create stream.
The stream is now saved but inactive. To start processing data, activate it by selecting Paused, which changes to Running once active.
Create and modify stream rules
Stream rules determine which messages flow into your streams. You can manage rules from either the Manage Rules menu or via Quick add rule.
Create rule via manage rules
To create a rule from the manager rules page, follow these steps:
On the Streams page, find the stream you want to modify and select More, then Manage Rules.
Under 1. Load a message to test rules, select a recent log message or load one by Message ID.
Under 2. Manage stream rules, select Add stream rule and complete the fields:
Field: The field name to filter on (e.g., source_ip).
Type: How to match values (e.g., match exactly, greater than, or contains).
Value: The value to match against (e.g., a specific IP address).
Inverted: Makes the rule exclusionary (e.g., exclude a specific IP).
Description (optional): A short note describing the rule.
Select Create Rule.
Repeat as needed to add more conditions.
Specify whether messages must match all or at least one rule to enter the stream.
Select I’m done! to complete setup.
To edit or delete a rule later, return to Manage Rules and use the Edit or Delete options.
Create a rule using Quick Add Rule
To quickly add a rule, follow these steps:
From the Streams page, click More > Quick Add Rule for the desired stream.
Under 2. Manage stream rules, select Add stream rule and complete the fields:
Field: The field name to filter on (e.g., source_ip).
Type: How to match values (e.g., match exactly, greater than, or contains).
Value: The value to match against (e.g., a specific IP address).
Inverted: Makes the rule exclusionary (e.g., exclude a specific IP).
Description (optional): A short note describing the rule.
Here you will can add a new stream rule to filter log data to this stream. Follow the instructions in step 5 of the previous section to create your new rule.
Click Create Rule to save.
To modify or delete a rule created via Quick Add, use More > Manage Rules on the same stream.
Route Logs From a Stream
Streams are central to how log data moves through Security Data Lake.
Data Routing begins at the stream level, where you apply rules, attach pipelines, and define destinations. For detailed instructions on configuring data routing, see Data Routing.
Stream processing
Every incoming message in Security Data Lake is evaluated against the rules of all configured streams. If a message satisfies all or at least one of a stream’s rules (depending on how the stream is configured), the internal ID of that stream is added to the message’s streams array.
This classification allows all searches, alerts, and processing operations that reference a specific stream to automatically limit their scope to messages belonging to that stream. This filtering happens transparently, you don’t need to manually specify the streams:[STREAM_ID] parameter in your searches.
Stream processing runtime limits
During message processing, stream classification plays a crucial role in overall system performance. Each message is evaluated against user-defined stream rules, and the message is added to a stream if its content matches those conditions.
Because this classification occurs during message indexing, the time required for rule matching directly affects throughput and system responsiveness.
If certain stream rules take too long to evaluate — for example, due to complex regular expressions — processing can slow down or even stall. In severe cases, messages can accumulate in memory, potentially leading to instability or data loss.
To prevent this, Graylog enforces runtime limits on stream rule matching:
If matching a message against a stream’s rules takes longer than the configured timeout, that stream’s rule evaluation for the message is aborted.
Message processing continues normally, but the event is recorded as a fault.
If a stream exceeds the allowed number of faults, Graylog automatically disables the stream to preserve system stability.
This behavior prioritizes overall message flow and system performance over stream classification accuracy for a small subset of messages.
However, in high-load environments, excessive stream disabling can occur if processing times exceed the configured limit. This usually indicates that the system is overutilized and cannot handle the current message volume.
Configuring timeout values
If the default timeout settings are not suitable for your environment, you can adjust the following parameters in the Graylog server configuration file:
Parameter | Description | Default |
|---|---|---|
stream_processing_timeout | Defines the maximum allowed time (in milliseconds) for evaluating a stream’s rules. If this limit is exceeded, the rule matching process is aborted and a fault is recorded. |
|
stream_processing_max_faults | Sets the maximum number of allowed timeouts per stream. Once this threshold is exceeded, the stream is automatically disabled until manually re-enabled. | 3 |
Excessive runtime causes
If only a single stream is being disabled while others perform normally, one or more rules within that stream are likely inefficient.
In most cases, this is caused by regular expressions, which can be computationally expensive depending on their complexity and the input being matched.
In particular, complex or poorly optimized regex patterns can cause catastrophic backtracking, where the evaluation time increases exponentially for certain inputs.
If stream processing timeouts continue to occur, follow these troubleshooting steps:
Review the stream rules for the affected stream, especially those using regular expressions.
Simplify or remove inefficient rules to reduce evaluation time.
Re-enable the stream once changes are made and confirm that processing performance improves.
Stream connections
Pipelines do not process messages on their own. For a pipeline to act on incoming data, it must be connected to one or more streams. This connection determines which messages are evaluated by that pipeline, allowing precise control over how different types of log data are processed and enriched.
The built-in function route_to_stream can be used within a pipeline rule to route a message to another stream.
Once a message is routed, the Pipeline Processor automatically identifies and executes any pipelines connected to that target stream. Although pipelines can trigger other pipelines through message routing, all incoming messages must first pass through at least one stream that has pipelines attached.
Default Stream
All messages received by Graylog are initially routed into the Default Stream. You can use this stream as the entry point to pipeline processing, allowing incoming messages to be routed to more streams and to be processed subsequently.
However, if you prefer to use the original stream matching functionality (i.e. stream rules), you can configure the Pipeline Processor to run after the Message Filter Chain (in the Message Processors Configuration section of the System > Configurations page) and connect pipelines to existing streams. This gives you fine-grained control over the extraction, conversion, and enrichment process.
By default, all messages received by Security Data Lake are initially routed into the Default Stream (All messages). You can use this stream as the starting point for pipeline processing, for example, to route messages to additional streams or apply specific transformation rules.
If you prefer to use the legacy stream rule–based matching functionality, you can configure the Pipeline Processor to run after the Message Filter Chain under System → Configurations → Message Processors Configuration. This approach lets you connect pipelines to existing streams and gain detailed control over extraction, transformation, and enrichment.
Message processor ordering
The order of message processors significantly affects how messages are handled. When designing your streams and pipelines, ensure that the processor order supports your intended logic, particularly when later processors depend on data modified or enriched earlier in the chain.