Sidecars
Security Data Lake Sidecar is a lightweight configuration management system for log collectors. It provides a centralized framework to deploy, configure, and manage collectors such as Winlogbeat, Filebeat, and NXLog across multiple hosts. This simplifies log collection and ensures consistent configurations in distributed environments.
Sidecar architecture
The Security Data Lake node serves as the central hub for managing log collector configurations. The Sidecar daemon periodically retrieves the latest configurations for each target using the REST API. On its first run, or whenever a configuration change is detected, Sidecar generates the necessary backend configuration files and restarts the affected log collectors. All collector configurations are centrally managed through the Security Data Lake web interface, ensuring consistency and simplified administration.
Sidecar can run as a service (Windows host) or as a daemon (Linux host) on supported, message-producing devices.
Installing Sidecar
To install and make the necessary configurations for Sidecar, follow these steps:
1. Set up the sidecar in Security Data Lake
The following steps must be taken in Security Data Lake in order to create and configure a sidecar:
1.1 Set up an input
Before installing Sidecar, create a new Beats input in order to receive data from the Beats collector, and configure it to receive Sidecar logs on port 5044.
1.2 Create an API token
Select the user icon on the upper right side of the screen and select Edit profile.
Select the Edit tokens button on the upper right side of the screen.
Enter the token details and select Create Token.
Save the API server token in a safe yet accessible location in case you need to retrieve it.
1.3 Download Sidecar
You can find .deb, .rpm, .msi, .exe, .arm, .tar and .gz packages in our package repository. Please follow the version matrix to select the correct package and download it from our GitHub page.
2. Prepare your local environment
You can install Security Data Lake Sidecar on Windows either interactively or in silent mode.
Interactive installation
Run the installer:
graylog_sidecar_installer_1.5.1-1.exe
Follow the steps of the installation wizard:
Enter a name for your Sidecar instance.
Provide your server API token.
Select Install to complete the setup.
After installation, you can edit the configuration file located at:
C:\Program Files\Security Data Lake\sidecar\sidecar.ymlMost configuration parameters use default values. The only parameters that must be configured manually are server_url and server_api_token.
Silent Installation
To run the installer in silent mode, use the following command:
graylog_sidecar_installer_1.5.1-1.exe /S -SERVERURL=http://<your Security Data Lake IP or DNS name>/api -APITOKEN=<your_api_token>The Windows installer also supports additional options in silent mode, such as:
-TAGS=["example","IIS"] -NODENAME=mynodename -NODEID=1234 -SENDSTATUS=false -TLSSKIPVERIFY=true -UPDATEINTERVAL=10s
Adding a Winlogbeat collector configuration
Follow these steps to add a Winlogbeat collector configuration to your Sidecar and begin ingesting Windows event logs.
In Security Data Lake, go to System > Sidecars, and identify your Windows device.
Under your Windows Sidecar, select the Winlogbeat collector.
From the Configure drop-down menu on the right, select the
windows_sidecarconfiguration you created earlier.Open the Process drop-down menu and select the configuration you want to apply.
Select Start to begin data collection.
Once started, your Security Data Lake instance will begin receiving Windows event logs from the configured machine.
You can now continue to Getting Started with Sidecar for more details on managing and monitoring collectors.
The following steps must be taken in the remote machine from where you want to collect log data to prepare it for sending data to Security Data Lake
Once Security Data Lake Sidecar is installed, the next step is to install the Sidecar collectors. Collectors are the agents responsible for gathering log data from your system and forwarding it to Security Data Lake. Examples include Filebeat, Winlogbeat, and NXLog.
You can install one or more collectors depending on your environment and data sources. After installation, Sidecar will automatically detect the available collectors and manage their configurations through the Security Data Lake web interface.
sidecar.yml configuration reference
Below is a list of parameters used in the sidecar.yml configuration file for your reference:
Parameter | Description |
|---|---|
| Specifies the base URL for the Security Data Lake REST API. The path must include Example: |
| API token used for authenticating with the Security Data Lake REST API. This parameter is required. Example: |
| Unique identifier for the Sidecar instance. Can be a file URI or a literal string. Example file path: Example ID string: Default: |
| Display name for the Sidecar instance in the web interface. Defaults to the system hostname if not set. |
| Specifies how often (in seconds) the Sidecar fetches updated configurations from the server. Default: Sidecars that regularly update are marked as “active.” To configure the inactivity threshold, go to System > Configuration > Sidecars System in the web interface. |
| Determines whether TLS certificate validation is skipped for HTTPS connections. Default: |
| Controls whether the Sidecar reports detailed status, including collector states, metrics, and log listings. Disable this option to reduce load on the server. Default: |
| Defines directories that will be listed on the host status page. Can include one or multiple paths. Example: Default: |
| Directory used by Sidecar to store local cache data. Default: |
| Directory where generated collector configuration files are stored. Default: |
| Directory where the Sidecar writes its own log files. Default: |
| Maximum size of a log file before rotation occurs. Default: |
| Number of old rotated log files to retain. |
| Specifies which collector executables the Sidecar is allowed to run. Leave empty to disable this restriction. Default: |
| Defines a list of configuration tags. The Sidecar retrieves and merges all server configurations that match these tags. |
Sidecar collectors
A collector is a lightweight agent installed on a host system that gathers and forwards log data to Security Data Lake. It’s responsible for collecting logs from local files, system logs, or applications, then sending them securely to a Security Data Lake server or input endpoint for central analysis.
Collectors are typically managed through Sidecar, which allows administrators to configure, deploy, and monitor multiple collectors from a single interface. This setup helps ensure consistent log collection across different servers or environments and reduces the need for manual configuration on each host.
There are two types of collectors:
Default collectors (using ready-to-use configurations)
Default collectors
Several default collector configurations are included with Sidecar. These ready-to-use configurations are automatically assigned after installation and immediately begin collecting data such as event logs and audit framework information. The following collectors are currently available:
Operating System | Collectors |
|---|---|
Linux x86/x86_64 | Filebeat, Auditbeat |
Windows x86/x86_64 | Filebeat, Winlogbeat |
To view the collectors that will be running by default follow these steps:
Go to System > Sidecars.
Select the Configuration tab.
Editing default collectors
You can modify any Sidecar configuration to suit your needs. However, ensure that the variable ${user.graylog_host} uses the same server properties defined in the server.conf file (for example, http_bind_address or http_external_uri).
If you remove the default tag under Configuration Assignment Tags, the configuration will no longer be automatically applied to the Sidecar.
Custom collectors
Security Data Lake provides default collector configurations for Filebeat, Winlogbeat, and Auditbeat, however you can nstall and configure additional collectors to use with your Sidecar.
The following section demonstrates how to install NXLog as an example, but you can install and integrate other collectors as needed.
Because Sidecar allows you to define your own collector backends, you can also deploy tools such as Sysmon, auditd, or Packetbeat for specific data collection requirements.
Installing NXLog
To install NXLog on Ubuntu systems, follow these steps:
Download and install the NXLog package.
Stop any running NXLog instances and remove the default system service configuration.
Because the Sidecar manages the NXLog service (including starting and stopping it), you must disable the existing NXLog service before Sidecar can take control:
sudo systemctl stop nxlog sudo update-rc.d -f nxlog remove sudo gpasswd -a nxlog adm sudo chown -R nxlog.nxlog /var/spool/nxlog
Note
This step is required because the Sidecar manages the NXLog service, including starting and stopping it.
These steps ensure that the Sidecar has full control over the NXLog service and can manage its lifecycle correctly.
To install NXLog on Red Hat–based systems, follow these steps:
Download and install the NXLog package.
Stop all running instances of NXLog and unconfigure the default system service.
Because the Sidecar manages the NXLog service (including starting and stopping it), you must disable the existing NXLog service before Sidecar can take control
sudo service nxlog stop sudo chkconfig --del nxlog sudo gpasswd -a nxlog root sudo chown -R nxlog.nxlog /var/spool/nxlog
To install NXLog on Windows systems, follow these steps:
Download and install the NXLog package.
Deactivate the system service.
Only the NXLog binaries are required for use with the Sidecar. Run the following command:
"C:\Program Files\nxlog\nxlog" -u
Tip
When using PowerShell, prefix the command with &.
Ingest application data
Application data can be received through various methods, depending on the technology stack and logging framework in use. To illustrate how a configuration might work in practice, this article demonstrates how to configure a Ruby on Rails application to send logs to Security Data Lake.
Integrating Ruby on rails with Security Data Lake configuring your application to send structured log data using Lograge and the GELF gem.
Log all requests and logger calls
The recommended approach for sending structured information (such as HTTP return codes, controller names, and actions) about every request, as well as explicit Rails.logger calls, is to use Lograge with GELF output.
Lograge consolidates each request into a single structured log entry (instead of multiple lines generated by the default Rails logger) and supports Security Data Lake output starting from version 0.2.0.
To set up Lograge and GELF to send request and log data to Security Data Lake, follow these steps:
Add Lograge and the GELF gem to your Gemfile:
gem "gelf" gem "lograge"
Configure both in your Rails application. We recommend adding this configuration in
config/environments/production.rb:config.lograge.enabled = true config.lograge.formatter = Lograge::Formatters::Graylog2.new config.logger = GELF::Logger.new( "graylog.example.org", 12201, "WAN", { host: "hostname-of-this-app", facility: "heroku" } )
This configuration ensures that both request logs and explicit logger calls (for example, Rails.logger.error "Something went wrong") are sent to Security Data Lake.
Log Only Explicit Logger Calls
If you want to log only explicit logger calls rather than all requests, configure only the GELF logger:
Add the GELF gem to your Gemfile:
gem "gelf"
Configure the GELF logger in your Rails application. We recommend adding this configuration in
config/environments/production.rb:config.logger = GELF::Logger.new( "graylog.example.org", 12201, "WAN", { host: "hostname-of-this-app", facility: "heroku" } )
Note
This setup will send only the messages explicitly written through Rails.logger to Security Data Lake.
Heroku
Heroku injects its own logger (rails_log_stdout) by default, which overrides custom logger configurations. To enable custom logging with GELF on Heroku, create a dummy file to make Heroku detect a pre-existing logger:
$ touch vendor/plugins/rails_log_stdout/heroku_fix
This workaround prevents Heroku from replacing your custom logger, allowing Rails to forward logs to Security Data Lake via GELF.
Ingest from files
Log files come in many different formats, which can make them difficult to parse and normalize.
For this reason, Security Data Lake does not collect log files directly. Instead, it relies on a variety of collectors and agents that are purpose-built for gathering logs from different systems and applications.
Collectors can be:
Configured and managed through existing configuration management tools in your environment.
Centrally managed and monitored through the Sidecar (after it has been installed and configured).
Set up manually, if preferred, for standalone deployments.
Additionally, any program that supports the GELF or Syslog protocols can be used to send logs to Security Data Lake.
One of the most effective and widely recommended collectors for both Windows and Linux systems is Filebeat. Filebeat is designed to read log files and forward them to a central destination. To send log data to Security Data Lake, configure Filebeat to use the Logstash output, which connects to a Beats input in Security Data Lake.
Example Filebeat configuration for Linux
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
filebeat.inputs:
- type: filestream
id: var-log
paths:
- /var/log/*.log
output.logstash:
hosts: ["graylog:5044"]
path:
data: /var/lib/graylog-sidecar/collectors/filebeat/data
logs: /var/lib/graylog-sidecar/collectors/filebeat/log
Example Filebeat configuration for Windows
fields_under_root: true
fields.collector_node_id: ${sidecar.nodeName}
fields.gl2_source_collector: ${sidecar.nodeId}
output.logstash:
hosts: ["graylog:5044"]
path:
data: C:\Program Files\Graylog\sidecar\cache\filebeat\data
logs: C:\Program Files\Graylog\sidecar\logs
tags:
- windows
filebeat.inputs:
- type: filestream
id: c-log-log
paths:
- C:\logs\log.log
Installing Sidecar
To install and make the necessary configurations for Sidecar, follow these steps:
1. Set up the sidecar in Security Data Lake
The following steps must be taken in Security Data Lake in order to create and configure a sidecar:
1.1 Set up an input
Before installing Sidecar, create a new Beats input in order to receive data from the Beats collector, and configure it to receive Sidecar logs on port 5044.
1.2 Create an API token
Select the user icon on the upper right side of the screen and select Edit profile.
Select the Edit tokens button on the upper right side of the screen.
Enter the token details and select Create Token.
Save the API server token in a safe yet accessible location in case you need to retrieve it.
1.3 Download Sidecar
You can find .deb, .rpm, .msi, .exe, .arm, .tar and .gz packages in our package repository. Please follow the version matrix to select the correct package and download it from our GitHub page.
2. Prepare your local environment
You can install Security Data Lake Sidecar on Windows either interactively or in silent mode.
Interactive installation
Run the installer:
graylog_sidecar_installer_1.5.1-1.exe
Follow the steps of the installation wizard:
Enter a name for your Sidecar instance.
Provide your server API token.
Select Install to complete the setup.
After installation, you can edit the configuration file located at:
C:\Program Files\Security Data Lake\sidecar\sidecar.ymlMost configuration parameters use default values. The only parameters that must be configured manually are server_url and server_api_token.
Silent Installation
To run the installer in silent mode, use the following command:
graylog_sidecar_installer_1.5.1-1.exe /S -SERVERURL=http://<your Security Data Lake IP or DNS name>/api -APITOKEN=<your_api_token>The Windows installer also supports additional options in silent mode, such as:
-TAGS=["example","IIS"] -NODENAME=mynodename -NODEID=1234 -SENDSTATUS=false -TLSSKIPVERIFY=true -UPDATEINTERVAL=10s
Adding a Winlogbeat collector configuration
Follow these steps to add a Winlogbeat collector configuration to your Sidecar and begin ingesting Windows event logs.
In Security Data Lake, go to System > Sidecars, and identify your Windows device.
Under your Windows Sidecar, select the Winlogbeat collector.
From the Configure drop-down menu on the right, select the
windows_sidecarconfiguration you created earlier.Open the Process drop-down menu and select the configuration you want to apply.
Select Start to begin data collection.
Once started, your Security Data Lake instance will begin receiving Windows event logs from the configured machine.
You can now continue to Getting Started with Sidecar for more details on managing and monitoring collectors.
The following steps must be taken in the remote machine from where you want to collect log data to prepare it for sending data to Security Data Lake
Once Security Data Lake Sidecar is installed, the next step is to install the Sidecar collectors. Collectors are the agents responsible for gathering log data from your system and forwarding it to Security Data Lake. Examples include Filebeat, Winlogbeat, and NXLog.
You can install one or more collectors depending on your environment and data sources. After installation, Sidecar will automatically detect the available collectors and manage their configurations through the Security Data Lake web interface.
sidecar.yml configuration reference
Below is a list of parameters used in the sidecar.yml configuration file for your reference:
Parameter | Description |
|---|---|
| Specifies the base URL for the Security Data Lake REST API. The path must include Example: |
| API token used for authenticating with the Security Data Lake REST API. This parameter is required. Example: |
| Unique identifier for the Sidecar instance. Can be a file URI or a literal string. Example file path: Example ID string: Default: |
| Display name for the Sidecar instance in the web interface. Defaults to the system hostname if not set. |
| Specifies how often (in seconds) the Sidecar fetches updated configurations from the server. Default: Sidecars that regularly update are marked as “active.” To configure the inactivity threshold, go to System > Configuration > Sidecars System in the web interface. |
| Determines whether TLS certificate validation is skipped for HTTPS connections. Default: |
| Controls whether the Sidecar reports detailed status, including collector states, metrics, and log listings. Disable this option to reduce load on the server. Default: |
| Defines directories that will be listed on the host status page. Can include one or multiple paths. Example: Default: |
| Directory used by Sidecar to store local cache data. Default: |
| Directory where generated collector configuration files are stored. Default: |
| Directory where the Sidecar writes its own log files. Default: |
| Maximum size of a log file before rotation occurs. Default: |
| Number of old rotated log files to retain. |
| Specifies which collector executables the Sidecar is allowed to run. Leave empty to disable this restriction. Default: |
| Defines a list of configuration tags. The Sidecar retrieves and merges all server configurations that match these tags. |
Managing sidecars
You can manage and monitor your Sidecar instances from System > Configurations > Sidecars.
Once a sidecar is installed and running, it appears on this page along with its current status. Each Sidecar instance reports its health and configuration details back to Security Data Lake, allowing you to verify that collectors are connected and functioning correctly.
Note
Obsolete or disconnected Sidecars are automatically removed from the list after 14 days of inactivity. You can adjust this default retention period from the Sidecars page.
To view detailed information about a specific sidecar, click its entry under Node details. This opens a dedicated status page showing configuration parameters, active collectors, and other runtime information reported by that sidecar.
Failure tracking
In large deployments with many Sidecars, identifying the cause of individual collector failures can be time-consuming. To simplify troubleshooting, Security Data Lake includes a Failure Tracking page within the Sidecars interface.
This searchable and sortable page lists each collector along with its name, status, error message, and detailed (verbose) error message, allowing administrators to quickly identify and address issues across multiple sidecars.
Create and edit variables
Configuration variables can contain arbitrary strings such as the IP address of your Security Data Lake server or the port number of an input. These variables can be reused across multiple collector configurations, helping to avoid duplication and simplify overall management.
To create a configuration variable, follow these steps:
Go to the System > Sidecars page and select the Configuration tab.
Under the Collector Configuration Reference section, select Create Variable.
Enter the required details and select Save.
To update a variable, select the Edit button next to the variable you want to update, make the required modifications, and select Save.
Example
In the following example, we replace the hardcoded IP address and port in a Beats input with a variable named ${user.graylog_host}.
This approach allows you to easily update connection details or environment-specific parameters without modifying multiple configuration files.
Securing communication
To secure the communication between the collector and Security Data Lake, enable TLS in your input configuration. For example, when configuring a Beats input, select Enable TLS. Security Data Lake will then generate a self-signed certificate for the input, ensuring encrypted data transfer between the collector and the Security Data Lake server.
Assigning tags
You can use tags to control how collector configurations are assigned to hosts. Tags determine which configurations each endpoint receives.
For example, you can create a configuration to collect Apache access logs and assign it the tag apache. Any Sidecar instance running on a web server with the same apache tag will automatically fetch and apply this configuration, allowing it to collect and forward web access logs to Security Data Lake.
This is the relevant section of a typical Sidecar configuration YAML file found on an endpoint:
# A list of tags to assign to this Sidecar. Collector configuration matching any of these tags will automatically be # applied to the Sidecar. # Example: # tags: # - apache-logs # - dns-logs
Once a system tag is applied to an endpoint, it automatically begins collecting logs and forwarding them to Security Data Lake.
This feature simplifies configuration management by automatically assigning collector configurations to new clients whose YAML files include the corresponding tags.
Tags can be defined on any endpoint, such as Windows Servers, Windows Workstations, or Linux systems, and can be used across different collectors including Winlogbeat, Filebeat, Metricbeat, and NXLog.
Sidecar also supports assigning multiple collector configurations to the same endpoint. For instance, a Windows client could have configurations for both winlogbeat and sysmon tags applied simultaneously.
Tip
You can also use snippets to include additional collector configuration details that may not be easily defined through the system interface.
Snippets can contain entire configurations (such as an NXLog setup) or extend existing input and output definitions.
All snippets are directly included in the generated collector configuration, regardless of whether their inputs or outputs are already defined.
Troubleshooting
The Sidecar writes its log files to the directory specified in the log_path configuration parameter. Each backend (collector) has its own log file where you can check for issues such as file permission errors or log transmission problems.
The Sidecar itself writes to a file named sidecar.log, which contains general operational messages. Errors such as failed connection to the Graylog API are typically recorded in this file.
You can also start the sidecar in the foreground with debug output enabled to monitor its activity in real time:
graylog-sidecar -debug
This command is useful for troubleshooting connection issues, configuration errors, or collector startup problems.
Uninstalling
The process to uninstall a sidecar differes based on the operaing system where it was installed.
Linux
On Linux systems, you can simply uninstall the package using your package manager. For example:
sudo apt-get remove graylog-sidecar
or
sudo yum remove graylog-sidecar
Windows
On Windows systems, stop and uninstall the Sidecar service by running the following commands:
"C:\Program Files\Graylog\graylog-sidecar.exe" -service stop "C:\Program Files\Graylog\graylog-sidecar.exe" -service uninstall
Note
When using PowerShell, prefix each command with &.
After running these commands, you can safely remove the sidecar directory if no longer needed.