Illuminate
Illuminate is a collection of content comprising pipelines, parsing rules, lookup tables, and more. This content enables various event logs to be processed using a standard methodology, leveraging the Security Data Lake Information Model (GIM) schema, to make searching and analyzing common log sources more efficient.
By enriching and normalizing your log data so that the username or IP address is always in the same field, searching for logs becomes much easier and faster. Additionally, you can create more universal dashboards that will work across any data type (as they are mapped to the schema) and regardless of which firewall connection(s) you may have.
To accomplish this, Illuminate works by ingesting logs, sorting them, and processing them. The sorting process occurs on the original log message as it comes into Security Data Lake, so how the log data is sent affects whether Illuminate will pick up and process the message correctly. For example, some devices can send logs in multiple formats, like syslog-compliant messages, BSD-compliant messages, and free-form messages, but a specific form is still required to make parsing rules work. For specifics on system versions, specific formats, or settings, please refer to the individual content pack documentation.
Illuminate Architecture
Illuminate is designed with a processing hierarchy that breaks up processing into three key areas.
Processing Packs
Individual packs for parsing and processing logs:
Identify logs from the collection of all logs received by a Security Data Lake instance.
Perform parsing and/or normalization and apply the Security Data Lake schema.
Identify specific event message types and assign type codes.
Enrich event messages.
Illuminate Core
The Illuminate core processor:
Provides common processing logic to event log messages.
Identifies common private or reserved IP addresses.
Enriches event messages that have been assigned event type codes with category, subcategory, and event type data.
Optionally provides Geolocation and ASN enrichment to eligible messages using either MaxMind or IPinfo databases.
Optionally provides GIM enforcement, which will ensure events have required fields for categories and subcategories, and identifies potential event categorization issues.
Spotlight Packs
Individual content packs that operate on parsed and processed logs and provide:
Dashboards to visualize processed logs
Sigma Rules and Event Definitions to detect unusual activity within logs
Content Hub
The Illuminate Content Hub is a centralized location where you can browse, enable, and manage Illuminate content packs. It provides access to both newly released and existing content packs, highlighting new or updated ones automatically at the top of the list. The hub offers search and filtering options to help users quickly find relevant content based on status, type, tags, or keywords. Administrators can install or uninstall packs through an intuitive interface that also manages dependencies between packs, helping to streamline the deployment and maintenance of Illuminate content within the Security Data Lake environment.
Performance Impact of Illuminate
Illuminate log processing allows for items like alert rules, anomaly detectors, and dashboards to work across various log sources. With Illuminate processing log data, you do not have to create separate rules like "Windows Logon Brute Force" and "Linux Logon Brute Force." You only need to create one rule to cover them both.
As with all processing in Security Data Lake, there will be performance implications as each log message goes through the process described above. Gates or sorting rules are the first set evaluated to limit logs to be processed further, shortening the number of rules each message touches.
Processing rules can range from simple key-value extractors, which perform very quickly, to complex regex statements or GROK patterns. Each rule can have a different performance impact, and each rule can perform differently based on the log type, so finding an actual cost per rule is subjective to an environment.
Indexes and Shards
Illuminate does not use unique values for index and shard settings; instead, it currently takes the system's default for those settings. After the indexes and streams are created, you can adjust the default settings if a replica is needed or for more or fewer shards.
Illuminate sets up indexes with a retention time based on common practices and standards. These settings allow the dashboards, anomaly rules, and alert rules to have enough online data to operate. Adjustments to these settings can be made, but note that any previously saved settings can be affected.