Security Data Lake Information Model Schema
This guide is a reference for the schema used in Illuminate. We will keep this updated as changes are made, but if you feel a change should be included, please open a GitHub issue here.
How To Use this Guide
Welcome to the Security Data Lake Information Model (GIM) schema. The schema is broken down into sections, like “File” or “Destination,” with all metadata fields below.
Illuminate will utilize this schema for all the content we create. This guide will the be official reference and should be used if you are creating your own content and want it to match our content.
For example, if you take any log source and put the user name into a field called user_name, then any dashboard or alert created with Illuminate will work as well.
Security Data Lake Template
When logs are processed, data from the logs are inserted into OpenSearch as “keywords,” meaning they are not modified in any way and stored as-is. This means the following data points are unique:
Administrator or administrator.
If you are searching in the Security Data Lake UI, you would have to search for both of these terms or know exactly which one to search for. Fields like user_name make sense to be able to search without worrying about the case of the word.
In order to ensure these options are accounted for, a custom analyzer has been included in the Security Data Lake Schema template called “loweronly.” Fields normalized with “loweronly” will be converted to lowercase before the data is indexed, and search query strings for these fields will be converted to lowercase when ran as well. Pages in the schema will list these fields as keyword/loweronly for reference.
Information Model Entities
The following metadata field categories comprehensively represent the Security Data Lake Information Model Schema:
Alert Fields
For messages that are an alert, such as an IDS alert
For Vendor alert severity levels, the
vendor_event_severity* fields will be used
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 2020.14092348 | keyword | Version or identification value that indicates the version a collection of signatures (A/V etc.) is in use |
| malware, trojan, ransomeware | keyword | Future: How do we define this field considering vendors will have their own categories? Or is that not a concern? Possibly movie this to derived fields & set only allowed values |
| malware.exe, http://badsite | keyword | A filename, URL, packet snippet or other artifact that is related to the event that caused the alert to be generated. |
| 0, 1, 2 | byte | Numeric value representing the type of action taken in response to an alert/threat. 0 = Nothing (allowed, ignored), 1 = prevent (blocked, quarantined), 2 = eradicate (deleted). This allows the use of numeric functions to detect unblocked threats where products may log multiple events for a single threat. |
| keyword | Vendor-provided Alert text description | |
| keyword | Vendor specific unique identifier for alert signature (e.g., 1:1905345:5 for Snort signatures.) |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| critical, high, medium, low, informational | keyword | Severity of Alert |
| 1-5 | byte | Numeric representation of the severity rating of the source message: 1 = informational, 2 = low, 3 = medium, 4 = high, 5 = critical |
Application Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| Facebook, SQL, windows_rdp | keyword ( | Name of the application, this can be a layer 7 application name for network traffic, the name of an authenticating service/program for authentication, etc. |
| keyword | Amount of time Applications Take to give response to a request | |
| keyword | For Single Sign-On (SSO) events this is the method used to access the application | |
| keyword | For SSO events this is the name of the application being accessed |
Associated Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| keyword | TBD: Not sure if this is useful | |
| 6f9efb466e043b9f3635827ce446e13c | keyword | All associated md5, sha1, sha256, sha512, imp hashes from a log message |
| 10.1.2.3,corpdc01,corpdc01.corpdomain.local | keyword | FUTURE: copy of any identifying host information - IP, Hostname, etc. from a log message, not implmented yet. |
| 10.1.2.3,fe80:5cc3:11:4::2c | ip | Associated IP addresses for a log message |
| a0:b4:44:01:a9:d1 | keyword | Associated MAC addresses for a log message, colon-delimited and lower case |
| 0xa72c | keyword | Associated session IDs for a log message |
| 999,S-1-5-18 | keyword | This will be a field that maps to all user ID values (uids, SIDs, etc.) that are associated with a user context. This can/may eventually be populated from the user framework. |
| administrator,[email protected] | keyword ( | Any associated/alternate user ID or email, can be a set of multiple values. |
Autonomous System (AS) Sub-Fields
Autonomous System (AS) fields for the internet (nested as needed)
AS fields have data referencing organization information related to an IP address
AS fields apply to source, destination, and host entities
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
… | 15169 | keyword | Unique number. ASN identify each network on internet |
… | Bitdefender | keyword | Organization Name |
… | keyword | ISP associated with IP address | |
… | keyword | Domain associated with IP address |
Container Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| keyword | Unique container ID | |
| keyword | Container Name | |
| keyword | Container's Namespace it is running in |
Destination Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| facebook, twitter | keyword | Describes the target application |
| 203948 | long | Network bytes sent by destination to the source. Some sources may present this as source bytes received, bytes received, or similar. |
| iPad | keyword | Device Model Name |
| Apple, ASUS | keyword | Device Vendor Name |
| corp.local | keyword (normalized:loweronly) | Destination domain context |
| corpdc01 | keyword (normalized:loweronly) | |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | IPv4 and IPv6 addresses |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | translated IP address assigned by a network device performing the NAT function |
| 2356 | integer | translated network port assigned by a network device performing the NAT function |
| IOS, Android | keyword | Operating System Name |
| IOS 10.0 | keyword | Version number of Operating System |
| 73458324 | long | Number of packets delivered to the destination endpoint |
| 80, 443 | integer | Service port associated with a network connection port, 0-65535 |
| ssh, ftp | keyword | The IANA-registered service name associated with the network application. Illuminate Core will use this value to define |
| us-east-1 | keyword | Name of region source device is located in |
| 09VX93DD | keyword | Identifying value for the destination such as a serial number |
| keyword | Destination device information such as model number | |
| keyword | Virtual system name (not to be confused with the hostname) | |
| 1f5398c7-4d84-4499-84ee-d5e9246c52f8 | keyword | Destination virtual system UUID |
| internal | keyword | Network zone for the destination |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| See: as_* fields | ||
| keyword | Future: from entity mapping | |
| See: geo_* fields | ||
| Chicago, US, Datacenter 01, Bismark - Finance | keyword | Field is derived either from an internal enterprise network definition or the Geo location fields if availble |
| a0:b4:44:01:a9:d1 | keyword | MAC address of host, colon-delimited and lower case |
| critical, high, medium, low | keyword | Future: from entity mapping |
| 4-Jan | byte | Numeric value representing the priority of the destination device, 1 = low, 2 = medium, 3 = high, 4 = critical |
| IPv4, IPv6, hostname,fqdn | keyword (normalized:loweronly) | Automatically mapped from the following fields: |
Email Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| attachment.exe | array | The file name(s) of an attachment. |
| 1024 | long | The size in bytes of the attachments. |
| keyword | The email address of BCC recipient/destination. | |
| keyword | The email address of CC recipient/destination. | |
| keyword | The | |
| inbound, outbound, lateral | keyword | Indicates the direction of the observed email flow. Must be either inbound, outbound or lateral, this should be mapped to these values if vendors provide network direction differently. |
| keyword | Per RFC 5322, specifies the address responsible for the actual transmission/sender of the message. | |
| keyword | The globally-unique message identifier. | |
| keyword | The email authentication header. | |
| keyword | The address that replies should be delivered to based on the value in the RFC 5322 | |
| 234 | long | The size of an email in bytes. |
| RE: FWD: Testing | keyword | The email subject. |
| keyword | The email address of recipient/destination. | |
| 123456789A | keyword | The email unique identifier internally used by an email software to track a message. |
| 192.168.2.3 | array | The X-Originating-IP header identifying the email's originating IP address(es). |
| spambot | keyword | Tool that created and sent the email. |
Event Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| blocked, allowed, | keyword | Action that was described in a log such as a firewall log or an antivirus agent log |
| 4624, 1 | long | Numeric event defined by the vendor representing the source message type, e.g. EventCode/Event ID for Microsoft. This field is treated as a numeric value in order to support ranged queries. Any leading 0 values will be removed |
| 2020-02-20 08:23:15.102, 1602080607 | date | Date/time that the event actually occured or when the original event message was created |
| 10293874 | long | Length of time, in seconds, for the event being described |
| 2021-03-26T11:25:13.113 | date | Date/time that event described in the log message had concluded, usually associated with an event that has a duration. |
| 0xC00008 | keyword | Vendor-provided error code associated with the current message |
| ERROR_ACCESS_DENIED, Not Found | keyword | Description of error associated with the current message |
| 0023425, 90EF8 | keyword | Vendor-provided identifier representing a message type. This is similar to |
| security, auth.log | keyword | Reference to log, such as 'Security', 'auth.log', etc. - this differs from |
| /var/log/syslog | keyword | Full path of log file source |
| SERVER01.server01.corp.internal | keyword/loweronly | Hostname or FQDN of a system such as an IDS or IPS that generates an message (such as an alert) based on inspection of a thing, such as network traffic. |
| 234cd78sc | keyword | Unique ID of the Observer Device, Serial Number, etc |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | IP address of the event observer |
| keyword | Unique identifier (such as a serial number or asset ID) associated with the event observer | |
| 2020-02-20 08:00:00, 1602080607 | date | Date/time that the event was received by the reporting host. Normally applicable to logs relayed by a centralized log server. |
| 5, 3, 9185 | long | Count of times a message has been repeated |
| SERVER01.server01.corp.internal | keyword | Hostname or IP for system that delivered the message to Security Data Lake - a WEC server, syslog collector, etc. |
| LAPTOP01,laptop01.corp.internal | keyword | Hostname or IP of source system that generated the event |
| keyword | API version of source where logs are collected via API | |
| windows, linux, okta | keyword | System responsible for generating the event, e.g. “windows,” “okta,” etc. |
| 2020-02-20 08:00:00, 1602080607 | date | Beginning time of an event described in a log message, usually associated with an event that has a duration. |
| 1123523564, 0122e2b3-9923-11ea-ab51-061b68b4ca16 | keyword | Unique identification associated with a single event/message (e.g. “record number” from Windows event logs, a Security Data Lake message ID) |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| success, failure | keyword | The outcome (success/failure) of the action described by |
| critical, high, medium, low, informational | keyword | This will be added by Illuminate Core if only the |
| 1-5 | byte | Numeric representation of the severity rating of the source message: 1 = informational, 2 = low, 3 = medium, 4 = high, 5 = critical. This will be added by Illuminate core when only |
File Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| Microsoft | keyword | Company name associated with a file taken from the file metadata |
| date | Compiled date/time that a binary file was compiled | |
| keyword | Contents of a file | |
| WMI | keyword | Description of file |
| true, false | boolean | Flag indicating if file is executable |
| 1 | boolean | Flag indicating if file has been digitally signed |
| file.zip, file.exe, file | keyword | File name, not including path |
| C:\\temp\\file.exe | keyword | Full path and file name |
| keyword | Product name the file was shipped with | |
| keyword | Product version the file was shipped with | |
| valid | keyword | Status of file signature |
| Microsoft Windows | keyword | Title of file signer |
| 23894713 | long | File size in bytes |
| gzip compressed data, application/pdf | keyword | Description of file contents |
| 10.0.14393.4169 (rs1_release.210107-1130) | keyword | Version of file |
Geolocation Sub-Fields
Geo fields have data referencing location of event/host/ip
Geo fields apply to source, destination, and host entities
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
… | Hamburg, Houston | keyword | City Name |
… | America | keyword | Continent Name |
… | US, DE, CA | keyword | Country ISO Alpha-2 code |
… | USA, Canada | keyword | Country Name |
… | 34.1186,-118.3004 | keyword | Latitude, Longitude Coordinate |
… | Hamburg, DE | keyword | Location Name, can be derived by combining other values |
… | Hamburg | keyword | State name |
GIM Fields
The GIM fields are meta fields used by Security Data Lake to assign a standard category, subcategory, and type to messages.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 100000 | long | This field is assigned during the normalization process. Based on this field messages will have category, subcategory, and type fields applied. |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| process, audit, authentication | keyword | The category the associated log message falls under. Message categories are groupings of related messages that often have common fields. |
| endpoint, protocol | keyword | This is an optional field that is used for related categories. For example, the process and service categories are part of the Endpoint |
| network connection | keyword | A description of the event described in the associated log message. |
| credential validation, process | keyword | A secondary grouping of events under a category where individual events share many common characteristics. |
Hash Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 4c583e00d47108f809282d5d595f5fb0 | keyword | MD5 hash value |
| 5d4d04eff6aba8467ebd26c43008ab028203be35 | keyword | SHA1 hash value |
| keyword | SHA256 hash value | |
| keyword | SHA512 hash value | |
| 0c2803c4e9a2102c4dc65963dad36cdf | keyword | IMP hash value |
Host Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| \\Device\\HarddiskVolume2 | keyword | Identifier for a device (drive network adapter) connected to a system |
| corpdc01, corpdc01.local, lab01.corpdomain.com | keyword (normalized:loweronly) | NetBIOS or dns hostname |
| keyword | Host unique identifier (e.g. SID for Microsoft) | |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | IPv4 and IPv6 addresses |
| fe80:5cc3:11:4::2c | ip | IPv6 addresses |
| 02:a1:f9:c2:d5:04 | keyword | MAC address of host, colon-delimited and lower case |
| 127.0.0.1, corpdc01, corpdc01.local, lab01.corpdomain.com | keyword (normalized:loweronly) | Mapped from |
| us-east-1 | keyword | Name of region source device is located in |
| keyword | Operating sytem version of host | |
| keyword/loweronly | For firewalls that operate as partitioned services this is the name of the logical device | |
| keyword | For firewalls that operate as partitioned services this is the ID value of the logical device | |
| keyword | Unique identifier such as a UUID value representing a virtual host | |
| keyword | Virtual system name (not to be confused with the hostname) |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| See: as_* fields | ||
| keyword | Future: from entity mapping | |
| See: geo_* fields | ||
| Chicago, US, Datacenter 01, Bismark - Finance | keyword | Field is derived either from an internal enterprise network definition or the Geo location fields if available |
| critical, high, medium, low | keyword | Future: from entity mapping |
| 2 | byte | Numeric value representing the priority of the host device, 1 = low, 2 = medium, 3 = high, 4 = critical |
| IPv4,IPv6, hostname,fqdn | keyword (normalized:loweronly) | Automatically mapped from the following fields: |
| keyword | Machine “type” |
HTTP Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| keyword | Layer 7 application name | |
| 29347485 | Long | Sum of request + response bytes |
| application/octet-stream | keyword | Mime type of http content |
| keyword | ||
| Host: wwww.mycorp.local | keyword | host: … header from request, if present |
| http://mycorp.local/ | keyword | “referer” header value if present |
| 239478 | long | Size of request |
| GET, POST | keyword | |
| /path/to/resource?option=test | keyword | Need to review field length/truncation at 8192 characters (consider utf-8). Some may consider the path not to include the “query” (text after the last “/”), but this value may include it. |
| 498274 | long | Size of response |
| OK, Moved Permanently | keyword | |
| 200, 404, 500 | integer | Numeric server response code |
| https://www.graylog.org, https://www.graylog.org/blog, https://www.mycorp.local/workspaces/team#posts | keyword | Full request string; Need to review field length/truncation at 8192 characters (consider utf-8) |
| Suspicious, Games | keyword | Categorization of associated web site/URL |
| Default.htm | keyword | The target of the request. For Example: http://www.test.com/test.jsp?hello=y the URI stem is /test.jsp |
| hello=y | keyword | The query the client was trying to perform. Example http://www.test.com/test.jsp?hello=y the query is hello=y |
| Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:74.0) Gecko/20100101 Firefox/74.0) | keyword | User Agent string |
| Firefox | keyword | Attempted identification of the browser client usually based on user agent analysis |
| Windows 10 | keyword | Operating System of User Agent |
| 1.0, 1.1, 2.0 | keyword | HTTP version |
| X-Forwarded-For: 10.1.2.3 | keyword | HTTP x-forwarded-for header value. Future: May map as IP, need to account for different ways this is presented. |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| ** TBD | Need to review best analyzer configuration for HTTP paths / consider truncation | |
| ftp://ftp01.server.internal/file.tar.gz, https://www.graylog.org, https://www.graylog.org/blog | text/standard | Optionally copied when a URL must be tokenized. Future: will have to research best analyzer config / consider truncation |
| 9283 | long | String length of HTTP user agent |
| text/standard | This is a copy of the | |
| 54 | long | String length of original user agent |
Network Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| facebook, instagram | keyword/loweronly | Application name - Facebook, etc. |
| 71238 | long | Bytes transferred during a connection, may be calculated by summing bytes sent/received ( |
| DEPRECATED - use | Deprecated field reference | |
| DEPRECATED - use | Deprecated field reference | |
| keyword | ||
| 0:23:45 | keyword | Duration of time a network connection was established |
| CMdzit1AMNsmfAIiQc | keyword | Unique identifier value for a network connection |
| 71238 | long | Total bytes of the data payload |
| inbound, outbound, lateral | keyword | Indicates the direction of the observed network flow. Must be either inbound or outbound, this should be mapped to these values if vendors provide network direction differently. |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | Forwarded IP addresses associated with network events |
| 71238 | long | Total bytes of packet header information |
| 6, 17, 41 | integer | https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml |
| echo, time exceeded | keyword | https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml |
| TBD | Nested or encapsulated network data | |
| gi0/1 | keyword/loweronly | Name of interface traffic receiving traffic |
| gi0/1 | keyword/loweronly | Name of interface traffic sending traffic |
| 4, 6 | keyword | IPv4 or IPv6 |
| TBD | Logical or descriptive name for network context | |
| 71238 | long | Count of packets transferred during a connection, may be calculated by summing packets sent/received ( |
| DEPRECATED - use | Deprecated field reference | |
| DEPRECATED - use | Deprecated field reference | |
| ipv4, ipv6, icmp | keyword/loweronly | Protocol names, preferrably from the Keyword column in https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml |
| udp, tcp | keyword/loweronly | transport layer protocol of packet/connection |
| gre, ipsec | keyword/loweronly | tunnel type |
| 2093847 | long | time in seconds for tunnel duration |
| TBD - maybe not needed since | Optional field for specifying custom network type |
Policy Fields
Related to system/device policies for operating systems, firewalls, etc.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 6da61e4c-84a8-4136-900d-f86c09bb3774 | keyword | Unique identifier of a policy |
| keyword | ||
| admin-user-template | keyword | Name of a policy |
Privilege Fields
Privilege fields are fields related to operating system privilege assignments. These are typically either considered either roles or attributes that can be assigned to account groups or individual accounts.
Microsoft Windows is an example of an attribute-based privilege solution. While Windows privileges are often managed through group membership, security tokens are key to determining what actions an account is authorized to perform. These tokens are generated during the authentication process and include information about the account's privileges, such as SeDebugPrivilege, which allows an account to debug and interact with sensitive system processes. Each privilege in the token corresponds to specific rights that are granted to the account, enabling or restricting particular actions.
Microsoft Entra ID is an example of a role-based privilege solution. Entra ID provides a number of built-in roles and allows users to define custom roles, which are used to determine which actions Entra ID users can take.
The ..._category sub-fields are a common enrichment used on a per-source type basis add context to vendor-provided or custom privileges. The tag value elevated_privilege is the value to assign to the category field when a value in the privilege_name, privilege_assigned_name, or privilege_removed_name value is one which gives an account the ability to perform sensitive activities.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| elevated_privilege | keyword | Tag-type values which can be used to add supplemental metadata about the privilege in question. The value 'elevated_privilege' is assigned here to identify when an attribute or role listed in the privilege_id field indicates the privilege provides access to perform sensitive tasks on a system. |
| ffd52fa5-98dc-465c-991d-fc073eb59f8f | keyword | Identification of the privilege attribute or role, this is the field used by compliance content. |
| SeDebugPrivilege | keyword | A short descriptive name of the privilege, not all systems will generate this. |
| built_in | keyword | Tag-type values which can be used to add supplemental metadata about the privilege in question. The value 'elevated_privilege' is assigned here to identify when an attribute or role listed in the privilege_id field indicates the privilege provides access to perform sensitive tasks on a system. |
| c430b396-e693-46cc-96f3-db01bf8bb62a | keyword | Identification of the privilege attribute or role, this is the field used by compliance content. |
| Attack Simulation Administrator | keyword | A short descriptive name of the privilege, not all systems will generate this. |
| built_in | keyword | Tag-type values which can be used to add supplemental metadata about the privilege in question. The value 'elevated_privilege' is assigned here to identify when an attribute or role listed in the privilege_id field indicates the privilege provides access to perform sensitive tasks on a system. |
| c430b396-e693-46cc-96f3-db01bf8bb62a | keyword | Identification of the privilege attribute or role, this is the field used by compliance content. |
| SeLoadDriverPrivilege | keyword | A short descriptive name of the privilege, not all systems will generate this. |
Process Fields
Process is related to the execution of binaries
The
process_ namescan also be prefixed withtarget_… andparent_…, e.g.parent_process_id,target_process_name, etc.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| WMI Commandline Utility | keyword | Description of executed process |
| c:\\tmp\\runme.exe, /tmp/runme | keyword/loweronly | Full command line of executed process |
| 29347 | long | Length of |
| 2045,0x3e7 | keyword/loweronly | Process identifier associated with executed process |
| medium, high, trusted | keyword | Integrity level of executed process |
| c:\\tmp\\runme.exe, /tmp/runme | keyword/loweronly | Full command line of parent process |
| 2045,0x3e7 | keyword/loweronly | Process identifier associated with parent process |
| whoami, whoami.exe | keyword/loweronly | File name of parent process, excluding path |
| C:\\Windows\\system32\\whoami.exe, /usr/bin/whoami | keyword/loweronly | Full path of parent process |
| {73123815-5caa-4e39-90dc-d25d4013bf15} | keyword | GUID or unique identifier for parent process that is not the |
| whoami, whoami.exe | keyword/loweronly | File name of executed process, excluding path |
| C:\\Windows\\system32\\whoami.exe, /usr/bin/whoami | keyword/loweronly | Full path of executed process |
| 2045,0x3e7 | keyword | The process ID of the targeted process of some action that was taken against that process |
| whoami, whoami.exe | keyword | The name of the targeted process of some action that was taken against that process |
| C:\\Windows\\system32\\whoami.exe, /usr/bin/whoami | keyword | The full path and name of the targeted process of some action that was taken against that process |
| {73123815-5caa-4e39-90dc-d25d4013bf15} | keyword | The process unique identifier of the targeted process of some action that was taken against that running process |
| {73123815-5caa-4e39-90dc-d25d4013bf15} | keyword | GUID or unique identifier for executed process that is not the |
| C:\\Windows\\Temp | keyword | The current working directory that the process was called from |
Query Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| IN | keyword | Class of name query, usually IN for DNS |
| A, AAAA, MX, SRV | keyword | Record type being requested |
| 1, 3, 5 | keyword | IANA assigned code for record type being requested |
| www.graylog.org | keyword | Name being resolved in DNS request |
| 25 | Long | Length of name resolution request |
| keyword | Name resolution answer | |
| 25 | Long | Length of name resolution response |
| NXDOMAIN, NOERROR | keyword | Status of name resolution request |
| 0, 3 | keyword | IANA assigned DNS RCODE |
Rule Fields
Related to system/device rules for operating systems, firewalls, etc.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 6da61e4c-84a8-4136-900d-f86c09bb3774 | keyword | Unique identifier of a rule |
| admin-user-template | keyword | Name of a Rule (ex. Outbound Web Traffic) |
Service Fields
Service describes the service/application from which the data was collected.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| graylog-server.service, sshd, graylog-sidecar | keyword | Name of service |
| 1.0.1054 | keyword | Version Number of service or underlying application |
| running, started, stopped | keyword | State of service |
Session Fields
A network session, logon session, any kind of thing with a beginning and and end.
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| Keyword | Vendor-provided unique identifier. This can be a random alphanumeric string, a hex value, a GUID value, etc. |
Source Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 29834710 | long | Network bytes sent by source, some sources may present this as source bytes tx, bytes tx or something similar. |
| iPad | keyword | Device Model Name |
| Apple, ASUS | keyword | Device Vendor Name |
| corpdc01, corpdc01.local, lab01.corpdomain.com | keyword (normalized:loweronly) | NetBIOS or dns hostname, converted to lowercase |
| 09VX93DD | keyword | Identifying value for the source such as a serial number |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | IPv4 and IPv6 addresses |
| fe80:5cc3:11:4::2c | ip | Only IPv6 addresses |
| 10.1.2.3, fe80:5cc3:11:4::2c | ip | translated IP address assigned by a network device performing the NAT function |
| 2384 | integer | translated network port assigned by a network device performing the NAT function |
| IOS, Android | keyword | Operating System Name |
| IOS 10.0 | keyword | Version number of Operating System |
| 23094823 | long | Count of packets sent by source |
| 45392 | integer | numeric port, 0-65535 |
| ssh, ftp | keyword | The IANA-registered service name associated with the network application. Illuminate Core will use this value to define |
| us-east-1 | keyword | Name of region source device is located in |
| keyword | Source device information such as model number | |
| keyword | Virtual system name (not to be confused with the hostname) | |
| keyword | ||
| keyword |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| See: | ||
| keyword | Future: from entity mapping | |
| See: | ||
| Chicago, US, Datacenter 01, Bismark - Finance | keyword | Field is derived either from an internal enterprise network definition or the Geo location fields if availble |
| a0:b4:44:01:a9:d1 | keyword | MAC address of host, colon-delimited and lower case |
| critical, high, medium, low | keyword | Future: from entity mapping |
| 4-Jan | byte | Numeric value representing the priority of the source device, 1 = low, 2 = medium, 3 = high, 4 = critical |
| IPv4,IPv6, hostname,fqdn | keyword (normalized:loweronly) | Automatically mapped from the following fields: |
Threat Fields
Information around threats
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| malware | trojan | Keyword |
| true, false | Keyword | Is a threat detected |
Trace Fields
Tracing makes it possible to track events across multiple logs on a unique ID (Micro-service, Web App)
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| Keyword | Unique ID of multiple events belonging together | |
| Keyword | Stack trace related to process call |
User Fields
Possible Field Prefixes:
source_* (e.g.source_user_name) ordestination_* (e.g.destination_user_name)Where messages describe an action taken by one account impacting another account, the actor (account taking the action) will be described by the
source_user_* fields and the subject (account for which the action was taken) will be described by theuser_* fields. Examples include:Authentication, where the authenticating service account context is provided
IAM events, where a user or service has performed an action that impacts a user or group
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| keyword | ||
| keyword | ||
| mycorp.internal | keyword | AD or LDAP domain |
| keyword | ||
| keyword | Mapped to SID or UID, etc. | |
| keyword (normalized:loweronly) | ||
| 0x534, 1055 | keyword | User logon session identifier |
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| vip, default account, finance, help desk | keyword | Future: From entity mapping |
| Built in\Administrators | keyword (normalized:loweronly) | When a user identity or identities is mapped from a source outside of the message itself it is written to this field. This is where Windows well-known SIDs are resolved. |
| critical, high, medium, low | keyword | Future: From entity mapping |
| 4-Jan | byte | Numeric value representing the priority of the user account, 1 = low, 2 = medium, 3 = high, 4 = critical |
| user, computer, well-known sid, group, {any vendor-provided value} | keyword | Experimental field ** This is still being researched - need to look at what winlogbeats/nxlog may provide in terms of SID resolution in different configurations, and consider different technologies use of “types” |
Included here are fields specific to a vendor's technology, which does not fall under the common schema:
Microsoft Windows Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| S-1-0-0 | keyword | Initial “authority” with SID preamble. For well-known non-domain SIDs this will be the only field contianing SID information. |
| keyword | The domain authority portion of the SID | |
| 500 | keyword | This is the user RID |
| S-1-0-0 | keyword | Initial “authority” with SID preamble. For well-known non-domain SIDs this will be the only field containing SID information. |
| keyword | The domain authority portion of the SID | |
| keyword | This is the user RID | |
| keyword | Initial “authority” with SID preamble. For well-known non-domain SIDs this will be the only field containing SID information. | |
| keyword | The domain authority portion of the SID | |
| keyword | This is the user RID | |
| keyword | This field is defined only when the | |
| NTLM, Kerberos | keyword | Authentication information from Event ID 4624/4625 |
| winlogon.exe | keyword | Authentication information from Event ID 4624/4625 |
| 2, 3, 10 | byte | https://docs.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4624 |
| Interactive, Network, RemoteInteractive | keyword | Description mapped to the logon type field |
| 0x12 | keyword | The Windows kerberos encryption hex value |
| AES256_CTS_HMAC_SHA1 | keyword | |
| krbtgt/CORP.LOCAL | keyword | Name of service targeted for Kerberos ticket requests |
Palo Alto Fields
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| 0, 1 | keyword | Indicates the direction of the attack, client-to-server or server-to-client: 0—direction of the threat is client to server. 1—direction of the threat is server to client |
| keyword | This field is in custom logs only; it is not in the default format. - It contains the full xpath after the configuration change. | |
| keyword | Number to idetify all connections for an association between to SCTP endpoints | |
| LDAP, RADIUS, SAML | keyword | A string showing the authentication type, such as LDAP, RADIUS or SAML |
| keyword | This field is in custom logs only; it is not in the default format. - It contains the full xpath after the configuration change. | |
| wildfire.example.com | keyword | FQDN of WildFire appliance or Cloud where file was uploaded |
| 1, 2, 3, 4 | keyword | ID Numbers that indicate the device groups location within DG Hierarchy |
| Corp-Quarantine | keyword | Name of the dynamic user group that contains the user who initiated the session. |
| system, config, threat | keyword | String showing the name of the event. |
| keyword | Name of the object associated with the system event. | |
| Host visited known malware URL (19 times) | keyword | A summary statement that indicates how many times the host has matched against the conditions defined in the correlation object. For example, Host visited known malware URl (19 times). |
| 0x00000001 | keyword | 32-bit field that provides details on session |
| 6.0.4 | keyword | The client’s GlobalProtect app version. |
| on-demand, user-login | keyword | A string showing the how the GlobalProtect app connects to Gateway, (for example, on-demand or user-login) |
| portal-auth-failed | keyword | A string showing that error that has occurred in any event. |
| 11001 | keyword | An integer associated with any errors that occurred |
| certificate expired | keyword | Additional information for any event that has occurred. |
| gp-gateway01 | keyword | The name of the GlobalProtect portal or gateway. |
| 123e4567-e89b-12d3-a456-426614174000 | keyword | Unique ID GlobalProtect assigns to identify the host. |
| EU-West | keyword | A string showing the administrator-defined location of the GlobalProtect portal or gateway. |
| host-quarantine | keyword | A string that shows the reason for the quarantine |
| HIP-Profile-Workstations | keyword | Name of the HIP object or profile. |
| object, profile | keyword | Whether the hip field represents a HIP object or a HIP profile. |
| parent-session-id, 0 | keyword | Identifies if traffic used an HTTP/2 Connection by displaying one of the following values: Parent session ID—HTTP/2 connection. OR. 0—SSL session |
| 3 | keyword | Number of link flaps during session |
| linkA,linkB,linkC | keyword | Contains up to four link flap entries, with each entry containing the link name, link tag, link type, physical interface, timestamp, bytes read, bytes written, link health, and link flap cause. |
| forwarded-to-panorama | keyword | Log Forwarding Profile Applied to Session |
| 0, 1 | keyword | A bit field indicating if the log was forwarded to Panorama |
| system, config, threat | keyword | Subtype of Given Log |
| auth, general, ha | keyword | It provides additional information about the sub-system generating the log |
| 356938035643809 | keyword | IMEI 15/16 Digit number |
| object-123 | keyword | Name of the object associated with the system event. |
| correlation-object-01 | keyword | Name of the correlation object that was matched on. |
| 1234567890 | keyword | ID of the session in which this session is tunneled |
| 2025-10-27T11:22:33Z | keyword | Time the Tunnel Session began |
| pcap-42 | keyword | Packet Capture ID |
| 17 | keyword | ID of the protocol for the payload of the data chunk |
| 1024 | keyword | Sum of SCTP chunks sent and received for an association. |
| 512 | keyword | Number of SCTP chunks sent for an association. |
| 512 | keyword | Number of SCTP chunks received for an association. |
| SDWAN-Europe | keyword | Name of the SD-WAN cluster. |
| mesh, hub-spoke | keyword | Type of cluster (mesh or hub-spoke) |
| hub, branch | keyword | Type of device (hub or branch) |
| policy-EdgeQoS-01 | keyword | Name of the SD-WAN policy. |
| Paris Branch | keyword | Name of the SD-WAN site |
| aged-out, tcp-rst-from-client | keyword | The reason the session was terminated |
| EMEA, US | keyword | The region for the user who initiated the session. |
| IMSI-310260123456789 | keyword | International Mobile Subscriber Identity Number |
| before-login, login, tunnel | keyword | A string showing the stage of the connection (for example, before-login, login, or tunnel) |
| 42 | keyword | Counter allowing you to correlate order of log entries in URL Filtering/WildFire |
| 4c583e00d47108f809282d5d595f5fb0 | keyword | Binary Hash of file sent to WildFire |
| WF-2025-10-001234 | keyword | Identifies the analysis request on Wildfire Cloud/Appliance |
Vendor Fields
The vendor fields are to capture data provided by source, as-is
The vendor fields are intended to capture information that is either used in the content we develop, or can be used to provide background on how a field such as
event_outcomewas defined
Field Name | Example Values | Field Type | Notes |
|---|---|---|---|
| critical, high, medium, low | keyword | When the message is an alert this is the vendor-provided text description of the alert severity |
| 4, 3, 2, 1 | integer | When the message is an alert this is the vendor-provided numeric value for the alert severity |
| Active Directory | keyword | Vendor defined action - Quick description of the service providing credential validation |
| password, token | keyword | Vendor-defined credential type |
| allow, deny, pass, fail | keyword | Vendor defined action - this should be a short, typically one-word, description of what action the event is describing. The value is to be used verbatim, including case, from the source log. |
| Removable Media, Registry, File System | keyword | Vendor defined category of an event |
| keyword | Vendor defined description of the action with more detail than is included in | |
| block, drop, report, allow, reject | keyword | Vendor-defined result of the action defined in the message |
| keyword | Vendor-provided text detailing the reason for the vendor-provided action and/or outcome the message is describing | |
| critical, high, medium, low, informational | keyword | Vendor-defined text description of the severity rating |
| 0, 1, 5, 10 | integer | Vendor-defined numeric severity rating for this event |
| ip | ||
| ip | ||
| ip | ||
| ip | ||
| keyword | ||
| ids, dnsmasq, kernel, threat | keyword | Vendor-defined subtype of log - this differs from |
| keyword | ||
| keyword | ||
| keyword | ||
| keyword |
Message Categories
The below table shows how Security Data Lake is mapping gim_event_type_code, created in a pipeline, to a normalized category in Illuminate content. Normalized categories allow for dashboards, searches, and alert rules to use a common name across all device types utilizing this format. An example of this line in lookup tables is:
"100000","|authentication|","|logon|","logon"
The code 100000 is attached to the log in the processing pipeline and allows for the lookup function to attach a category, sub category, and event type further down the processing chain.
The category in the above case is |authentication|, where many types of events can fall. Log on, log off and, session disconnect all fall under authentication for easy grouping on dashboards. A subcategory of |logon| is applied as well to this log to signify this is happening during the log on process. There can be many under log on, like log on success and log on failure. Finally, the event type is added as logon for further granularity of what this event was processed as.
Note
This document is a work in progress, and fields will be added as content is developed. If you have suggestions for changes or additions, please open a GitHub issuehere.
gim_event_type_code | gim_event_class | gim_event_category | gim_event_subcategory | gim_event_type |
|---|---|---|---|---|
000000 | message | message.log_message | message | |
100000 | authentication | authentication.logon | logon | |
100003 | authentication | authentication.logon | logon with alternate credentials | |
100004 | authentication | authentication.logon | session reconnect | |
100500 | authentication | authentication.credential validation | credential validation | |
100501 | authentication | authentication.credential validation | error | |
100502 | authentication | authentication.credential validation | mfa | |
100503 | authentication | authentication.credential validation | sms_send_message | |
100504 | authentication | authentication.credential validation | voice_call | |
101000 | authentication | authentication.access notice | special logon | |
101001 | authentication | authentication.access notice | error | |
101500 | authentication | authentication.access policy | access policy violation | |
101501 | authentication | authentication.access policy | device policy violation | |
101502 | authentication | authentication.access policy | account policy violation | |
102000 | authentication | authentication.kerberos request | service ticket renewed | |
102001 | authentication | authentication.kerberos request | service ticket requested | |
102002 | authentication | authentication.kerberos request | tgt request | |
102003 | authentication | authentication.kerberos request | error | |
102500 | authentication | authentication.logoff | logoff | |
102501 | authentication | authentication.logoff | session disconnect | |
109999 | authentication | authentication.default | authentication message | |
110000 | iam | iam.object create | account created | |
110001 | iam | iam.object create | error | |
110002 | iam | iam.object create | group created | |
110500 | iam | iam.object delete | account deleted | |
110501 | iam | iam.object delete | group deleted | |
111000 | iam | iam.object modify | account modified | |
111001 | iam | iam.object modify | privileges assigned | |
111002 | iam | iam.object modify | privileges removed | |
111003 | iam | iam.object modify | account renamed | |
111004 | iam | iam.object modify | password change | |
111005 | iam | iam.object modify | administrative password reset | |
111006 | iam | iam.object modify | error | |
111007 | iam | iam.object modify | group member added | |
111008 | iam | iam.object modify | group member removed | |
111009 | iam | iam.object modify | group properties modified | |
111500 | iam | iam.object disable | account locked | |
111501 | iam | iam.object disable | account disabled | |
112000 | iam | iam.object enable | account unlocked | |
112001 | iam | iam.object enable | account enabled | |
112002 | iam | iam.object enable | error | |
119500 | iam | iam.information | group membership enumerated | |
119999 | iam | iam.default | iam message | |
120000 | network | network.network connection | network connection | |
120100 | network | network.routing | network routing | |
120200 | network | network.open | network connection initiated | |
120300 | network | network.close | network connection ended | |
120500 | network | network.flow | flow record | |
120600 | network | network.icmp_request | icmp_request | |
120700 | network | network.icmp_reply | icmp_reply | |
129999 | network | network.default | network message | |
130000 | messaging | messaging.email | email sent | |
130500 | messaging | messaging.email | email blocked | |
131000 | messaging | messaging.email | email rejected | |
131500 | messaging | messaging.email | email quarantined | |
132000 | messaging | messaging.email | email deleted | |
139999 | messaging | messaging.default | message | |
140000 | protocol | name resolution | name resolution.dns request | dns query |
140200 | protocol | name resolution | name resolution.dns answer | dns response |
140300 | protocol | name resolution | name resolution.error | dns error |
140500 | protocol | name resolution | name resolution.ddns update | ddns update |
149999 | protocol | name resolution | name resolution.default | dns message |
150000 | database | database.query | database query | |
150500 | database | database.update | update rows | |
151000 | database | database.add | insert rows | |
151001 | database | database.add | add table | |
151002 | database | database.add | create database | |
151500 | database | database.delete | delete rows | |
151501 | database | database.delete | drop table | |
151502 | database | database.delete | drop database | |
159999 | database | database.default | database message | |
170000 | alert | alert.network alert | ids alert | |
170001 | alert | alert.network alert | network alert | |
170002 | alert | alert.network alert | network dlp alert | |
171000 | alert | alert.host alert | malware alert | |
171001 | alert | alert.host alert | host dlp alert | |
171002 | alert | alert.host alert | hips alert | |
171003 | alert | alert.host alert | fim alert | |
179999 | alert | alert.default | alert message | |
180000 | protocol | http | http.default | http message |
180100 | protocol | http | http.request | http request |
180200 | protocol | http | http.communication | http communication |
180300 | protocol | http | http.proxied | http proxied communication |
190000 | endpoint | process | process.execute | process started |
190100 | endpoint | process | process.end | process stopped |
190500 | endpoint | process | process.interaction | process accessed |
190501 | endpoint | process | process.interaction | remote thread created |
191000 | endpoint | process | process.action | process altered |
191001 | endpoint | process | process.action | image loaded |
199990 | endpoint | process | process.default | process message |
200000 | endpoint | file | file.create | file created |
200100 | endpoint | file | file.delete | file deleted |
201000 | endpoint | file | file.modify | file modified |
201001 | endpoint | file | file.modify | file timestamp modified |
201002 | endpoint | file | file.modify | file stream created |
201500 | endpoint | file | file.access | file accessed |
201501 | endpoint | file | file.access | raw file access |
202000 | endpoint | file | file.integrity | file signature invalid |
202001 | endpoint | file | file.integrity | file integrity notice |
209999 | endpoint | file | file.default | file event |
210000 | endpoint | service | service.start | service started |
210100 | endpoint | service | service.stop | service stopped |
211000 | endpoint | service | service.configuration | service configuration change |
211500 | endpoint | service | service.state | service installed |
211501 | endpoint | service | service.state | service removed |
211502 | endpoint | service | service.state | service enabled |
211503 | endpoint | service | service.state | service disabled |
211504 | endpoint | service | service.state | service error |
219999 | endpoint | service | service.default | service event |
220000 | endpoint | audit | audit.integrity | audit log cleared |
220100 | endpoint | audit | audit.state | audit service started |
220101 | endpoint | audit | audit.state | audit service stopped |
220102 | endpoint | audit | audit.state | audit error |
220500 | endpoint | audit | audit.policy | audit policy changed |
229999 | endpoint | audit | audit.default | audit event |
230000 | endpoint | pipe | pipe.add | pipe created |
230100 | endpoint | pipe | pipe.remove | pipe deleted |
230500 | endpoint | pipe | pipe.state | pipe connected |
239999 | endpoint | pipe | pipe.default | pipe event |
240000 | endpoint | wmi | wmi.filter | wmi filter created |
240001 | endpoint | wmi | wmi.filter | wmi filter removed |
240500 | endpoint | wmi | wmi.consumer | wmi consumer created |
240501 | endpoint | wmi | wmi.consumer | wmi consumer removed |
241000 | endpoint | wmi | wmi.binding | wmi binding created |
249999 | endpoint | wmi | wmi.default | wmi event |
250000 | endpoint | registry | registry.value_change | registry value set |
250001 | endpoint | registry | registry.value_change | registry value added |
250002 | endpoint | registry | registry.value_change | registry value deleted |
250003 | endpoint | registry | registry.value_change | registry value modified |
250500 | endpoint | registry | registry.key_change | registry key added |
250501 | endpoint | registry | registry.key_change | registry key deleted |
250502 | endpoint | registry | registry.key_change | registry key renamed |
251000 | endpoint | registry | registry.object_renamed | registry object renamed |
259999 | endpoint | registry | registry.default | registry event |
260000 | endpoint | system_time | system_time.time_change | system time changed |
269999 | endpoint | system_time | system_time.default | system time event |
270000 | endpoint | driver | driver.loaded | system driver loaded |
270100 | endpoint | driver | driver.unloaded | system driver unloaded |
279999 | endpoint | driver | driver.default | system driver event |
280000 | endpoint | agent | agent.activity | agent activity |
280001 | endpoint | agent | agent.activity | antivirus and malware scan |
280100 | endpoint | agent | agent.update | agent update |
280200 | endpoint | agent | agent.status | agent status |
289999 | endpoint | agent | agent.default | agent default |
290000 | protocol | dhcp | dhcp.request | dhcp request |
290100 | protocol | dhcp | dhcp.offer | dhcp offer |
290200 | protocol | dhcp | dhcp.discovery | dhcp discovery |
290300 | protocol | dhcp | dhcp.acknowledgement | dhcp acknowledgement |
299999 | protocol | dhcp | dhcp.default | dhcp default event |
300000 | detection | detection.network_detection | ids_detection | |
300001 | detection | detection.network_detection | network_detection | |
300002 | detection | detection.network_detection | network_dlp_detection | |
301000 | detection | detection.host_detection | host_malware_detection | |
301001 | detection | detection.host_detection | host_dlp_detection | |
301002 | detection | detection.host_detection | hips_detection | |
301003 | detection | detection.host_detection | fim_detection | |
309999 | detection | detection.default | detection_message |