Data ingestion components
Data Insights supports several methods for collecting and forwarding log data from cloud, on-premises, and hybrid environments. The method you choose affects scalability, data quality, and real-time analysis. These are the main data ingestion components used by Data Insights:
Inputs - This is the main mechanism Data Insights uses to receive data from external sources. Each input is tied to a specific protocol or format, such as Syslog, Beats, GELF, or HTTP, and delivers log data into the Data Insights server. Inputs are configured for the entire cluster, with options such as port numbers, authentication, and parsing. They are best suited for applications or devices that can send logs directly to Data Insights.
Sidecar - A centralized management service for log shipper agents such as Filebeat, Winlogbeat, and NXLog. It lets you deploy, configure, and monitor log shippers across multiple servers or endpoints without managing each one separately. Sidecar helps maintain consistent log collection, simplifies configuration updates, and reduces effort when working with large numbers of systems.
Forwarder - It securely collects and forwards logs from remote or distributed environments to a central Data Insights cluster. It supports encryption, buffering, and queuing to ensure reliable delivery, even in low-bandwidth or high-latency conditions.