How to send data to Unomaly
Getting data into Unomaly means configuring the data source to send the log data to the Unomaly instance. Unomaly analyzes this data without specific parsers or predefined knowledge of the data format or structure. Because some ingestion methods are better suited for certain types of log data, choosing the right protocol or integration to send your data can improve how Unomaly learns and detects anomalies.
Input options based on data sources
The input options that you have for sending data to Unomaly depend on where the data is coming from: Directly from the host machines that generate the data, from log collectors or log servers, from cloud services, or from other applications and technologies. For just about any data source, you can use one of our plugins (Logstash, Graylog, or Fluentd) to send their data to Unomaly. But, you can also send the data directly to the Unomaly ingestion API using HTTP.
Direct from host machines
Most operating systems have built-in support or services for collecting and forwarding their own data. For example, Unix-based systems, VMWare, and networking devices have pre-installed versions of syslog that you can configure to forward data to Unomaly.
| Data source | Data input options |
|---|---|
| Unix and Linux | Unix-based systems have built-in support for syslog (syslogd, rsyslog, or syslog-ng) which you can configure to forward all your local logs. See guidelines for sending data from Unix and Linux. If you use Logstash, Graylog, or Fluentd to collect your logs, see how to forward their data in the next section. |
| VMWare ESX | Forward logs from your ESXi hosts using the built-in vmsyslogd service. See the guidelines for sending data from VMWare ESX. |
| Windows | Windows EventLogs require an agent to collect and forward to Unomaly. For example, you can use nxLog to forward the data to syslog on Unomaly. See guidelines for sending Windows EventLogs to Unomaly using nxLog. |
From log collectors and log servers
There are many standard agents that specialize in collecting log data and forwarding them to another platform for processing. If you're using Logstash, Graylog, or Fluentd to aggregate your logs, use on of Unomaly's plugins to forward that data to a Unomaly instance.
| Data source | Data input options |
|---|---|
| Logstash | Forward Logstash events to the Unomaly ingestion API for analysis. See the Logstash plugin. |
| Graylog | Forward streaming logs from Graylog to the Unomaly ingestion API for analysis. See the Graylog plugin. |
| Fluentd | Forward data from Fluentd to the Unomaly ingestion API for analysis. See the Fluentd plugin. |
| Syslog servers | Forward unstructured log data to Unomaly using standard TCP/UDP protocols. This is not the recommended option for forwarding structured data. See the Syslog server guidelines. |
| Data source | Data input options |
|---|---|
| AWS Cloudtrail | Send JSON formatted CloudTrail logs directly to Unomaly's ingestion API. See our guidelines for AWS Cloudtrail. |
| AWS Cloudwatch | We provide lambda functions to. push AWS Cloudwatch logs directly to Unomaly's ingestion API. See our guidelines for AWS Cloudwatch. |
From other applications and technologies
We provide integrations and guidelines to get data from many of the applications, devices, and services that make up a typical IT infrastructure. If you already use other data technologies or log management tools to collect and process your data, you can use one of of our integrations to forward that data to Unomaly for analysis. For example, Unomaly provides integrations to analyze from Docker, Splunk, and others.
Troubleshooting data input issues
The following can help you investigate when you have an issue getting the data sources into Unomaly:
- The new system doesn't show up in Unomaly. New systems are added in training, so check to make sure that you have selected to "Show systems in training". If you still do not see the new systems, check that the communications settings are configured to receive data and that services that handle ingestion and queuing of the data are running.
- Unomaly receives the data but it does not look correct. This may indicate an issue with the tokenization of the events in the data.
- The system is still in training after more than two weeks. The most common reason for this is that the system does not produce enough logs for Unomaly to learns its normal behavior. You may consider manually taking the systems out of training. Another reason may be that there is an issue tokenizing the structures of the log events.
See Troubleshooting data ingestion.
Next steps
While Unomaly learns your data, you can learn more about "How Unomaly detects anomalies" and how to "Manage systems and groups" to organize your data.