No systems are visible

If new systems do not show up in the interface, Unomaly may not be configured to receive the data. You can troubleshoot this issue by checking that the communications settings are configured to receive data and that the services that handle ingestion and queuing of the data are running.

Verify that data is being received on syslog

1. Open a console in Unomaly
2. Dump traffic using one of the following commands:
   • If direct from systems: sudo tcpdump -s0 -Anneevvi eth0 port 514
   • If from a centralized syslog server: sudo tcpdump -s0 -Anneevvi eth0 port 5514
3. Investigate received data:
   • If data is coming in here, it’s reaching the Unomaly engine and if the data also looks correctly formatted, there should be no issues on the network end of the setup.
   • If no data is coming in here, it’s most likely an issue outside of the Unomaly setup, as logs are not reaching the syslog port.

Log data in Unomaly has timestamp and hostname as part of the event

Sometimes incoming syslog messages are in the RFC 5424 format, which the old-style syslog-ng receiver cannot parse correctly. The timestamp is visible, and so is the hostname. This can be handled by setting up an alternative listening port in syslog-ng.

Example of syslog RFC:

<14>1 2016-04-20T13:46:10.698+02:00 host.example.com config://server-log4j2.xml - - [mdc@18060 instanceName="web1" method="DatastructureProtocolHandler.checkHeartbeatAndQueueSizes"] PdaConnection Checking hearbeats and queue sizes for 1 session(s). Average load last minute incoming/outgoing 0.104/0.095

Setting up a new listening port:

Add to the following file: /etc/syslog-ng/conf.d/90unomaly-sources.conf

# rfc 5424-compliant log reception
source s_rfc5424 {
 syslog(ip(0.0.0.0) port(10000) transport("udp"));
 syslog(ip(0.0.0.0) port(10000) transport("tcp") max-connections(5000) log_fetch_limit(10) log_iw_size(750000)); };

log {
 source(s_rfc5424);
 rewrite(r_rewrite_subst);
 destination(d_sinkd);
};

Next, open the new port in the firewall:

sudo ufw allow 10000/tcp

If you send your logs to port 10000, they should now be handled properly.

Verify that services are running correctly

1. Check that all services are running.

   unomaly@unomaly:~$ unomaly

2. Check that Unomaly is receiving data from the system in question, if applicable.

   unomaly@unomaly:~$ sudo tcpdump -i eth0 port 514 or port 5514

   If Unomaly is receiving the data, you will see output similar to the following:

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    10:16:23.158834 IP jumpfwd.50637 > 192.168.1.155.5514: Flags [P.], seq 3192370025:3192370173, ack 907877001, win 229, options [nop,nop,TS val 280598562 ecr 642251340], length 148
    10:16:23.158876 IP 192.168.1.155.5514 > jumpfwd.50637: Flags [.], ack 148, win 1833, options [nop,nop,TS val 642251435 ecr 280598562], length 0
    10:16:23.691781 IP jumpfwd.50637 > 192.168.1.155.5514: Flags [P.], seq 148:273, ack 1, win 229, options [nop,nop,TS val 280598695 ecr 642251435], length 125
    10:16:23.691794 IP 192.168.1.155.5514 > jumpfwd.50637: Flags [.], ack 273, win 1833, options [nop,nop,TS val 642251568 ecr 280598695], length 0

   If you do not get a response, Unomaly is not receiving data on the specified communication ports. In this case, you can check the specific system (data source) or, if applicable, firewalls between the source and Unomaly.

3. Check that the queue, ingestion, is receiving the data

   unomaly@unomaly:~$ unomaly logs ingestion
    ==> Press Ctrl-C to exit <==
    ngestion
    INFO 2018-06-05T08:17:55Z events_in/s=95.289689 events_in_queue=0 events_out/s=95.289689 events_received=25563954 total_memory_MB=319 pid=24036 pkg=l/u/u/g/u/ingestion
    INFO 2018-06-05T08:18:00Z events_in/s=93.891290 events_in_queue=0 events_out/s=93.891290 events_received=25564343 total_memory_MB=319 pid=24036 pkg=l/u/u/g/u/ingestion
    INFO 2018-06-05T08:18:05Z events_in/s=96.634463 events_in_queue=0 events_out/s=96.634463 events_received=25564984 total_memory_MB=319 pid=24036 pkg=l/u/u/g/u/ingestion

   If the events_in/s > 0, the queue is receiving data and the problem is downstream.

   If the events_in/s = 0, the queue is not receiving data and you need to restart the receiving and queue services: syslog-ng and ingestion.

    unomaly@unomaly:~$ unomaly restart syslogng

    unomaly@unomaly:~$ unomaly restart ingestion

4. Check that tad is pulling data from the queue

   unomaly@unomaly:~$ unomaly logs ingestion
    ==> Press Ctrl-C to exit <==
    ngestion
    INFO 2018-06-05T08:17:55Z events_in/s=95.289689 events_in_queue=0 events_out/s=95.289689 events_received=25563954 total_memory_MB=319 pid=24036 pkg=l/u/u/g/u/ingestion
    INFO 2018-06-05T08:18:00Z events_in/s=93.891290 events_in_queue=0 events_out/s=93.891290 events_received=25564343 total_memory_MB=319 pid=24036 pkg=l/u/u/g/u/ingestion
    INFO 2018-06-05T08:18:05Z events_in/s=96.634463 events_in_queue=0 events_out/s=96.634463 events_received=25564984 total_memory_MB=319 pid=24036 pkg=l/u/u/g/u/ingestion

   If events_in_queue is growing or events_out/s is zero, tad is not pulling the data properly.

   Restart tad.

   unomaly@unomaly:~$ unomaly restart tad

   If restarting the service does not fix the issue, contact Unomaly Support.