Fluentd

Fluentd is an open source data collector that supports different formats, protocols, and customizable plugins for reading and writing log streams.

If you have data in Fluentd, we recommend using the Unomaly plugin to forward that data directly to a Unomaly instance for analysis.
As a fallback option for data ingestion, Unomaly also runs with Fluentd pre-installed on the instance. This topic helps you to configure Fluentd on the instance to receive data from other Fluentd, Docker, or syslog.

The Unomaly plugin for Fluentd

Send fluent records to the Unomaly ingestion API with the Unomaly plugin for Fluentd: https://github.com/unomaly/fluent-plugin-unomaly

Sending data to Fluentd on Unomaly

1. Fluentd listens on port 24224, which is the default fluentd forwarder port. This means that the port needs to be accessible through the firewall. See "Edit network and communication settings".

2. The main Fluentd configuration file on your Unomaly instance is located at /DATA/fluentd/etc/fluent.conf.

Do not make any changes to this file, because it will be overwritten on upgrades. Instead, add your configurations to /DATA/fluentd/etc/conf.d as separate files.

Fluentd configuration options
Directive	Description
source	Decides which interface fluentd should use to read data.
filter	Allows you to define the custom filters that modify event streams.
match	Tells fluentd what to do.
label	Used to define different sources.

Refer to Fluentd's documentation for more descriptions of their configuration options: https://docs.fluentd.org/v0.12/articles/config-file.

3. After you make any changes to the configuration file, you need to restart Fluentd:

unomaly restart fluentd

You can view the stdout/log of Fluentd by running:

unomaly logs fluentd

Or, you can run the following for tail-mode viewing:

unomaly logs fluentd -f

Receive data from other Fluentd or Docker

The following configuration example receives data forwarded from other Fluentd installations or Docker.

<source>
  @type forward
  port 24224
  bind 0.0.0.0
</source>
<filter **.docker.**>
  @type record_transformer
  <record>
    hostname "${tag_parts[2]}"
  </record>
</filter>
<match **.docker.**>
  @type unomaly
  host https://127.0.0.1
  flush_interval 1s
  source_key hostname
  message_key log
  accept_self_signed_certs true
</match>

Specifically in this case, the following statements are declared:

source

@type forward means that this plugin is mainly used to receive event logs from other Fluentd instances, the fluent-cat command, or client libraries. This is by far the most efficient way to retrieve the records.
port 24224 designates the port on which fluentd should listen for data.
bind 0.0.0.0 means that it will listen to any network interface.

filter

filter **.docker.** means that messages are transformed.
In this case we re-write the hostname in the <record>-block.
tag_parts[2] means to against the second index of the tag in the sender’s message.

match

match **.docker.** means that messages from docker matched.
@type unomaly means that fluentd will look for a file called out_unomaly.rb in /DATA/fluentd/plugins and pass log data into it. The out_unomaly.rb plugin will ingest data into Unomaly.

Receive standard syslog data

The following configuration example receives standard syslog data and ingests them into Unomaly.

<source>
    @type syslog
    @label @mystream
    port 51400
    bind 0.0.0.0
    tag system
</source>
<label @mystream>
    <match system.**>
      @type unomaly
      host https://172.16.238.1
      flush_interval 1s
      source_key hostname
      message_key log
      accept_self_signed_certs true
    </match>
</label>

You may add the following source section to switch protocols to TCP syslogs (by default, UDP is used):

protocol_type tcp

If you have problems receiving syslog messages, it may be the format doesn't match. You can change the format in the source section using message_format:

message_format auto
message_format rfc3164
message_format rfc5424

Installing Fluentd plugins

You can install Fluentd plugins by using unomaly_fluent_gem:

unomaly restart fluentd

For a list of available commands run:

unomaly logs fluentd

After installing new plugins you need to restart the fluentd service to be able to use them in a configuration:

unomaly restart fluentd

You can write your own plugins or find existing ones for Fluentd and save them into /DATA/fluentd/plugins:

Make sure that they are registers with the name you use in the fluentd.conf
Make sure that the source-code registers them in the same name
Make sure that the file-name matches and has the correct type (such as out_ for output type plugins)

Debugging

To see whether data comes into fluentd at all, you can use for example:

<match **>
    @type stdout
</match>

This will print the message on the stdout of the running fluentd process.

You can use for instance fluent-cat (a fluentd tool) or simply logger (a standard Linux syslog tool) to produce log message input for fluentd.

1. Example of using fluent-cat:

echo '{"message":"hello world"}' | fluent-cat --host 10.164.0.7 --port 24224 --format json hello

2. Example of using logger:

logger --server 10.164.0.7 --port 51400 '<6>Jan 13 12:34:56 sunworld myapp[31802]: 
[info] test logline in rfc3164'