Use a custom Fluentd in Unomaly

If you need to install fluentd plugins that require system libraries, you may find it easier to build and run a custom installation of Fluentd. In the following example, we will install a custom version of Fluentd with support for ingesting data from Kafka.

Create a directory that will contain your custom configuration and build the Docker image:

mkdir -p /home/unomaly/fluentd/{etc,plugins,log}
cd /home/unomaly/fluentd

Create a file called "Dockerfile" containing the following contents, and adding any custom dependencies that you may need:

FROM fluent/fluentd:v1.5-1

USER root

RUN apk add --no-cache --update --virtual .build-deps \
  build-base ruby-dev \
  && gem install --no-document \
    fluent-plugin-unomaly \
    fluent-plugin-kafka \
 && gem sources --clear-all \
 && apk del .build-deps \
 && rm -rf /home/fluent/.gem/ruby/2.5.0/cache/*.gem

USER fluent

Create a file called "fluent.conf" in the "etc" folder created above with your custom configuration:

<source>
@type kafka

brokers <broker1_host>:<broker1_port>,<broker2_host>:<broker2_port>
 topics <listening topics(separate with comma',')>
 format <input text type (text|json|ltsv|msgpack)> :default => json
 message_key <key (Optional, for text format only, default is message)>
 add_prefix <tag prefix (Optional)>
 add_suffix <tag suffix (Optional)>

# Optionally, you can manage topic offset by using zookeeper
 offset_zookeeper <zookeer node list (<zookeeper1_host>:<zookeeper1_port>,<zookeeper2_host>:<zookeeper2_port>,..)>
 offset_zk_root_node <offset path in zookeeper> default => '/fluent-plugin-kafka'

# ruby-kafka consumer options
 max_bytes (integer) :default => nil (Use default of ruby-kafka)
 max_wait_time (integer) :default => nil (Use default of ruby-kafka)
 min_bytes (integer) :default => nil (Use default of ruby-kafka)
</source>

<match kafka.*>
  @type unomaly
  host http://127.0.0.1:6010
  api_path /batch
  flush_interval 1s
  use_ssl false
  debug false
  source_key host
  message_key message
  accept_self_signed_certs true
</match>

Now build your Docker image:

sudo docker build . -t myfluentd

Now you can test your Docker image:

docker run --rm -ti -v $(pwd):/fluentd myfluentd

To run in production, use a restart policy to always have it started:

docker run -d --restart=always -v $(pwd):/fluentd myfluentd