Logstash 101 with Docker

We, at Shippable, are using Docker extensively and ALL of our services(frontend, backend, messaging, etc) are now successfully running inside containers. The latest addition to this is Logstash, which is an awesome tool to store, manage and make sense of tons of logs that are spewed out by a typical application. Its for the same use case we decided to use Logstash which basically collects the data from all our containers to make it easier for us to track down exceptions and some important events. I’ll start by defining a simple Logstash setup to do this and then conclude by wrapping everything into a neat Dockerfile which, in essence, will be all you need to then run Logstash successfully.

  1. Download the logstash jar
  1. Create the config file (name it ls.conf) to be used by Logstash which defines how(and from where) Logstash takes the input data and how(and where) it dumps the output. The simplest config is something like this (taken from logstash getting-started tutorial). There are two sections in the config file by the name “input” and “output”. In each section, we can put multiple options which implies that Logstash can take input from various channels like rabbitmq, redis, files etc and dump the output to similar channels. In this case, there is only one input (console) and one output(console).

Run the following command to start Logstash agent.

It takes about 30 seconds for the server to fire up. Typing anything on the console should show an output like the following, which implies that the server is up and running.

  1. Let’s add a few more features to the server so that it receives input from rabbitmq server and dumps the ouput to an elasticsearch instance. Logstash ships with an elasticsearch server which can be used to index the output. Add the following lines to “ls.conf” to take the two inputs from rabbitmq exchange called “rabbitmq_exchange” from different keys and dumps the output to the embedded elasticsearch server.
  1. Time to Docker-ify everything. The final objective is to wrap up the setup, installation, download and configuration of Logstash and its dependencies into one Dockerfile. Create a folder and put “ls.conf” defined in step 3 alongwith the Dockerfile described below. Run following to build the container

$ docker build -rm=true -t ls/logstash .

  1. running Logstash inside the container.

$ docker run -d ls/logstash

That’s about it. We now have a logstash instance listening  for messages from a rabbitmq server and pushing the data into elasticsearch. Do read more about setting up initial Logstash configuration HERE . The only change to the container would be to update the config file (ls.conf) and build the container again.

Share Comments
comments powered by Disqus