Forwarding Docker logs to Logstash

Here at Sandtable we’re as excited about Docker as the next guy. Shippable lightweight containers, yay! We’re running services in Docker containers using Marathon on Mesos in AWS EC2. Now that’s a buzzstack™.

It’s actually going rather well. But when things are not going so well, we turn to logs. We’re already fans of the ELK stack: Elasticsearch + Logstash + Kibana, so naturally we wondered whether we could forward logs from Docker containers to Logstash, all to be explored in the slick new Kibana dashboard.

Here’s our approach.

The Setup

The Approach

We started with a constraint: we didn’t want to change the configuration of Docker containers currently running. Consequently, solutions that required mounting volumes or modifying the Docker images didn’t suit us.

After some searching, we figured we could take a similar approach to that Jason Wilder did but using Logstash instead of Fluentd.

To achieve this we decided to use filebeat (that replaced logstash-forwarder) and Jason’s docker-gen tool (thanks Jason!).

Filebeat is used to transmit data to logstash.

Docker-gen is neat little tool used to automatically generate the configuration file for filebeat given the running containers. Docker-gen watches for Docker events (for example, a new container is started, or a container is stopped), regenerates the configuration, and restarts filebeat.

Step-by-step

Note that we do not provide details of the configuration of the Docker images for the ELK stack. See their respective hub.docker.com pages for more details.

Step 1: Configure logstash

Filebeat requires the Beats input plugin for Logstash.
The “input” in the logstash configuration looks like this:

input {
  beats {
    port => "5044"
  }
}
filter {
[....]

For more configuration options for the Beats input plugin, see the official documentation.

Step 2: Install docker-gen

# wget https://github.com/jwilder/docker-gen/releases/download/0.7.2/docker-gen-linux-amd64-0.7.2.tar.gz
# mkdir /opt/docker-gen
# tar xvzf docker-gen-linux-amd64-0.7.2.tar.gz -C /opt/docker-gen
# mkdir /etc/docker-gen

Step 3: Create the docker-gen template for filebeat

This is the template for the filebeat configuration file.
Create the file /etc/docker-gen/filebeat.tmpl (replace the text in <> by your values):

{{/* this is a docker-gen template. See docker-gen for more details */}}
filebeat.prospectors:
  {{ range $key, $value := . }}
    - paths:
        - /var/lib/docker/containers/{{ $value.ID }}/{{ $value.ID }}-json.log
      fields:
        type: {{ $value.Image.Repository }}
  {{ end }}
output.logstash:
    hosts: ["<your_logstash_domain>:5044"]
logging.level: warning

 

This will create as many prospectors as running containers.

For more configuration options, see the official filebeat documentation.

Step 4: Automatically start docker-gen during boot

Option 1: Using Systemd

Create the file: /etc/systemd/system/docker-gen.service :

[Unit]
Description=A file generator that renders templates using Docker Container meta-data.
Documentation=https://github.com/jwilder/docker-gen
After=network.target docker.socket
Requires=docker.socket

[Service]
ExecStart=/usr/bin/docker-gen -notify "/etc/init.d/filebeat restart > /dev/null 2>&1 &" -notify-output -watch /etc/docker-gen/filebeat.tmpl /etc/filebeat/filebeat.yml
Restart=always

[Install]
WantedBy=multi-user.target
Alias=docker-gen.service

 
To activate it, run:
# systemctl enable docker-gen
 

Option 2: Using Upstart

Create the file /etc/init/docker-gen.conf :

#!upstart

description "docker-gen upstart script"

start on startup
stop on shutdown

script
  exec /opt/docker-gen/docker-gen -notify "/etc/init.d/filebeat restart > /dev/null 2>&1 &" -notify-output -watch /etc/docker-gen/filebeat.tmpl /etc/filebeat/filebeat.yml
end script

# Log to /var/log/upstart/<job>.log
console log

 

Step 5: Install filebeat

When writing this blog, the latest version of filebeat is 5.1.2. You can also refer to the official documentation for installation instructions.
Example for Ubuntu installation:
$ curl -L -O https://download.elastic.co/beats/filebeat/filebeat-5.1.2-amd64.deb
$ sudo dpkg -i filebeat-5.1.2-amd64.deb

We don’t touch at the default configuration file as docker-gen will auto-generate this file.

Step 6: Start docker-gen

# start docker-gen

And that’s all folks. Docker-gen will then create the filebeat configuration file based on the template and restart the service.

Issues?

Check the log file:

/var/log/upstart/docker-gen.log

Filebeat send the logs to syslog by default. You can change the log level by modifying the logging level value in the filebeat configuration file (in our case we will modify the docker-gen template /etc/docker-gen/filebeat.tmpl)

References:

https://github.com/jwilder/docker-gen/blob/master/README.md

Last update: January 20th 2017 – Thanks to Richard Laing for his contribution!

Comments

  1. Richard says:

    Hi there I was wished to enquirer about updating the upstart script for 16.04 ? Also in my ELK setup I currently use TLS for the filebeat forwarder, could take a look at this config and tell me if it would work?

    output:
    logstash:
    hosts: [“YYYY.XXXX.5044”]
    timeout: 15
    worker: 1
    loadbalance: true
    index: filebeat
    tls:
    certificate_authorities: [“/etc/ssl/certs/filebeat.crt”]

Leave a comment

Please prove that you are human: