humio

README

Humio Module for Logspout

This module allows Logspout to send Docker logs in the HEC format to Humio via TCP optionally with TLS. We provide a way to build a container that includes Logspout and the Humio adapter module so you can forward Docker logs in HEC format using humio://hostname:port as the Logspout command.

Why

Because Humio is capable of accepting Splunk HEC formatted log events, but Logspout doesn't support that out of the box. Other changed included reformatting the time into 3339 format.

Build

To build the docker container:

docker build --no-cache -t $(whoami)/logspout-humio:0.0.1 .

And publish it to Docker Hub:

docker push $(whoami)/logspout-humio:0.0.1

Or archive and distribute it by hand as a tar file you will need to save the Docker image, then copy it to some new host (scp, rsync, ...) and load it.

docker save -o /tmp/logspout-humio.tar $(whoami)/logspout-humio:0.0.1
scp /tmp/logspout-humio.tar user@host.example.com:/tmp

Then on that remote host:

ssh user@hostexample.com
docker load -i /tmp/logspout-humio.tar

Or do it in one step:

docker save $(whoami)/logspout-humio:0.0.1 | bzip2 | pv | ssh user@host.example.com 'bunzip2 | docker load'

Examples

Command Line

If the load-balancer for Humio is accepting HTTPS requests on 443 for the API then you need to include the port (443) as below.

When testing:

docker run \
    --rm \
	--env HUMIO_INGEST_TOKEN=h9ghusufdghsfghsfghosfhg --env HUMIO_DOCKER_LABELS=true --env DEBUG=1 \
    --name logspout-humio \
	--hostname $(hostname -f) \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	--publish=127.0.0.1:8000:80 \
	$(whoami)/logspout-humio:0.0.1 \
	humio://cloud.us.humio.com:443

You can run curl http://127.0.0.1:8000/logs to view what Logspout is shipping to Humio.

As a daemon, for production/daily unattended use:

docker run \
    --detach \
	--env HUMIO_INGEST_TOKEN=h9ghusufdghsfghsfghosfhg --env HUMIO_DOCKER_LABELS=true \
    --name logspout-humio \
	--hostname $(hostname -f) \
	--volume=/var/run/docker.sock:/var/run/docker.sock \
	--restart=unless-stopped \
	$(whoami)/logspout-humio:0.0.1 \
	humio://cloud.us.humio.com:443

Docker Compose

You could use this image with the following docker-compose file:

version: '2'

services:
  logspout:
    image: blockfi/logspout-humio:latest
    hostname: my.message.source
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    command: humio://cloud.us.humio.com:443
    restart: unless-stopped

Don't forget to prune containers on your system docker system prune --volumes.

Configuration

Logspout

• http.proxy - proxy URI for requests
• http.buffer.capacity - bytes of log data to buffer before shipping it to Humio
• http.buffer.timeout - timeout after which log data ships to Humio regardless of amount buffered
• http.gzip - compress data for shipping
• http.crash - crash or error message on failed request

Environment Variables

• HUMIO_INGEST_TOKEN: the ingest token created for the repository you're targeting, required and unset.
• HUMIO_TLS: true - when true we expect the endpoint to require TLS authentication
• HUMIO_HEC_PATH: /api/v1/ingest/hec - appended to the URI, Humio HEC supports both the standard (/services/collector) and this Humio-specific.
• HUMIO_SOURCETYPE: docker - the sourcetype field in the JSON for each event will contain this value
• HUMIO_DOCKER_LABELS: - include the docker container labels in the events
• DEBUG: false - when true output more debugging messages

Development

• Edit, fix, etc. the code.

• Build locally for testing.

$ go get -d -v
$ go build -v -ldflags "-X main.Version=3.2.11"

• Edit the VERSION environment variable in the top of the Dockerfile to your new version.
• Commit your changes.

$ git commit -m "blah blah" .

Tag the commit.

$ git tag -f -a v0.0.1 -m "Functional, if not complete."

Push the commit and the tags to your remote branch.

$ git push --follow-tags

Build the Docker container according to the instructions above.

License

ASLv2. See License

Disclaimer

This image provided to you as-is, yadda yadda etc. YMMV, and that's not our fault. :)

Documentation

Index

• func BytesToString(data []byte) string
• func NewHumioAdapter(route *router.Route) (router.LogAdapter, error)
• type FieldTemplates
• type Format
• type HumioAdapter
  • func (a *HumioAdapter) Stream(logstream chan *router.Message)
• type HumioMessage
• type HumioMessageEvent
• type Message
  • func (m *Message) ContainerName() string
  • func (m *Message) ContainerNameSplitN(sep string, n int) []string
  • func (m *Message) Hostname() string
  • func (m *Message) Priority() syslog.Priority
  • func (m *Message) Render(format Format, tmpl *FieldTemplates) ([]byte, error)
  • func (m *Message) Timestamp() string

Functions

func BytesToString

func BytesToString(data []byte) string

func NewHumioAdapter

func NewHumioAdapter(route *router.Route) (router.LogAdapter, error)

NewHumioAdapter creates an HumioAdapter

Types

type FieldTemplates

type FieldTemplates struct {
	// contains filtered or unexported fields
}

FieldTemplates for rendering Syslog messages

type Format

type Format string

Format represents the RFC spec to use for messages

const (
	// Rfc5424Format is the modern syslog protocol format. https://tools.ietf.org/html/rfc5424
	Rfc5424Format Format = "rfc5424"
	// Rfc3164Format is the legacy BSD syslog protocol format. https://tools.ietf.org/html/rfc3164
	Rfc3164Format Format = "rfc3164"
)

type HumioAdapter

type HumioAdapter struct {
	// contains filtered or unexported fields
}

HumioAdapter is an adapter that POSTs logs to an HTTP endpoint

func (*HumioAdapter) Stream

func (a *HumioAdapter) Stream(logstream chan *router.Message)

Stream implements the router.LogAdapter interface

type HumioMessage

type HumioMessage struct {
	Time       string            `json:"time"`
	Source     string            `json:"source"`
	SourceType string            `json:"sourcetype"`
	Hostname   string            `json:"host"`
	Event      HumioMessageEvent `json:"event"`
}

HumioMessage is a simple JSON representation of the log message.

type HumioMessageEvent

type HumioMessageEvent struct {
	Message string            `json:"message"`
	Labels  map[string]string `json:"labels"`
}

type Message

type Message struct {
	*router.Message
}

Message extends router.Message for the syslog standard

func (*Message) ContainerName

func (m *Message) ContainerName() string

ContainerName returns the message's container name

func (*Message) ContainerNameSplitN

func (m *Message) ContainerNameSplitN(sep string, n int) []string

ContainerNameSplitN returns the message's container name sliced at most "n" times using "sep"

func (*Message) Hostname

func (m *Message) Hostname() string

Hostname returns the os hostname

func (*Message) Priority

func (m *Message) Priority() syslog.Priority

Priority returns a syslog.Priority based on the message source

func (*Message) Render

func (m *Message) Render(format Format, tmpl *FieldTemplates) ([]byte, error)

Render transforms the log message using the Syslog template

func (*Message) Timestamp

func (m *Message) Timestamp() string

Timestamp returns the message's syslog formatted timestamp