kafka-firehose-nozzle

README

kafka-firehose-nozzle

kafka-firehose-nozzle is CloudFoundry (CF) nozzle for Apache Kafka. It consumes data from the Loggregator Firehose and then publishes it to Apache Kafka.

The firehose generates events which are defined on dropsonde-protocol. You can set Kafka topic for each event type (e.g., by default, LogMessage events are publish to log-message topic). Events are encoded in protocol buffers between CF components but when to publish to kafka, events are decoded to plain json text.

NOTE: Currently we only support LogMessage and ValueMetric events (We will support others soon).

kafka-firehose-nozzle is written by Golang and built with rakutentech/go-nozzle package.

Usage

Basic usage is,

$ kafak-firehose-nozzle [options]

The following are available options,

-config PATH       Path to configuraiton file
-username NAME     username to grant access token to connect firehose
-password PASS     password to grant access token to connect firehose
-worker NUM        Number of producer worker. Default is number of CPU core
-subscription ID   Subscription ID for firehose. Default is 'kafka-firehose-nozzle'
-debug             Output event to stdout instead of producing message to kafka
-log-level LEVEL   Log level. Default level is INFO (DEBUG|INFO|ERROR)

You can set password via UAA_PASSWORD environmental variable.

Configuration

You can configure it via .toml file. You can see the example and description of this configuration file in example directory.

Install

To install, you can use go get command,

$ go get github.com/rakutentech/kafka-firehose-nozzle

You can deploy this as Cloud Foundry application with go-buildpack.

Contribution

1. Fork (https://github.com/rakutentech/kafka-firehose-nozzle/fork)
2. Create a feature branch
3. Commit your changes
4. Rebase your local changes against the master branch
5. Run test suite with the make test-all command and confirm that it passes
6. Create a new Pull Request

Author

Taichi Nakashima

Documentation

Overview

kafka-firehose-nozzle is Cloud Foundry nozzle which forwards logs from the loggeregagor firehose to Apache kafka (http://kafka.apache.org/).

Usage:

kafak-firehose-nozzle [options]

Available options:

-config PATH       Path to configuraiton file
-username NAME     username to grant access token to connect firehose
-password PASS     password to grant access token to connect firehose
-worker NUM        Number of producer worker. Default is number of CPU core
-subscription ID   Subscription ID for firehose. Default is 'kafka-firehose-nozzle'
-debug             Output event to stdout instead of producing message to kafka
-log-level LEVEL   Log level. Default level is INFO (DEBUG|INFO|ERROR)