burrowx

command module
Version: v0.0.0-...-875b2bb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 27, 2020 License: Apache-2.0 Imports: 8 Imported by: 0

README

WARNING: This repo is not maintained anymore, please use https://github.com/danielqsj/kafka_exporter

burrowx - kafka offset lag monitor,stored by influxdb

A simple, lightweight kafka offset monitor, currently metrics stored by influxdb. Motivated by Burrow, but much faster and cleaner and more stable. burrowx is good integration with influxdb and grafana.

DemoView

consumer rate

burrowx with influxdb and granfana

Install
$ go get github.com/sundy-li/burrowx
$ cd $GOPATH/src/github.com/sundy-li/burrowx
$ go build && go install
Running burrowx
## new workspace for burrowx
mkdir -p /data/app/burrowx 
## cd to the workspace
cd /data/app/burrowx 
## cp the files to here
cp $GOPATH/bin/burrowx ./
cp -rf $GOPATH/src/github.com/sundy-li/burrowx/config ./

## you should create the burrowx database in influxdb manually
## then modify server.json file config and run it 
./burrowx
Docker

A Docker file is available which builds this project on top of an Alpine Linux image.

  1. Create your desired configuration files (server.yaml, logging.xml)
  2. Run docker build -t burrowx . It will include the server.yaml and logging.xml and automatically start service.
  3. Run the container on your favourite container platform
Test the data
  • Create a new test topic

        bin/kafka-topics.sh --create  --zookeeper localhost:2181 --replication-factor 1 --partitions 8 --topic test_burrowx_topic
    

    ``

  • Produce the data to the topic

    	for i in `seq 1 10000`;do echo "33" |  bin/kafka-console-producer.sh   --topic test_burrowx_topic --broker-list localhost:9092 ; sleep 1; done
    

    ``

  • Create a consumer to consume the data

    $ pip install kafka

     from kafka import KafkaConsumer
     consumer = KafkaConsumer('test_burrowx_topic', group_id='my_group2')
     for msg in consumer:
         print(msg)
    
     print("end")
    

Then you will find the data in the influxdb database burrowx .

Schema in influxdb
  • cluster : cluster name
  • topic : topic name
  • consumer_group : group name
  • partition : partition id
  • logsize : partition logsize
  • offsize : partition consumer offsize
  • lag : partition consumer log
Query Example
SELECT sum("logsize") FROM "consumer_metrics" WHERE ("cluster" = 'your_cluster' AND "topic" = 'your_topic' AND "consumer_group" = 'your_consumer') AND $timeFilter  GROUP BY time(10s) 

SELECT sum("offsize") FROM "consumer_metrics" WHERE ("cluster" = 'your_cluster' AND "topic" = 'your_topic' AND "consumer_group" = 'your_consumer') AND $timeFilter  GROUP BY time(10s) 

SELECT sum("lag") FROM "consumer_metrics" WHERE ("cluster" = 'your_cluster' AND "topic" = 'your_topic' AND "consumer_group" = 'your_consumer') AND $timeFilter  GROUP BY time(10s) 
Features
  • Light weight and extremely simple to use, metrics are stored in influxdb, and could be easily viewed on grafana
  • Only support kafka version >= 0.9.X, which stores the consumer offsets in the topic __consumer_offsets,if you are using kafka 0.8.X, try my previous repo https://github.com/shunfei/Dcmonitor

Documentation

The Go Gopher

There is no documentation for this package.

Source Files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL