README
¶
gafka 
__ _
/ _| | |
__ _ __ _ | |_ | | __ __ _
/ _` | / _` | | _| | |/ / / _` |
| (_| | | (_| | | | | < | (_| |
\__, | \__,_| |_| |_|\_\ \__,_|
__/ |
|___/
A full ecosystem for kafka/redis/PubSub/ElasticSearch/Zookeeper/haproxy.
Hope it can help you.
Components
-
Elastic haproxy that sits in front of kateway.
-
A fully-managed real-time secure and reliable RESTful Cloud Pub/Sub streaming message/job service.
-
kateway job scheduler and webhook dispatcher.
-
Unified multi-datacenter multi-cluster kafka swiss-knife management console.
-
A handy zookeeper CLI that supports recursive operation without any dependency.
-
ElasticSearch console.
-
Kafka clusters body guard that emits health info to InfluxDB and exports key warnings to zabbix for alerting.
Install
export PATH=$PATH:$GOPATH/bin
#========================================
# install go-bindata and go annotations
#========================================
go install github.com/jteeuwen/go-bindata/go-bindata
go install github.com/funkygao/goannotation
#========================================
# install gafka
#========================================
go get github.com/funkygao/gafka
cd $GOPATH/src/github.com/funkygao/gafka
./build.sh -a # build all components
#========================================
# try the gafka command 'gk'
#========================================
gk -h
Status
Currently gafka manages:
- 4 data centers
- 50+ kafka clusters
- 100+ kafka brokers
- 500+ kafka topics
- 2000+ kafka partitions
- 10Billion messages per day
- peak load
- 1Million message per second
- 8TB transfered per hour
- 5TB redis
Documentation
¶
Overview ¶
Package gafka is a full ecosystem built for real-time cloud PubSub system.
Index ¶
Constants ¶
This section is empty.
Variables ¶
View Source
var ( // Version is the unified version of the whole gafka project. // Each component shares the same version info. Version = "unknown" // BuildId is the SCM commit id. BuildId = "?" // BuiltAt is the time when build.sh was run. BuiltAt = "1970" )
Functions ¶
This section is empty.
Types ¶
This section is empty.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
es
es is the ElasticSearch console.
|
es is the ElasticSearch console. |
|
kateway/bench
Pub/Sub benchmark webhookd is a webhook endpoint that is used to demo kateway webhook feature.
|
Pub/Sub benchmark webhookd is a webhook endpoint that is used to demo kateway webhook feature. |
|
kateway/gateway
Package gateway implements the full PubSub gateway features.
|
Package gateway implements the full PubSub gateway features. |
|
kateway/hh
Package hh provides a hinted handoff service for Pub.
|
Package hh provides a hinted handoff service for Pub. |
|
kateway/hh/disk
Package disk implements a disk-backend hinted handoff which uses raft for replication.
|
Package disk implements a disk-backend hinted handoff which uses raft for replication. |
|
kateway/hh/kafka
Package kafka implements a kafka-backend hinted handoff.
|
Package kafka implements a kafka-backend hinted handoff. |
|
kateway/inflight
Deprecated pkg Package inflight provides storage for manipulating inflight message offsets.
|
Deprecated pkg Package inflight provides storage for manipulating inflight message offsets. |
|
kateway/job
Package job implements the schedulable message(job) underlying storage.
|
Package job implements the schedulable message(job) underlying storage. |
|
kateway/job/mysql
Package mysql implements a job store with mysql as backend.
|
Package mysql implements a job store with mysql as backend. |
|
kateway/manager
Package manager bridges data with pubsub manager.
|
Package manager bridges data with pubsub manager. |
|
kateway/meta
Package meta manages the global topology information.
|
Package meta manages the global topology information. |
|
kguard/sos
Package sos is a web server that accepts SOS message from any gafka components.
|
Package sos is a web server that accepts SOS message from any gafka components. |
|
Package ctx provides configurations loading and exporting that is shared across gafka project.
|
Package ctx provides configurations loading and exporting that is shared across gafka project. |
|
diagnostics
|
|
|
agent
Package agent provides an HTTP endpoint for a program providing diagnostics and statistics for a given task.
|
Package agent provides an HTTP endpoint for a program providing diagnostics and statistics for a given task. |
|
Package mpool provides recycleable memory buffer to reduce gc.
|
Package mpool provides recycleable memory buffer to reduce gc. |
|
Package registry defines interfaces that kateway will use to register its running instance health and meta data.
|
Package registry defines interfaces that kateway will use to register its running instance health and meta data. |
|
Package sla provides the SLA definitions for topics.
|
Package sla provides the SLA definitions for topics. |
|
Package telemetry perists github.com/funkygao/go-metrics metrics.Registry to durable storage.
|
Package telemetry perists github.com/funkygao/go-metrics metrics.Registry to durable storage. |
|
Package zk is a helper lib that manages kafka cluster meta data and consumer meta data.
|
Package zk is a helper lib that manages kafka cluster meta data and consumer meta data. |
Click to show internal directories.
Click to hide internal directories.