README
¶
Banshee
Banshee is a real-time anomalies(outliers) detection system for periodic metrics.
Requirements
- Go >= 1.4 and godep.
- Node and gulp.
Build
- Clone the repo.
- Build binary via
make. - Build static files via
make static.
Documentation
https://godoc.org/github.com/eleme/banshee
Statsd Backend
https://www.npmjs.com/package/statsd-banshee
Migrate from bell
https://godoc.org/github.com/eleme/banshee#hdr-Migrate_from_bell
Authors
License
MIT Copyright (c) 2015 - 2016 Eleme, Inc.
Documentation
¶
Overview ¶
Banshee is a real-time anomalies(outliers) detection system for periodic metrics.
Use Case ¶
We are using it to monitor our website and rpc services intefaces, including called frequency, response time and exception calls. Our services send statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including banshee, banshee analyzes current metrics with history data, calculates the trending and alerts us if the trending behaves anomalous.
For example, we have an api named get_user, this api's response time (in milliseconds) is reported to banshee from statsd every 10 seconds:
20, 21, 21, 22, 23, 19, 18, 21, 22, 20, ..., 300
Banshee will catch the latest metric 300 and report it as an anomaly.
Why don't we just set a fixed threshold instead (i.e. 200ms)? This may also works but it is boring and hard to maintain a lot of thresholds. Banshee will analyze metric trendings automatically, it will find the "thresholds" automatically.
Features ¶
1. Designed for periodic metrics. Reality metrics are always with periodicity, banshee only peeks metrics with the same "phase" to detect.
2. Multiple alerting rule configuration options, to alert via fixed-thresholds or via anomalous trendings.
3. Coming with anomalies visualization webapp and alerting rules admin panels.
4. Require no extra storage services, banshee handles storage on disk by itself.
Requirements ¶
1. Go >= 1.4 and godep.
2. Node and gulp.
Build ¶
1. Clone the repo.
2. Build binary via `make`.
3. Build static files via `make static`.
Command Line ¶
Usage:
banshee [-c config] [-d] [-v]
Flags:
-c config Load config from file. -d Turn on debug mode. -v Show version.
Configuration ¶
See package config.
Statsd Integration ¶
In order to forward metrics to banshee from statsd, we need to add the npm module statsd-banshee to statsd's banckends:
1. Install statsd-banshee on your statsd servers:
$ cd path/to/statsd $ npm install statsd-banshee
2. Add module statsd-banshee to statsd's backends in config.js:
{
, backends: ['statsd-banshee']
, bansheeHost: 'localhost'
, bansheePort: 2015
}
Migrate from bell ¶
Require bell.js v2.0+ and banshee v0.0.7+:
./migrate -from bell.db -to banshee.db -with-projs -with-users mv banshee.db path/to/storage/admin
Compontents ¶
Banshee have 4 compontents and they are running in the same process:
1. Detector is to detect incoming metrics with history data and store the results.
2. Webapp is to visualize the detection results and provides panels to manage alerting rules, projects and users.
3. Alerter is to send sms and emails once anomalies are found.
4. Cleaner is to clean outdated metrics from storage.
Alerting Sender ¶
See package alerter and alerter/exampleCommand.
Implementation Details ¶
1. Detection algorithms, see package detector.
2. Detector input net protocol, see package detector.
3. Storage, see package storage.
4. Filter, see package filter.
License ¶
MIT (c) eleme, inc.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Godeps
|
|
|
_workspace/src/github.com/golang/snappy
Package snappy implements the snappy block-based compression format.
|
Package snappy implements the snappy block-based compression format. |
|
_workspace/src/github.com/jinzhu/inflection
Package inflection pluralizes and singularizes English nouns.
|
Package inflection pluralizes and singularizes English nouns. |
|
_workspace/src/github.com/julienschmidt/httprouter
Package httprouter is a trie based high performance HTTP request router.
|
Package httprouter is a trie based high performance HTTP request router. |
|
_workspace/src/github.com/mattn/go-sqlite3
Package sqlite3 provides interface to SQLite3 databases.
|
Package sqlite3 provides interface to SQLite3 databases. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb
Package leveldb provides implementation of LevelDB key/value database.
|
Package leveldb provides implementation of LevelDB key/value database. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/cache
Package cache provides interface and implementation of a cache algorithms.
|
Package cache provides interface and implementation of a cache algorithms. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/comparer
Package comparer provides interface and implementation for ordering sets of data.
|
Package comparer provides interface and implementation for ordering sets of data. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/errors
Package errors provides common error types used throughout leveldb.
|
Package errors provides common error types used throughout leveldb. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/filter
Package filter provides interface and implementation of probabilistic data structure.
|
Package filter provides interface and implementation of probabilistic data structure. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/iterator
Package iterator provides interface and implementation to traverse over contents of a database.
|
Package iterator provides interface and implementation to traverse over contents of a database. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/journal
Package journal reads and writes sequences of journals.
|
Package journal reads and writes sequences of journals. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/memdb
Package memdb provides in-memory key/value database implementation.
|
Package memdb provides in-memory key/value database implementation. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/opt
Package opt provides sets of options used by LevelDB.
|
Package opt provides sets of options used by LevelDB. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/storage
Package storage provides storage abstraction for LevelDB.
|
Package storage provides storage abstraction for LevelDB. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/table
Package table allows read and write sorted key/value.
|
Package table allows read and write sorted key/value. |
|
_workspace/src/github.com/syndtr/goleveldb/leveldb/util
Package util provides utilities used throughout leveldb.
|
Package util provides utilities used throughout leveldb. |
|
Package alerter implements an alerter to send sms/email messages on anomalies found.
|
Package alerter implements an alerter to send sms/email messages on anomalies found. |
|
exampleCommand
Example alerter command to echo message to console.
|
Example alerter command to echo message to console. |
|
Package cleaner checks outdated data every certain interval and delete them from the storage.
|
Package cleaner checks outdated data every certain interval and delete them from the storage. |
|
Package config handles configuration parser and container.
|
Package config handles configuration parser and container. |
|
Package detector is a tcp server to detect anomalies.
|
Package detector is a tcp server to detect anomalies. |
|
Package filter implements fast wildcard like filtering based on suffix tree.
|
Package filter implements fast wildcard like filtering based on suffix tree. |
|
Package health implements the health statistic aggregation.
|
Package health implements the health statistic aggregation. |
|
Package models implements metric, index, rule, project, user modeling.
|
Package models implements metric, index, rule, project, user modeling. |
|
Package storage implements banshee's persistence storage.
|
Package storage implements banshee's persistence storage. |
|
admindb
Package admindb handles the admin storage on SQLite3.
|
Package admindb handles the admin storage on SQLite3. |
|
indexdb
Package indexdb handles the storage for indexes.
|
Package indexdb handles the storage for indexes. |
|
metricdb
Package metricdb handles the storage for metrics.
|
Package metricdb handles the storage for metrics. |
|
tool
|
|
|
migrate
Tool to migrate admin rules, projects and users from bell.
|
Tool to migrate admin rules, projects and users from bell. |
|
Package util provides util functions.
|
Package util provides util functions. |
|
assert
Package assert provides a method to assert booleans.
|
Package assert provides a method to assert booleans. |
|
log
Package log implements leveled logging.
|
Package log implements leveled logging. |
|
safemap
Package safemap implements a map like container with rw lock to keep groutine safety.
|
Package safemap implements a map like container with rw lock to keep groutine safety. |
|
Package webapp implements a simple http web server to visualize detection results and to manage alerting rules.
|
Package webapp implements a simple http web server to visualize detection results and to manage alerting rules. |