Version: v0.1.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jan 28, 2020 License: BSD-3-Clause Imports: 7 Imported by: 0


Anomaly Detector

The anomaly service type implements anomaly detection for payloads. The anomaly detection algorithm is based on a statistical model for compression. The anomaly detection algorithm computes the relative complexity, K(payload | previous payloads), of a payload and then updates the statistical model. A running mean and standard deviation of the complexity is then computed using this algorithm. If the complexity of a payload is some number of deviations from the mean then it is an anomaly. An anomaly is a payload that is statistically significant relative to previous payloads. The anomaly detection algorithm uses real time learning, so what is considered an anomaly can change over time.

The available service settings are as follows:

Name Type Description
depth number The size of the statistical model. Defaults to 2

The available inputs for the request are as follows:

Name Type Description
payload JSON object A payload to do anomaly detection on

The available response outputs are as follows:

Name Type Description
complexity number How unusual the payload is in terms of standard deviations from the mean
count number The number of payloads that have been processed

A sample service definition is:

  "name": "Anomaly",
  "description": "Look for anomalies",
  "ref": "",
  "settings": {
    "depth": 3

An example step that invokes the above Anomaly service using payload is:

  "service": "Anomaly",
  "input": {
    "payload": "=$.payload.content"

Utilizing the response values can be seen in a response handler:

  "if": "($.Anomaly.outputs.count < 100) || ($Anomaly.outputs.complexity < 3)",
  "error": false,
  "output": {
    "code": 200,
    "data": "=$"




View Source
const (
	// CDF16Fixed is the shift for 16 bit coders
	CDF16Fixed = 16 - 3
	// CDF16Scale is the scale for 16 bit coder
	CDF16Scale = 1 << CDF16Fixed
	// CDF16Rate is the damping factor for 16 bit coder
	CDF16Rate = 5
	// CDF16Size is the size of the cdf
	CDF16Size = 256
	// CDF16Depth is the depth of the context tree
	CDF16Depth = 2


This section is empty.


func New


type Activity

type Activity struct {
	// contains filtered or unexported fields

Activity is an anomaly detector

func (*Activity) Eval

func (a *Activity) Eval(ctx activity.Context) (done bool, err error)

Eval executes the activity

func (*Activity) Metadata

func (a *Activity) Metadata() *activity.Metadata

Metadata return the metadata for the activity

type CDF16

type CDF16 struct {
	Root  *Node16
	Mixin [][]uint16

CDF16 is a context based cumulative distributive function model

func NewCDF16

func NewCDF16() *CDF16

NewCDF16 creates a new CDF16 with a given context depth

func (*CDF16) Model

func (c *CDF16) Model(ctxt *Context16) []uint16

Model gets the model for the current context

func (*CDF16) Update

func (c *CDF16) Update(s uint16, ctxt *Context16)

Update updates the model

type Complexity

type Complexity struct {

	// contains filtered or unexported fields

Complexity is an entorpy based anomaly detector

func NewComplexity

func NewComplexity(depth int) *Complexity

NewComplexity creates a new entorpy based model

func (*Complexity) Complexity

func (c *Complexity) Complexity(input []byte) (float32, int)

Complexity outputs the complexity

type Context16

type Context16 struct {
	Context []uint16
	First   int

Context16 is a 16 bit context

func NewContext16

func NewContext16(depth int) *Context16

NewContext16 creates a new context

func (*Context16) AddContext

func (c *Context16) AddContext(s uint16)

AddContext adds a symbol to the context

func (*Context16) ResetContext

func (c *Context16) ResetContext()

ResetContext resets the context

type Input

type Input struct {
	Payload interface{} `md:"payload"`

func (*Input) FromMap

func (r *Input) FromMap(values map[string]interface{}) error

func (*Input) ToMap

func (r *Input) ToMap() map[string]interface{}

type Node16

type Node16 struct {
	Model    []uint16
	Children map[uint16]*Node16

Node16 is a context node

func NewNode16

func NewNode16() *Node16

NewNode16 creates a new context node

type Output

type Output struct {
	Complexity float32 `md:"complexity"`
	Count      int     `"md:"count`

func (*Output) FromMap

func (o *Output) FromMap(values map[string]interface{}) error

func (*Output) ToMap

func (o *Output) ToMap() map[string]interface{}

type Settings

type Settings struct {
	Depth int `md:"depth"`


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL