skogul

README

.. image:: https://goreportcard.com/badge/github.com/telenornms/skogul
   :target: https://goreportcard.com/report/github.com/telenornms/skogul

.. image:: https://godoc.org/github.com/telenornms/skogul?status.svg
   :target: https://godoc.org/github.com/telenornms/skogul

.. image:: https://cloud.drone.io/api/badges/telenornms/skogul/status.svg
   :target: https://cloud.drone.io/telenornms/skogul

======================================
Skogul - generic metric/data collector
======================================

Skogul is a generic tool for moving metric data around. It can serve as a
collector of data, but is primarily designed to be a framework for building
bridges between data collectors and storage engines.

This repository contains the Skogul library/package, and ``cmd/skogul``,
which parses a JSON-config to set up Skogul.

The release also includes extensive documentation autogenerated, provided
both as a manual page (``man skogul``), and RST in ``docs/skogul.rst`` once
built.

See also the examples in `docs/examples/basics`_ 

.. contents:: Table of contents
   :depth: 2
   :local:

Quickstart - RPM
----------------

If you're on CentOS/RHEL 7 or newer, you should use our naively built RPM,
available at https://github.com/telenornms/skogul/releases/latest.

It will install skogul, set up a systemd service and install a simple
configuration in ``/etc/skogul/conf.d/default.json``.

There's also a 64-bit Linux build there, which should work for most
non-RPM-based installations. But we make no attempt to really maintain it.

Quickstart - Docker
-------------------

A docker image is published to ghcr.io/telenornms/skogul as of version
v0.17.0. As of version v0.18.0, it is also published as "latest".
See the container image repository page for details:
https://github.com/telenornms/skogul/pkgs/container/skogul

Quickstart - source
-------------------

Building from source is not difficult. First you need Golang. Get it at 
https://golang.org/dl/ (I think you want go 1.19 or newer).

Building ``skogul``, including cloning::

   $ git clone https://github.com/telenornms/skogul
   (...)
   $ make
   $ make install

You don't have to use make - there's very little magic involved for regular
building, but it will ensure ``-version`` works, along with building the
auto-generated documentation.

Running ``make install`` installs the binary and default configuration, but
does NOT install any systemd unit or similar.

Also see ``make help`` for other make targets.

About
-----

Skogul collects, mutates and transmits metric data. It was written to
support an extensive eco system of data collectors and storage engines in
constant motion. With Skogul, the goal is to disconnect how data is
collected from how it is stored: If you decide to change storage engine,
you should not have to even touch your collector. Or vice versa.

It can be used for very simple setups, and expanded to large,
multi-datacenter infrastructures with a mixture of new and old systems
attached to it.

To accomplish this, you set up chains that define how data is received, how
it is treated, where it goes and what happens if something goes wrong.

A Skogul chain is built from one or more independent receivers which
receive data and pass it on to a sender. A sender can either transmit data
to an external system (including an other Skogul instance), or add some
internal routing logic before passing it on to one or more other senders.

.. image:: docs/imgs/basic.png

Unlike most APIs or collectors of metrics, Skogul does NOT have a
preference when it comes to storage engine. It is explicitly designed to
disconnect the task of how data is collected from how it is stored.

The rationale is that the problem of writing an efficient SNMP collector
should not be tightly coupled to where you store the data. And where you
store the data should not be tightly coupled with how you receive it, or
what you do with it.

This enables an organization to gradually shift from older to newer stacks,
as Skogul can both receive data on old and new transport mechanisms,
and store it both in new and old systems. That way, older collectors can
continue working how they always how worked, but send data to Skogul.
During testing/maturing, Skogul can store the data in both legacy system
and replacement system. When the legacy system is removed, no change is
needed on the side of the collector.

Extra care has been put into making it trivial to write senders and
receivers.

See the package documentation over at godoc for development-related
documentation: 
https://godoc.org/github.com/telenornms/skogul

More discussion on architecture can be found in `docs/`.

Modules
-------

Skogul is all based around modules. We started out with three modules, but
today we have a total of five different module types. To see every module,
use ``skogul -help``, read the manual, or simply look in the receiver,
sender, parser, encoder and transformer directory and read the ``auto.go``
file there which lists all modules.

Each family of module has its own section in the JSON configuration file.

While all modules are fundamentally equal, receivers and senders are so
important that they are a little bit *more* equal, so we call them core
modules.

.. image:: docs/imgs/modules-drawio.png

............
Core modules
............

Senders and receivers
.....................

There are two essential module types, the receiver which defines how data
is acquired, and senders that determine what to do with the data. A handler
is just a collection of parser, optional transformer and reference to the
first sender.

Commonly used receivers are the HTTP receiver, UDP, kafka receiver
(consumer), various file receivers, SQL receiver

Senders all have the same general API, but come in two distinct types

External senders
................

External senders transmit data out of Skogul and are the classic and
easy-to-understand senders. Examples are InfluxDB sender to store data in
InfluxDB, UDP sender, Kafka sender (producer), SQL sender, MQTT, and more.

The debug or "print" sender is a little special: It just prints data to
stdout and is HIGHLY useful for testing.

Internal/Logic senders
......................

Logical senders are used internally to route or do something related with
data. The by far most important internal sender is the batch sender, which
accepts data, batches it into user-defined sizes, then passes them on to an
other sender. There are a large amount of small but important logical
senders that can be combined to form powerful chains.

Other dev favorites are: The count sender for getting statistics about how
much data passes through Skogul, the switch sender for sending data to
different other senders based on metadata, dupe sender for sending the same
data to multiple other senders, the null sender for simply discarding data,

...............
Support modules
...............

Additionally, three support-type modules exists:

Parsers
.......

The parser takes a set of bytes received and decodes it into a Skogul
internal container. E.g.: JSON decoding, protocol buffers for Juniper-data,
Influx Line protocol data, etc and is used by receivers through handlers.

Encoders
........

The encoder does the opposite: It takes an internal Skogul container and
encodes it as a byte stream for external tranmission. Today, only a small
amount of senders use encoders, as they are quite new, but they will be
used more extensively in the future. Currently, only JSON is supported.
More to come.

Transformers
............

Transformers are used to, you guessed it, transform or mutate parsed
containers. Typically used to re-arrange source data to better match target
data, to add metadata, or to sanitize data.

................
Implicit modules
................

All modules can be defined in configuration, but several modules have zero
configuration options, or very common options. E.g.: The `skogul` parser
doesn't require any configuration to work, the `debug` sender works fine
without any settings, the `now` transformer doesn't need any configuration
to add current time to a metric. To save you from having to define a whole
lot of empty modules, these type of modules can be referenced by their
implementation name (class, if you like) and an instance will be created
behind the scenes. These are listed as "auto modules" in the manual page.

E.g., without this feature::

        {
                "receivers": {
                        "foo": {
                                "type": "test",
                                "handler": "myhandler"
                        }
                },
		"handlers": {
			"myhandler": {
				"parser": "skogul",
				"sender": "debug"
			}
		},
                "parsers": {
                        "skogul": {
                                "type": "skogul"
                        }
                },
                "senders": {
                        "debug": {
                                "type": "debug"
                        }
                }
        }

But since the Skogul parser and the debug sender has no configuration, you
can just omit their definition and Skogul will implicitly create them for
you::

        {
                "receivers": {
                        "foo": {
                                "type": "test",
                                "handler": "myhandler"
                        }
                },
		"handlers": {
			"myhandler": {
				"parser": "json",
				"sender": "debug"
			}
		}
        }


Performance
-----------

Skogul is meant to scale well. Early tests on a laptop proved to work very
well:

.. image:: docs/imgs/skogul-rates.png

The above graph is from a very simple test on a laptop (with a quad core
i7), using the provided tester to write data to influxdb. It demonstrates
that despite well-known weaknesses at the time (specially in the
influx-writer), we're able to push roughly 600-800k values/s through
Skogul. This has since been exceeded.

This was an early test, and since then Skogul has been run in production on
large scale systems, and generally out-performs anything it communicates
with.

Name
----

Skogul is a Valkyrie. After extensive research (5 minutes on Wikipedia with
a cross-check on duckduckgo), this name was selected because it is
reasonably unique and is also a Valkyrie, like Gondul, a sister-project.

Hacking
-------

There is little "exotic" about Skogul hacking, so the following sections
are aimed mostly at people who are unfamiliar with Go.

A few sources for more documentation:

- docs/CODE_OF_CONDUCT.md
- docs/CONTRIBUTING
- docs/CODING
- doc.go

.......
Testing
.......

In short: Use ``make check``. It will run ``go test -short ./...`` and
various other checks. There's also ``make covergui`` to do coverage
analysis and open it in a browser.

``make check`` is run on every commit.

Use ``make fmtfix`` to fix formatting issues, which also makes sure to not
mess with the bundled/generated go files.

.............
Documentation
.............

Documentation comes in two forms. One is aimed at end-users. This is
provided mainly by adding proper labels to your data structures (see any
sender or receiver implementation), and through hard-coded text found in
``cmd/skogul/main.go``. In addition to this, stand-alone examples of setups
are provided in the ``examples/`` directory.

For development, documentation is written and maintained using code
comments and runnable examples, following the ``godoc`` approach. Some
architecture comments are kept in ``docs/``, but by and large,
documentation should be consumed from godoc.

See https://godoc.org/github.com/telenornms/skogul for the online
version, or use ``go doc github.com/telenornms/skogul`` or similar,
as you would any other go package.

Examples are part of the test suite and thus extracted from ``*_test.go``
where applicable. But aren't really used much.

Roadmap
-------

We are doing frequent releases on github. I honestly don't know why no 1.0
release has been made. Mainly lazyness, we've been "almost there" for 2+
years.

Overall, the core modules and the scaffolding is getting pretty good.

The bigger things moving right now except new modules is logging, which has
never been quite right, and dealing with some legacy/deprecation.

Similarly, test cases need to be refreshed. Tests are written very
isolated, and a good bit of spaghetti-logic has arisen. We have decent
coverage, but it's getting trickier to scale test case writing.

We also need better integration tests now that Skogul integrates with a
wide variety of services.

Other than that, there are modules to be written and features to be added
which are mostly a matter of what needs arise.

Documentation

Overview

Package skogul is a framework for receiving, processing and forwarding data, typically metric data or event-oriented data, at high throughput.

It is designed to be as agnostic as possible with regards to how it transmits data and how it receives it, and the processors in between need not worry about how the data got there or how it will be treated in the next chain.

This means you can use Skogul to receive data on a influxdb-like line-based TCP interface and send it on to postgres - or influxdb - without having to write explicit support, just set up the chain.

The guiding principles of Skogul is:

- Make as few assumptions as possible about how data is received

- Be stupid fast

End users should only need to worry about the cmd/skogul tool, which comes fully equipped with self-contained documentation.

Adding new logic to Skogul should also be fairly easy. New developers should focus on understanding two things:

1. The skogul.Container data structure - which is the heart of Skogul.

2. The relationship from receiver to handler to sender.

The Container is documented in this very package.

Receivers are where data originates within Skogul. The typical Receiver will receive data from the outside world, e.g. by other tools posting data to a HTTP endpoint. Receivers can also be used to "create" data, either test data or, for example, log data. When skogul starts, it will start all receivers that are configured.

Handlers determine what is done with the data once received. They are responsible for parsing raw data and optionally transform it. This is the only place where it is allowed to _modify_ data. Today, the only transformer is the "templater", which allows a collection of metrics which share certain attributes (e.g.: all collected at the same time and from the same machine) to provide these shared attributes in a template which the "templater" transformer then applies to all metrics.

Other examples of transformations that make sense are:

- Adding a metadata field

- Removing a metadata field

- Removing all but a specific set of fields

- Converting nested metrics to multiple metrics or flatten them

Once a handler has done its deed, it sends the Container to the sender, and this is where "the fun begins" so to speak.

Senders consist of just a data structure that implements the Send() interface. They are not allowed to change the container, but besides that, they can do "whatever". The most obvious example is to send the container to a suitable storage system - e.g., a time series database.

So if you want to add support for a new time series database in Skogul, you will write a sender.

In addition to that, many senders serve only to add internal logic and pass data on to other senders. Each sender should only do one specific thing. For example, if you want to write data both to InfluxDB and MySQL, you need three senders: The "MySQL" and "InfluxDB" senders, and the "dupe" sender, which just takes a list of other senders and sends whatever it receives on to all of them.

Today, Senders and Receivers both have an identical "Auto"-system, found in auto.go of the relevant directories. This is how the individual implementations are made discoverable to the configuration system, and how documentation is provided. Documentation for the settings of a sender/receiver is handled as struct tags.

Once more parsers/transformers are added, they will likely also use a similar system.

Index

• Variables
• func Assert(x bool, v ...interface{})
• func ConfigureLogger(requestedLoglevel string, logtimestamp bool, logFormat string)
• func ExtractNestedObject(object map[string]interface{}, keys []string) (map[string]interface{}, error)
• func GetCertPool(path string) (*x509.CertPool, error)
• func GetLogLevelFromString(requestedLevel string) logrus.Level
• func IsInf(f float32, sign int) bool
• func Logger(category, implementation string) *logrus.Entry
• func MissingArgument(field string) error
• func Now() time.Time
• type Container
  • func (c Container) Describe() string
  • func (c Container) String() string
  • func (c *Container) Validate(IgnorePartialErrors bool) error
• type Deprecated
• type Duration
  • func (d Duration) MarshalJSON() ([]byte, error)
  • func (d *Duration) UnmarshalJSON(b []byte) error
• type Encoder
• type EncoderRef
  • func (er *EncoderRef) MarshalJSON() ([]byte, error)
  • func (er *EncoderRef) UnmarshalJSON(b []byte) error
• type Handler
  • func (h *Handler) Handle(b []byte) error
  • func (h *Handler) Parse(b []byte) (*Container, error)
  • func (h *Handler) Send(c *Container) error
  • func (h *Handler) SetParser(p Parser) error
  • func (h *Handler) Transform(c *Container) error
  • func (h *Handler) TransformAndSend(c *Container) error
  • func (h Handler) Verify() error
• type HandlerRef
  • func (hr *HandlerRef) MarshalJSON() ([]byte, error)
  • func (hr *HandlerRef) UnmarshalJSON(b []byte) error
• type LoggerCopyHook
  • func (l *LoggerCopyHook) Fire(entry *logrus.Entry) error
  • func (l *LoggerCopyHook) Levels() []logrus.Level
• type Metric
  • func (m Metric) Describe() string
• type Module
• type ModuleMap
  • func (mm *ModuleMap) Add(item Module) error
  • func (mm ModuleMap) Lookup(name string) *Module
• type Parser
• type ParserRef
  • func (pr *ParserRef) MarshalJSON() ([]byte, error)
  • func (pr *ParserRef) UnmarshalJSON(b []byte) error
• type Receiver
• type Secret
  • func (s Secret) Expose() string
  • func (s Secret) String() string
• type Sender
• type SenderRef
  • func (sr *SenderRef) MarshalJSON() ([]byte, error)
  • func (sr *SenderRef) UnmarshalJSON(b []byte) error
• type Stats
• type Transformer
• type TransformerRef
  • func (tr *TransformerRef) MarshalJSON() ([]byte, error)
  • func (tr *TransformerRef) UnmarshalJSON(b []byte) error
• type Verifier

Variables

var AssertErrors int

AssertErrors counts the number of assert errors

var EncoderMap []*EncoderRef

EncoderMap keeps track of the named parsers.

var HandlerMap []*HandlerRef

HandlerMap keeps track of which named handlers exists. A configuration engine needs to iterate over this and back-fill the real handlers.

var Identity map[interface{}]string

Identity is used to map instances of modules to their configured name. E.g.: If you have 3 influx senders, the module can access skogul.Identity[] to distinguish between them. This is meant for logging and statistics. Note that this is independent of which type of module we're dealing with, since they all have unique addresses, while they don't have unique names (e.g.: You can have a receiver named "test" and a sender named "test" at the same time - it will still work fine).

var ParserMap []*ParserRef

ParserMap keeps track of the named parsers.

var SenderMap []*SenderRef

SenderMap is a list of all referenced senders. This is used during configuration loading and should not be used afterwards. However, it needs to be exported so skogul.config can reach it, and it needs to be outside of skogul.config to avoid circular dependencies.

var TransformerMap []*TransformerRef

TransformerMap keeps track of the named transformers.

Functions

func Assert

func Assert(x bool, v ...interface{})

Assert panics if x is false, useful mainly for doing error-checking for "impossible scenarios" we can't really handle anyway.

Keep in mind that net/http steals panics, but you can check AssertErrors, which is incremented with each assert error encountered.

func ConfigureLogger

func ConfigureLogger(requestedLoglevel string, logtimestamp bool, logFormat string)

ConfigureLogger sets up the logger based on calling parameters

func ExtractNestedObject

func ExtractNestedObject(object map[string]interface{}, keys []string) (map[string]interface{}, error)

ExtractNestedObject extracts an object from a nested object structure. All intermediate objects has to map[string]interface{}

func GetCertPool

func GetCertPool(path string) (*x509.CertPool, error)

GetCertPool accepts a path to a directory of certificate authorities to trust. Pass in the empty string to use system defaults

func GetLogLevelFromString

func GetLogLevelFromString(requestedLevel string) logrus.Level

GetLogLevelFromString returns the matching logrus.Level from a string

func IsInf

func IsInf(f float32, sign int) bool

IsInf is a direct copy of Math.IsInf(), converted to 32-bit. This is some times needed when encodings use 32 bit floats (e.g.: Juniper telemetry). The following is the original documentation.

IsInf reports whether f is an infinity, according to sign. If sign > 0, IsInf reports whether f is positive infinity. If sign < 0, IsInf reports whether f is negative infinity. If sign == 0, IsInf reports whether f is either infinity.

func Logger

func Logger(category, implementation string) *logrus.Entry

Logger returns a logrus.Entry pre-populated with standard Skogul fields. category is the typical family of the code/module: sender/receiver/parser/transformer/core, while implementation is the local implementation (http, json, protobuf, udp, etc).

func MissingArgument

func MissingArgument(field string) error

MissingArgument provides a standard error message for use in Verify to report missing arguments.

func Now

func Now() time.Time

Now returns an approximation of time.Now atomically. It his is achieved by syncing time in a separate go routine to reduce overhead. Currently synced 10 times per second. The reason you may want this is:

BenchmarkTimeNow-8     	130269282	        45.6 ns/op
BenchmarkSkogulNow-8   	1000000000	         0.878 ns/op

If ten times per second isn't accurate enough, we may consider making this configurable. The primary use case here is for transformers or parsers that need to insert time, but might get thousands of metrics per second.

Types

type Container

type Container struct {
	Template *Metric   `json:"template,omitempty"`
	Metrics  []*Metric `json:"metrics"`
}

Container is the top-level object for one or more metric.

If a Template is provided, it will used as the initial value for each of the metrics - this is expanded by the transformers.Template transformers, and internal code does not need to worry about this.

A single Container instance is typically the result of a single POST to the HTTP receiver or similar.

func (Container) Describe

func (c Container) Describe() string

Describe returns key properties of the container useful for debugging without excessively spamming the log.

func (Container) String

func (c Container) String() string

String returns a stringified variant of the container XXX: Do not use? Use Describe instead. Should we just replace String XXX: with Describe?

func (*Container) Validate

func (c *Container) Validate(IgnorePartialErrors bool) error

Validate checks the validity of the container, verifying that it follows the exepcted spec. It should be called by any HTTP receiver after accepting a Container from an external source. It is NOT required nor recommended to use Validate in senders - the data is already validated by that time.

type Deprecated

type Deprecated interface {
	Deprecated() error
}

Deprecated is an optional interface for modules which, if implemented, allows modules to signal deprecated usage. If implemented, the Deprecated() method will be called after configuration is loaded and verified. Returning an error signals deprecation, and can be used both to deprecate specific settings and entire modules. There is currently no plan on how to remove deprecated modules over time.

type Duration

type Duration struct {
	time.Duration
}

Duration provides a wrapper around time.Duration to add JSON marshalling and unmarshalling. It is used whenever time.Duration needs to be exposed in configuration.

func (Duration) MarshalJSON

func (d Duration) MarshalJSON() ([]byte, error)

MarshalJSON provides JSON marshalling for Duration

func (*Duration) UnmarshalJSON

func (d *Duration) UnmarshalJSON(b []byte) error

UnmarshalJSON provides JSON unmarshalling for Duration

type Encoder

type Encoder interface {
	Encode(c *Container) ([]byte, error)
	EncodeMetric(m *Metric) ([]byte, error)
}

Encoder is an *optional* way to encode data, it is used by senders where data encoding can vary, but not all senders use it.

type EncoderRef

type EncoderRef struct {
	E    Encoder
	Name string
}

EncoderRef is a string mapping to an Encoder.

func (*EncoderRef) MarshalJSON

func (er *EncoderRef) MarshalJSON() ([]byte, error)

MarshalJSON for a reference just prints the name

func (*EncoderRef) UnmarshalJSON

func (er *EncoderRef) UnmarshalJSON(b []byte) error

UnmarshalJSON will unmarshal a encoder reference by creating a EncoderRef object and putting it on the EncoderMap list. The configuration system in question needs to iterate over EncoderMap after it has completed the first pass of configuration

type Handler

type Handler struct {
	Transformers          []Transformer
	Sender                Sender
	IgnorePartialFailures bool
	// contains filtered or unexported fields
}

Handler determines what a receiver will do with data received. It requires a parser to interperet the raw data, 0 or more transformers to mutate Containers, and a sender to call after data is parsed and mutated and ready to be dealt with.

Whenever a new Container is created, it should pass that to a Handler, not directly to a Sender. This goes for artificially created data too, e.g. if a sender wants to emit statistics. This ensures that transformers can be used in the future.

IngorePartialFailures should probably be renamed to removeinvalidmetrics or something like that, as that's closer to what it does.

To make it configurable, a HandlerRef should be used.

func (*Handler) Handle

func (h *Handler) Handle(b []byte) error

Handle parses the byte array using the configured parser, issues transformers and sends the data off.

func (*Handler) Parse

func (h *Handler) Parse(b []byte) (*Container, error)

Parse parses the bytes into a Container

func (*Handler) Send

func (h *Handler) Send(c *Container) error

Send validates the container and sends it to the configured sender

func (*Handler) SetParser

func (h *Handler) SetParser(p Parser) error

SetParser sets the parser to use for a Handler

func (*Handler) Transform

func (h *Handler) Transform(c *Container) error

Transform runs all available transformers

func (*Handler) TransformAndSend

func (h *Handler) TransformAndSend(c *Container) error

TransformAndSend transforms the already parsed container and sends the data off.

func (Handler) Verify

func (h Handler) Verify() error

Verify the basic integrity of a handler. Quite shallow.

type HandlerRef

type HandlerRef struct {
	H    *Handler
	Name string
}

HandlerRef references a named handler. Used whenever a handler is defined by configuration.

func (*HandlerRef) MarshalJSON

func (hr *HandlerRef) MarshalJSON() ([]byte, error)

MarshalJSON just returns the Name of the handler reference.

func (*HandlerRef) UnmarshalJSON

func (hr *HandlerRef) UnmarshalJSON(b []byte) error

UnmarshalJSON will create an entry on the HandlerMap for the parsed handler reference, so the real handler can be substituted later.

type LoggerCopyHook

type LoggerCopyHook struct {
	Writer *logrus.Logger
}

LoggerCopyHook is simply a wrapper around a logrus logger

func (*LoggerCopyHook) Fire

func (l *LoggerCopyHook) Fire(entry *logrus.Entry) error

Fire logs the log entry onto a copied logger to stdout

func (*LoggerCopyHook) Levels

func (l *LoggerCopyHook) Levels() []logrus.Level

Levels returns the levels this hook will support

type Metric

type Metric struct {
	Time     *time.Time             `json:"timestamp,omitempty"`
	Metadata map[string]interface{} `json:"metadata,omitempty"`
	Data     map[string]interface{} `json:"data,omitempty"`
}

Metric is a collection of measurements related to the same metadata and point in time.

A good example of a single metric is port statistics for a single interface on a router.

Both Metadata and Data is proided. The difference is generally in how data is accessed. Metadata is data you will search for - e.g. the name of the router, the name of the interface, etc. It is also possible to add more dynamic data as metadata, such as OS versions, but exactly how this will be handled will be up to underlying storage engines. E.g.: for influxdb, anything in metadata will be an (indexed) tag, so having reasonably rich metadata is perfectly fine, but you may want to keep an eye on the granularity.

A simple rule of thumb:

Metadata is what you search with.

Data is what you search for.

Example:

{
	"time": "2019-03-25T12:00:00Z",
	"metadata": {
		"device": "routera",
		"os": "JUNOS 15.4R1",
		"chassisId": "something"
	},
	"data": {
		"uptime": 124125124,
		"cputemp": 22
	}
}

It is possible to have nested data, however, it is NOT a requirement that a sender accepts this. And in general, it is better to "flatten" out data into multiple metrics instead. This can be done with a (custom) transformer.

Example of a nested structure:

{
	"time": "2019-03-25T12:00:00Z",
	"metadata": {
		"device": "routera",
		"os": "JUNOS 15.4R1",
		"chassisId": "something"
	},
	"data": {
		"ports": {
			"ge-0/0/0": {
				"ifHCInOctets":  5,
				"ifHCOutOctets": 10
			},
			"ge-0/0/1": {
				"ifHCInOctets":  2,
				"ifHCOutOctets": 20
			}
		}
	}
}

This is legal, but it's probably wise to use a transformer to change it into:

{
	"time": "2019-03-25T12:00:00Z",
	"metadata": {
		"device": "routera",
		"os": "JUNOS 15.4R1",
		"chassisId": "something",
		"port": "ge-0/0/0"
	},
	"data": {
		"ifHCInOctets":  5,
		"ifHCOutOctets": 10
		}
	}
},
{
	"time": "2019-03-25T12:00:00Z",
	"metadata": {
		"device": "routera",
		"os": "JUNOS 15.4R1",
		"chassisId": "something",
		"port": "ge-0/0/1"
	},
	"data": {
		"ifHCInOctets":  2,
		"ifHCOutOctets": 20
		}
	}
}

func (Metric) Describe

func (m Metric) Describe() string

Describe tries to briefly describe a metric.

FIXME: In addition to only including the first X fields, it should probably also look at raw size, as logging an entire telemetry packet is a bit much. On the other hand, this should be configurable as it's also useful...

type Module

type Module struct {
	Name     string             // short name of the module (e.g: "http")
	Aliases  []string           // optional aliases (e.g. "https")
	Alloc    func() interface{} // allocation of a blank module structure
	Extras   []interface{}      // Optional additional custom data structures that should be exposed in documentation.
	Help     string             // Human-readable help description.
	AutoMake bool               // If set, this module is auto-created with default variables if referenced by implementation name without being defined in config.
}

Module is metadata for a skogul module. It is used by the receiver, sender and transformer package. The Alloc() function must return a data structure that implements the relevant module interface, which is checked primarily in config.Parse.

See */auto.go for how to utilize this, and config/help.go, cmd/skogul/main.go for how to extract information/help, and ultimately config/parse.go for how it is applied.

type ModuleMap

type ModuleMap map[string]*Module

ModuleMap maps a name of a module to the Module data structure. Each type of module has its own module map. E.g.: receiver.Auto, sender.Auto and transformer.Auto.

func (*ModuleMap) Add

func (mm *ModuleMap) Add(item Module) error

Add adds a module to a module map, ensuring basic sanity and announcing it to the world, so to speak.

func (ModuleMap) Lookup

func (mm ModuleMap) Lookup(name string) *Module

Lookup will return a module if the name exists AND it should be autocreated. It is used during config loading to look up a module which is subsequently allocated.

FIXME: This should probably be a replaced by Make() which returns an allocated module using Alloc() in the future.

type Parser

type Parser interface {
	Parse(data []byte) (*Container, error)
}

Parser is the interface for parsing arbitrary data into a Container

type ParserRef

type ParserRef struct {
	P    Parser
	Name string
}

ParserRef is a string mapping to a Parser. It is used during configuration setup.

func (*ParserRef) MarshalJSON

func (pr *ParserRef) MarshalJSON() ([]byte, error)

MarshalJSON just returns the Name of the parser reference.

func (*ParserRef) UnmarshalJSON

func (pr *ParserRef) UnmarshalJSON(b []byte) error

UnmarshalJSON will create an entry on the ParserMap for the parsed parser reference, so the real parser can be substituted later.

type Receiver

type Receiver interface {
	Start() error
}

Receiver is how we get data. Receivers are responsible for getting raw data and the outer boundaries of a Container, but should explicitly avoid parsing raw data. This ensures that how data is transported is not bound by how it is parsed.

type Secret

type Secret string

Secret is a common type that wraps a string where the contents of the string can be sensitive, such as a credential. The String() func will output `***` to prevent accidental exposure, but the raw contents can be `Expose()`d.

func (Secret) Expose

func (s Secret) Expose() string

Expose must be called when the underlying secret is to be revealed, such as to the service that requires the data.

func (Secret) String

func (s Secret) String() string

String replaces the underlying data with the string "<redacted>" so that it is not accidentally revealed in logs or other debug related outputs.

type Sender

type Sender interface {
	Send(c *Container) error
}

Sender accepts data through Send() - and "sends it off". The canonical sender is one that implements a storage backend or outgoing API. E.g.: accept data, send to influx.

Senders are not allowed to modify the Container - there could be multiple goroutines running with same Container. If modification is required, the Sender needs to take a copy.

A sender should assume that the container has been validated, and is non-null. Slightly counter to common sense, it is NOT recommended to verify the input data again, since multiple senders are likely chained and will thus likely redo the same verifications.

Senders that pass data off to other senders should use a SenderRef instead, to facilitate configuration.

type SenderRef

type SenderRef struct {
	S    Sender
	Name string
}

SenderRef is a reference to a named sender. This is required to allow references to be resolved after all senders are loaded. Wherever a Sender is loaded from configuration, a SenderRef should be used in its place. The maintenance of the sender is handled in the configuration system.

func (*SenderRef) MarshalJSON

func (sr *SenderRef) MarshalJSON() ([]byte, error)

MarshalJSON for a reference just prints the name

func (*SenderRef) UnmarshalJSON

func (sr *SenderRef) UnmarshalJSON(b []byte) error

UnmarshalJSON will unmarshal a sender reference by creating a SenderRef object and putting it on the SenderMap list. The configuration system in question needs to iterate over SenderMap after it has completed the first pass of configuration

type Stats

type Stats interface {
	GetStats() *Metric
}

Stats is an optional interface for all skogul modules. It is used by modules to export internal stats about them, such as received data, specific errors, successes and sent metrics. The GetStats() method is called by the stats package for each configured module and the stats are sent onto the stats channel.

type Transformer

type Transformer interface {
	Transform(c *Container) error
}

Transformer mutates a collection before it is passed to a sender. Transformers should be very fast, but are the only means to modifying the data.

type TransformerRef

type TransformerRef struct {
	T    Transformer
	Name string
}

TransformerRef is a string mapping to a Transformer. It is used during configuration/transformer setup.

func (*TransformerRef) MarshalJSON

func (tr *TransformerRef) MarshalJSON() ([]byte, error)

MarshalJSON just returns the Name of the transformer reference.

func (*TransformerRef) UnmarshalJSON

func (tr *TransformerRef) UnmarshalJSON(b []byte) error

UnmarshalJSON will create an entry on the TransformerMap for the parsed transformer reference, so the real transformer can be substituted later.

type Verifier

type Verifier interface {
	Verify() error
}

Verifier is an *optional* interface for modules. If implemented, the configuration engine will issue Verify() after all configuration is parsed. The module should never modify state upon Verify(), but should simply check that internal state is usable.