svipul

README

Svipul
======

Svipul is a set of tools for gathering data from network devices. Today,
only through SNMP. In the future, the architecture should support multiple
protocols for collection.

The design goals of Svipul are, roughly:

- Scale horizontally, support massive scaling
- Shield network hardware from accidental or intentional flooding
- Theoretically replace all use cases of SNMP data collection
- Stateless worker(s), no concept of a set list of machines/boxes (e.g.:
  dynamically configured targets)
- Modularized architecture, no tight coupling.

This repository contains the SNMP worker and a simple convenience tool for
manually adding orders. The first parts written.

The worker relies on RabbitMQ to queue orders for individual polling-jobs.

The SNMP worker uses Skogul (https://github.com/telenornms/skogul) to
transmit the results. Skogul supports a large number of different protocols
and transformations, making Svipul storage-agnostic.

Testing
-------

Install go, make and RabbitMQ.

Run ``make`` (or ``make install``).

Start rabbitmq, and ``./svipul-snmp``.

Test stuff with ``./svipul-addjob docs/examples/orders/...`` (or make your
own order).

The default binary uses localhost and guest:guest for authentication of
rabbitmq, but this can be configured.

To do something reasonable with the output, you can configure the
skogul-part. See skogul's documentation for details. Svipul works by using
the skogul handler named "svipul", and otherwise uses everything you can do
with Skogul (except set up other receivers, because why would you?).

Current feature-set and stability
---------------------------------

Most of the work so far has gone into making the SNMP worker stable, fast
and support a set of key features. All other components are, at the moment,
non-existent. But because Svipul sends the results with Skogul and can take
orders, it is still fully possible to use it for SNMP polling.

The worker currently has the following basic features:

- A single worker will only ever perform a single request for a given
  target at a time: If multiple orders to poll a target is received at the
  same time, only one will be carried out on a worker. In the future, this
  exclusivity will be extended to include all workers.
- Fetch data using GET or BULK WALK
- Parse MIB files (using gosmi). Standard mibs are bundled. Others can be
  configured.
- SMI: You can request data by numeric OID or logical name, or a mix.
- Optimized "table gets"/GET element: If fetching data from a table, e.g.
  ifTable, Svipul can build a name<->index cache and fetch each counter
  with precise GET, even using regular expressions to match the elements in
  the table, e.g.: fetch ifHCInOctets/ifHCOutOctets for interfaces matching
  ge-.* and et-.*.
- Stateless, horizontally scaled

Stability and performance
-------------------------

Svipul is still considered experimental, but is slowly being put into
limited production.

We do not currently have hard data on how fast it is, or how stable. But
tests indicate that it is fast.

But if you are considering using it in production, you should probably
expect some surprises. Through the entire 0.1.x release, basically
everything can change.

Basic design
------------

Design is subject to change, massively.

.. image:: docs/pollng.drawio.png

The basic concept is to have independent pollers that listen for tasks on a
queue. A locking service is needed too, but not (currently) shown. The
locking service is to prevent independent pollers from working on the same
device.

Each poller then uses Skogul to report back. This initial implementation
has Skogul embeded, thus an external one is not strictly speaking needed.

Results are streamed to a pub/sub-style exchange on RabbitMQ, and multiple
listeners can wait for it.

Each component is meant to be largely independent, and the design is meant
as a reference design where one need not implement all of it for it to be
useful. E.g.: You can get away with a simple poller, a shell script for
scheduler and skogul writing to influxDB directly if need be, using what
already exists today.

Name
----

Svipul is a Valkyrie. Valkyries are cool. Working name until recently was
tpoll, but it just changed.

Documentation

Overview

Package svipul is an advanced library and toolset for polling large amounts of network devices, initially with SNMP, but conceptually with any relevant protocol.

It works, or will work, through a central work-distribution database/api, and multiple, largely independent pollers. The results are reported using Skogul.

Index

• Variables
• func Debug(v ...any)
• func Debugf(format string, v ...any)
• func Debugln(v ...any)
• func Fatal(v ...any)
• func Fatalf(format string, v ...any)
• func Fatalln(v ...any)
• func Init()
• func Log(v ...any)
• func Logf(format string, v ...any)
• func Logln(v ...any)
• func ParseConfig(f string) error
• type Node
• type Walker

Variables

var Config conf = conf{
	DefaultCommunity: "public",
	Workers:          10,
	Debug:            false,
	MibPaths:         []string{"mibs/modules"},
	MaxMapAge:        time.Second * 3600,
	Broker:           "amqp://guest:guest@localhost:5672/",
	OutputConfig:     "/etc/svipul/output.d/",
	MibModules: []string{
		"ADSL-LINE-MIB",
		"ADSL-TC-MIB",
		"ALARM-MIB",
		"ATM-FORUM-MIB",
		"ATM-MIB",
		"ATM-TC-MIB",
		"BGP4-MIB",
		"BRIDGE-MIB",
		"DIFFSERV-DSCP-TC",
		"DIFFSERV-MIB",
		"DISMAN-PING-MIB",
		"DISMAN-TRACEROUTE-MIB",
		"DLSW-MIB",
		"DRAFT-MSDP-MIB",
		"DS1-MIB",
		"DS3-MIB",
		"ENTITY-MIB",
		"ENTITY-STATE-MIB",
		"ENTITY-STATE-TC-MIB",
		"ESO-CONSORTIUM-MIB",
		"EtherLike-MIB",
		"ETHER-WIS",
		"FRAME-RELAY-DTE-MIB",
		"FR-MFR-MIB",
		"GGSN-MIB",
		"GMPLS-LSR-STD-MIB",
		"GMPLS-TC-STD-MIB",
		"GMPLS-TE-STD-MIB",
		"HCNUM-TC",
		"HOST-RESOURCES-MIB",
		"HOST-RESOURCES-TYPES",
		"IANA-ADDRESS-FAMILY-NUMBERS-MIB",
		"IANA-GMPLS-TC-MIB",
		"IANAifType-MIB",
		"IANA-RTPROTO-MIB",
		"IEEE8021-CFM-MIB",
		"IEEE8021-CFM-V2-MIB",
		"IEEE8021-PAE-MIB",
		"IEEE8021-TC-MIB",
		"IEEE8023-LAG-MIB",
		"IF-MIB",
		"IGMP-STD-MIB",
		"INET-ADDRESS-MIB",
		"INTEGRATED-SERVICES-MIB",
		"IP-FORWARD-MIB",
		"IP-MIB",
		"IPMROUTE-MIB",
		"IPMROUTE-STD-MIB",
		"IPV6-FLOW-LABEL-MIB",
		"IPV6-MIB",
		"IPV6-TC",
		"ISIS-MIB",
		"LLDP-MIB",
		"MPLS-L3VPN-STD-MIB",
		"MPLS-LSR-STD-MIB",
		"MPLS-TC-STD-MIB",
		"MPLS-TE-STD-MIB",
		"MPLS-VPN-MIB",
		"OPT-IF-MIB",
		"OSPF-MIB",
		"OSPF-TRAP-MIB",
		"OSPFV3-MIB",
		"P-BRIDGE-MIB",
		"PerfHist-TC-MIB",
		"PIM-MIB",
		"POWER-ETHERNET-MIB",
		"PPP-LCP-MIB",
		"PTOPO-MIB",
		"Q-BRIDGE-MIB",
		"RADIUS-ACC-CLIENT-MIB",
		"RADIUS-AUTH-CLIENT-MIB",
		"RMON2-MIB",
		"RMON-MIB",
		"RSTP-MIB",
		"SNA-SDLC-MIB",
		"SNMP-COMMUNITY-MIB",
		"SNMP-FRAMEWORK-MIB",
		"SNMP-MPD-MIB",
		"SNMP-NOTIFICATION-MIB",
		"SNMP-PROXY-MIB",
		"SNMP-TARGET-MIB",
		"SNMP-USER-BASED-SM-MIB",
		"SNMPv2-MIB",
		"SNMPv2-SMI",
		"SNMPv2-TC",
		"SNMPv2-TM",
		"SNMP-VIEW-BASED-ACM-MIB",
		"SONET-MIB",
		"SYSAPPL-MIB",
		"TCP-MIB",
		"TOKEN-RING-RMON-MIB",
		"TRANSPORT-ADDRESS-MIB",
		"TUNNEL-MIB",
		"UDP-MIB",
		"VPN-TC-STD-MIB",
		"VRRP-MIB",
		"VRRPV3-MIB"},
}

Functions

func Debug

func Debug(v ...any)

func Debugf

func Debugf(format string, v ...any)

func Debugln

func Debugln(v ...any)

func Fatal

func Fatal(v ...any)

func Fatalf

func Fatalf(format string, v ...any)

func Fatalln

func Fatalln(v ...any)

func Init

func Init()

func Log

func Log(v ...any)

func Logf

func Logf(format string, v ...any)

func Logln

func Logln(v ...any)

func ParseConfig

func ParseConfig(f string) error

Types

type Node

type Node struct {
	Key       string // original input key, kept for posterity
	Name      string // sysName
	Numeric   string // .1.3.6.1.2.1.1.1 // root
	Qualified string // .1.3.6.1.2.1.1.1.0 // "full", with index
	Format    string // 255a <-- possibly obsolete
	Type      *models.Type
	Lookedup  bool // True if key was not a pure OID
}

Node is a rendered SMI node, e.g.: the result of a lookup. Usually handled by the smierte sub-package, but needs to be defined up here to avoid circular dependencies

type Walker

type Walker interface {
	BulkWalk(node []Node, cb func(pdu gosnmp.SnmpPDU) error) error
}

Walker is an interface for performing a BulkWalk, without having to worry about the underlying session. Today, only the session-subpackage and session.Session type implements it. Since it's tied to both a Node and a gosnmp.SnmpPDU type, it's rather strongly connected to SNMP atm.