gobgpdump

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 28, 2018 License: BSD-2-Clause Imports: 19 Imported by: 0

README

  1. Input 1.1) BGP updates gobgpdump can parse BGP updates and a large subset of BGP attributes. gobgpdump does not read from stdin as other analyzing tools do, all messages are read from a file or series of files. The input file should be a series of MRT messages in binary format.

     No special option must be used to read BGP update messages, message type
 is determined by MRT type and subtype.
 Example:
 gobgpdump <input file>

    1.2) RIB Dumps gobgpdump can also parse files containing RIB dumps. At the moment, only TABLE_DUMP_V2 is supported, and only AFI/SAFI subtypes are supported on it.

     Like bgp update messages, message type is determined from MRT type and
 subtype. However, in the case of a ribdump, if rib messages are mixed with
 bgp update messages in the same file, neither will be read correctly.
 Example:
 gobgpdump <input file>

    1.3) Multiple files 1.3.1) Command line gobpgdump can accept multiple input files as arguments. Any command line arguments not determined to be a part of the command flags will be considered an input file. Input files will be processed in the order they are provided on the command line. Example: gobgpdump <input file 1> <input file 2> etc. 1.3.2) The -conf option The -conf option was created to process large quantities of data files without having to specify all of them on the command line.
    The -conf option takes two arguments, which are used as templates to search the file system for designated files. No particular order is guaranteed with this system.

     	For more information on the -conf option, see README-config.md

    1.4) Bz2 If an input file is seen to have an extension of .bz2, it will be run through the bz2 decompression algorithm. Only the file extension is checked, never the file data. No special flag is used, gobgpdump will decide based off of file extensions. Note, when using the ID formatter, the output data will be the uncompressed version of the data.

  2. Output 2.1) Text The default option for a gobgdump output format is text. Depending on the input file recieved, it could look like: >MRT Header: --- >BGP4H Header: --- >BGP Header: --- >BGP Update: ---

     Or, if it were a rib file,
 MRT Header: ---
 RIB Header: ---
 RIB Entry: ---

 These will be repeatedly output for each entry in the input file to stdout
 unless redirected.
 Example:
 gobgpdump -fmtr text <input file>
 gobgpdump <input file>

    2.2) JSON JSON output is available for all supported protocols. The output is a series of messages (not an array) in valid JSON to stdout.

     Example:
 gobgpdump -fmtr json <input file>

    2.3) Protobuf gobgpdump does not currently have a user available option for dumping messages in protobuf format. However, the underlying library used to parse messages, protoparse, does parse directly into a protobuf.

     For more information, see github.com/CSUNetSec/protoparse

    2.4) ID This is the identity formatter. This will read in a binary file, and output the exact input data. This is mainly used for creating subsets of a file after applying some amount of filters to it. Example: gobgpdump -fmtr id 2.5) pup This formatter stands for print unique prefixes. With this option, gobgpdump produces no message output until it is finished parsing every message of every input file. After it is finished running, it prints a newline-separated list of every prefix seen at least once in one of the input files. If a prefix appears, and a parent prefix appears later in the input files, the child prefix is ignored. Example: gobgpdump -fmtr pup 2.6) pts This formatter stands for prefix time series. Like pup, this formatter records all unique prefixes, however this also records the timestamp and message number of every appearance of the highest level prefix. The output format is a gob, so redirecting outside of stdout is recommended. Example: gobgpdump -fmtr pts 2.7) Redirecting output gobgpdump produces three distinct varieties of output messages. The first is message output, which consists of only bgp or rib messages contained in input files. The second is statistical output, which contains details of messages filed, filters passed and the time taken to do so, and the last is log output. Log output is used to indicate any errors in parsing a message. Each output is directed to stdout by defualt. Each type of output may be redirected using a specific gobgpdump option, -o, -so, and -lo To redirect message output: gobgpdump -o 

     To redirect statistical output:
 gobgpdump -so <stat file> <input file>

 To redirect log output:
 gobgpdump -lo <log file> <input file>

 When working with a large quantity of input files, redirecting log output is
 recommended, as it can quickly clutter stdout.

    2.8) ML text output A textual formatter that prints one line per event, suitable for Machine Learning purposes.

  3. Filter options 3.1) Prefix filtering Possibly the most useful type of filtering, gobgpdump has the option of only outputting messages if a particular prefix appears in said message. If multiple prefixes are given, a message will pass on to output if any of the prefixes listed appear in the message. This is used on both BGP Update and RIB messages. Example: gobgpdump -prefixes 0.0.0.0/24 gobgpdump -prefixes 1.2.3.4/24,5.6.7.8/16 

     If a prefix string is improperly formatted, no message will pass, resulting in no output.

    3.2) src AS filtering Another option is to filter by src AS. This filter looks at the AS Path of every message in the input file, and if the last (source) AS in the AS path matches one of those given, the message will pass on to output. Example: gobgpdump -srcas 1234 gobgpdump -srcas 56,78 3.3) dest AS filtering The last filter option is by destination AS. This is the 0th AS in the AS path of every message. Example: gobgpdump -destas 1234 gobgpdump -destas 56,78

  4. Multicore options gobgpdump can operate with mulitple cores to dramatically increase the speed of a dump operation. Multiple cores can only be leveraged on multiple files, only 1 thread is ever operating on a single file. gobgpdump's concurrency option is accessed through the -wc option. This stands for worker count, and is the maximum number of threads to be launched by gobgpdump, with a cap of 16. Example: gobgpdump -wc 2 <input file 1> <input file 2>

  5. Complex examples This repository includes small example MRT files, uncompressed, in the /examples folder. These can be used to show the complex functionality of gobgpdump.

    If I wanted to read in an MRT file, find messages originating from autonomous system 6629 or 4847, and print all the prefixes advertized or withdrawn from that system and store it in a file called temp, I could run this: gobgpdump -src 4847,6629 -fmtr pup -o temp /examples/collector1/2017.01/arch0

    If I wanted to output every message that contained the prefix 153.2.224.0/24 as json, I could run: gobgpdump -prefixes 196.216.241.0/24 -fmtr json /examples/collector1/2017.02/arch1

    If I wanted to save every message originating from AS 4847 from multiple files as an MRT file for later analyzation in a file called 4847-messages, and I wanted to read the files with a max of 4 cores, I could run: gobgpdump -srcas 4847 -fmtr id -o 4847-messages -wc 4 examples/all/*

    If I was dealing with too many files in different directories to use wildcards effectively, I would use the configuration option. gobgpdump -conf example/example-formats example/conf-file For more information on how to build and modify those files, read README-config.md

Documentation

Overview

TODO: Add into the configuration option a list of allowed file extnsions, default being all, -conf option only

Index

Constants

This section is empty.

Variables

View Source 
var (
	DEBUG bool
)
View Source 
var EOP error = fmt.Errorf("End of paths")

This is the normal error returned by a stringsource, indicating there were no failures, there are just no more strings to recieve

Functions

func DumpWorker 

func DumpWorker(dc *DumpConfig, wg *sync.WaitGroup)

Simple worker function, launched in a new goroutine. Reads from stringsource and launches dumpfile

func GetMRTScanner 

func GetMRTScanner(fd *os.File) *bufio.Scanner

func NewMlFormatter 

func NewMlFormatter() mlFormatter

Types

type ConfigFile 

type ConfigFile struct {
	Collist  []string //List of collectors
	Start    string   // Start month		These first three are only used in configuration option, which is why they don't have flags
	End      string   //end month
	Lo       string   //Log output
	So       string   //Stat output
	Do       string   //dump output
	Wc       int      //worker count
	Fmtr     string   //output format
	Conf     bool     //get config from a file
	Srcas    string   `json:"Srcas,omitempty"`
	Destas   string   `json:"Destas,omitempty"`
	PrefList string   `json:"Prefixes,omitempty"`
	Debug    bool     // sets the global debug flag for the package
}

This is a struct to store all options in. This is just convenient so it can be read as a json object

type DayFormatter 

type DayFormatter struct {
	// contains filtered or unexported fields
}

func NewDayFormatter 

func NewDayFormatter(fd io.Writer) *DayFormatter

type DirectorySource 

type DirectorySource struct {
	// contains filtered or unexported fields
}

func NewDirectorySource 

func NewDirectorySource(dirs []string) *DirectorySource

func (*DirectorySource) Next 

func (ds *DirectorySource) Next() (string, error)

type DiscardCloser 

type DiscardCloser struct{}

func (DiscardCloser) Close 

func (d DiscardCloser) Close() error

func (DiscardCloser) Write 

func (d DiscardCloser) Write(data []byte) (n int, err error)

type DumpConfig 

type DumpConfig struct {
	// contains filtered or unexported fields
}

This struct is the complete parameter set for a file dump.

func GetDumpConfig 

func GetDumpConfig(configFile ConfigFile) (*DumpConfig, error)

func (*DumpConfig) CloseAll 

func (dc *DumpConfig) CloseAll()

func (*DumpConfig) GetWorkers 

func (dc *DumpConfig) GetWorkers() int

func (*DumpConfig) SummarizeAndClose 

func (dc *DumpConfig) SummarizeAndClose(start time.Time)

type Formatter 

type Formatter interface {
	// contains filtered or unexported methods
}

A Formatter takes the bufferstack and the underlying buffer and returns a representation of the data to be written to the dump file. The underlying buffer is necessary for the ID formatter

type IdentityFormatter 

type IdentityFormatter struct{}

Applies no formatting to the data But data is decompressed, may need to fix that However, golang bz2 doesn't have compression features

func NewIdentityFormatter 

func NewIdentityFormatter() IdentityFormatter

type JSONFormatter 

type JSONFormatter struct{}

Formats each update as a JSON message

func NewJSONFormatter 

func NewJSONFormatter() JSONFormatter

type MBSInfo 

type MBSInfo struct {
	// contains filtered or unexported fields
}

This contains miscellaneous info for the MBS structure

func NewMBSInfo 

func NewMBSInfo(file string, msg int) MBSInfo

type MultiWriteFile 

type MultiWriteFile struct {
	// contains filtered or unexported fields
}

The dump, stat, and log files are all accessed by multiple goroutines. This is a simple file wrapper to lock on a write, and unlock once the write is complete

func NewMultiWriteFile 

func NewMultiWriteFile(w io.WriteCloser) *MultiWriteFile

func (*MultiWriteFile) Close 

func (mwf *MultiWriteFile) Close() error

func (*MultiWriteFile) Write 

func (mwf *MultiWriteFile) Write(data []byte) (n int, err error)

func (*MultiWriteFile) WriteString 

func (mwf *MultiWriteFile) WriteString(s string) (n int, err error)

type PrefixEvent 

type PrefixEvent struct {
	Timestamp  time.Time
	Advertized bool
	ASPath     []uint32
}

type PrefixHistory 

type PrefixHistory struct {
	Pref string

	Events []PrefixEvent
	// contains filtered or unexported fields
}

func NewPrefixHistory 

func NewPrefixHistory(pref string, info MBSInfo, firstTime time.Time, advert bool, asp []uint32) *PrefixHistory

func (*PrefixHistory) String 

func (ph *PrefixHistory) String() string

type PrefixWalker 

type PrefixWalker struct {
	// contains filtered or unexported fields
}

type StringArray 

type StringArray struct {
	// contains filtered or unexported fields
}

Simple wrapper for a string array, so it can be accessed concurrently, and in the same way as a DirectorySource

func NewStringArray 

func NewStringArray(buf []string) *StringArray

func (*StringArray) Next 

func (sa *StringArray) Next() (string, error)

type TextFormatter 

type TextFormatter struct {
	// contains filtered or unexported fields
}

A simple text representation for the dump. The only formatter that needs the msgnum

func NewTextFormatter 

func NewTextFormatter() *TextFormatter

type UniquePrefixList 

type UniquePrefixList struct {
	// contains filtered or unexported fields
}

UniquePrefixList will look at all incoming messages, and output only the top level prefixes seen.

func NewUniquePrefixList 

func NewUniquePrefixList(fd io.Writer) *UniquePrefixList

type UniquePrefixSeries 

type UniquePrefixSeries struct {
	// contains filtered or unexported fields
}

UniquePrefixSeries does the same thing as UniquePrefixList, but rather than just a list, it will output a gob file containing each prefix and every event seen associated with that prefix

func NewUniquePrefixSeries 

func NewUniquePrefixSeries(fd io.Writer) *UniquePrefixSeries

Source Files

View all Source files

Directories

Path Synopsis
cmd
dumpanalyzer
gobgpdump

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.