gourmet

README

Gourmet

An exquisite network traffic analysis framework
Fast, simple, and customizable

Overview

Features

• Libpcap support
• AF_PACKET support
• Zero copy packet processing (fast!)
• Automatic TCP stream reassembly
• Berkeley Packet Filter support (currently only for libpcap)
• Easily extendable through Go Plugins (see Analyzers section below)

Upcoming Features

• BPF support for AF_PACKET
• Binary release w/ command-line configuration

Usage

Gourmet is not yet finished. But if you would like to give it a test ride, you can do the following:

git clone https://github.com/gourmetproject/gourmet
cd gourmet
cp example_config/get_started_config.yml my_config.yml
docker-compose up --build

Make sure you change the interface argument in my_config.yml to the network interface on your host machine that you want capture traffic on (to know about config.yml see the Configration section below). Gourmet will log all captured traffic to gourmet.log.

Once your container is running, you can just open gourmet.log file to see what gourmet is capturing.

Basic configuration

You can specify configuration file explicitly by adding option -c <path/to/config.yml>. You can see a bunch of example you can get started with in the example_configs folder. Full documentation for the configuration file can be found in the official documentation.

Design

Written in Go

Gourmet is designed from the ground up in Go, the number one language developers want to learn in 2019. It utilizes Google's gopacket library to quickly decode and analyze large amounts of network traffic. Go makes it fast, easy to maintain, and not C/C++.

Highly Concurrent

One of Go's shining features is goroutines. Goroutines are simply functions that run concurrently with other functions. They are much more lightweight, flexible, and easy to work with than standard threads. Goroutines communicate with each other using channels. Channels make it extremely simple to synchronize multithreaded Go programs.

These two language paradigms dramatically improve the speed, memory efficiency, and simplicity of concurrently processing thousands, or even millions, of packets per second.

Easily Customized through Go plugins

Go 1.8, released in February 2017, introduced a new plugin build mode. This build mode allows Go programs (and C programs, through cgo) to export symbols that are loaded and resolved by other Go programs at runtime. The Gourmet Project uses plugins as a way to load custom analyzers passed to the Gourmet sensor at runtime through a YAML configuration file defined by the user. More information how developers can create their own analyzers as Go plugins can be found below.

Analyzers

The Gourmet Project consists of the core Gourmet network sensor and a multitude of common protocol analyzers implemented as Go plugins. We provide a simple interface for other third-party developers to create and share their own analyzers as Go plugins.

In order to create your own analyzer, you must implement the Analyzer interface. This interface is fully documented in the Gourmet documentation. A simple example can be found in the simple_analyzer repository.

In order to implement the interface, you must create a new struct that has a Filter and Analyze function.

Filter

The Filter function takes a *gourmet.Connection object pointer as a parameter, determines whether the analyzer should analyze the connection, and returns true or false. The logic contained within the Filter function should be as simple as possible to filter out irrelevant packets or TCP streams. For example, if you want to write an Analyzer that only looks at DNS traffic, then your filter function should return true if the source or destination port is 53, and false otherwise.

Analyze

The Analyze function takes a gourmet Connection object as a parameter, conducts whatever logic necessary to analyze that connection, and returns an implementation of the Result interface. A Result object can be any data structure you like, such as a string, map, array, or struct. The Result interface only requires you implement the Key function, which returns a string. This string is used as the key value when we add the Result object to the JSON log for the Connection.

Analyzer List

• HTTP Analyzer - Logs information about HTTP traffic
• DNS Analyzer - Logs information about DNS traffic
• Simple Analyzer - Logs the number of bytes in the connection payload
• Bedtime Analyzer - If a specificed domain (such as Netflix) was accessed between certain hours of the day, a Slack bot sends you a message
  • Good example of analyzers depending on other analyzers and using the init() function to maintain state.

Gourmet vs. Zeek (aka Bro)

It is no secret that Zeek is the top choice for network security monitoring. One of the goals of this project is to provide an alternative to Zeek. The table below illustrates some key differences between the two projects.

Feature	Gourmet	Zeek
Log format	Single JSON file; each connection is a root-level JSON object	Multiple CSV files; connection data across files is linked through connection UIDs
Language	Pure Go	Zeek scripting language as a wrapper around C/C++
Customization	Go Plugins	Zeek scripts
Production-ready	Not yet, work in progress	Yes
Open Source	Yes	Yes
Multithreaded	Yes	No (see Zeek Cluster)

Contact Us

Support Us

Documentation

Overview

Package gourmet is an extendable network analysis and intrusion detection system.

Gourmet is designed to be fast, simple, and customized. To customize your Gourmet sensor, you can implement existing analyzers, or create your own.

Usage With No Analyzers

By default, gourmet analyzes Ethernet packets and logs basic information about the connections. This information is contained in a Connection type. This Connection type is marshalled into a JSON object and appended to the log file.

For UDP connections, each packet is transformed into a Connection object. However, for TCP connections, the stream is first reassembled and then turned into a Connection object.

Usage With Analyzers

If you wish to add an analyzer to Gourmet, you must add the analyzer repo URL to your config.yml file.

Creating Your Own Analyzer

Analyzers are an implementation of the Analyzer interface. They are written as a Go plugin. More information about Go plugins can be found here: https://golang.org/pkg/plugin.

Example custom analyzers can be found in the Gourmet Project repository at https://github.com/gourmetproject. simple_analyzer is the best one to start with.

Index

• func Start(config *Config)
• type Analyzer
• type Config
• type Connection
• type Result

Functions

func Start

func Start(config *Config)

Start is the entry point for Gourmet

Types

type Analyzer

type Analyzer interface {
	Filter(c *Connection) bool
	Analyze(c *Connection) (Result, error)
}

type Config

type Config struct {
	InterfaceType string `json:"type"`
	Interface     string
	Promiscuous   bool
	MaxCores      int `json:"max_cores"`
	ConnTimeout   int `json:"connection_timeout"`
	SnapLen       int `json:"snapshot_length"`
	Bpf           string
	LogFile       string `json:"log_file"`
	SkipUpdate    bool   `json:"skip_update"`
	Analyzers     map[string]interface{}
}

Config is the data structure used to expose Gourmet configuration settings to the user. Each of these fields have a default value, except for InterfaceType. For a list of default values and which values are allowed for each field, consult the web documentation at docs.gourmetproject.io

type Connection

type Connection struct {
	Timestamp       time.Time
	UID             uint64
	SourceIP        string
	SourcePort      int
	DestinationIP   string
	DestinationPort int
	TransportType   string
	Duration        float64
	State           string        `json:",omitempty"`
	Payload         *bytes.Buffer `json:"-"`
	Analyzers       map[string]interface{}
}

Connection contains basic information about an IP connection, including application layer bytes. If the connection is TCP-based, then the Connection contains basic information about the reassembled stream of packets for that TCP session.

A Connection is given to each Analyzer. The Result returned from an Analyzer is added to the Analyzers map for that Connection object. Once all Analyzers have been run against the Connection, it is marshaled as a JSON object into raw bytes and written to the log file.

type Result

type Result interface {
	Key() string
}