README




Go Report Card License Golang Linux macOS windows GoDoc Homepage Documentation

The Netcap (NETwork CAPture) framework efficiently converts a stream of network packets into platform neutral type-safe structured audit records that represent specific protocols or custom abstractions. These audit records can be stored on disk or exchanged over the network, and are well suited as a data source for machine learning algorithms. Since parsing of untrusted input can be dangerous and network data is potentially malicious, implementation was performed in a programming language that provides a garbage collected memory safe runtime.

It was developed for a series of experiments in my bachelor thesis: Implementation and evaluation of secure and scalable anomaly-based network intrusion detection. Slides from my presentation at the Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities are available on researchgate.

The project won the 2nd Place at Kaspersky Labs SecurIT Cup 2018 in Budapest.

Netcap uses Google's Protocol Buffers to encode its output, which allows accessing it across a wide range of programming languages. Alternatively, output can be emitted as comma separated values, which is a common input format for data analysis tools and systems. The tool is extensible and provides multiple ways of adding support for new protocols, while implementing the parsing logic in a memory safe way. It provides high dimensional data about observed traffic and allows the researcher to focus on experimenting with novel approaches for detecting malicious behavior in network environments, instead of fiddling with data collection mechanisms and post processing steps. It has a concurrent design that makes use of multi-core architectures. The name Netcap was chosen to be simple and descriptive. The command-line tool was designed with usability and readability in mind, and displays progress when processing packets. The latest version offers 58 audit record types of which 53 are protocol specific and 5 are flow models.

For more details please refer to the Documentation, visit the Homepage and read the thesis.

A simple demonstration of generating audit records from a PCAP dump file, querying and displaying the collected information in various ways:

asciicast

And live operation decoding traffic from my wireless network interface, while I am surfing the web:

asciicast

Design Goals

  • memory safety when parsing untrusted input
  • ease of extension
  • output format interoperable with many different programming languages
  • concurrent design
  • output with small storage footprint on disk
  • maximum data availability
  • allow implementation of custom abstractions
  • rich platform and architecture support

Use Cases

  • monitoring honeypots
  • monitoring medical / industrial devices
  • research on anomaly-based detection mechanisms
  • Forensic data analysis

License

Netcap is licensed under the GNU General Public License v3, which is a very permissive open source license, that allows others to do almost anything they want with the project, except to distribute closed source versions. This license type was chosen with Netcaps research purpose in mind, and in the hope that it leads to further improvements and new capabilities contributed by other researchers on the long term.

Documentation

Overview

Netcap is a framework for network packet processing at the protocol level. Learn more at https://netcap.io.

Index

Constants

View Source
const DefaultBufferSize = 1024 * 1024 * 10 // 10MB

Variables

View Source
var Commit = "6759e53197c4df3fc56d6e769863cbe7fbd35db2"
View Source
var Version = "v0.4.7"

Netcap version.

Functions

func CloseFile

func CloseFile(outDir string, file *os.File, typ string) (name string, size int64)

CloseFile closes the netcap file handle and removes files that do only contain a header but no audit records

func CloseGzipWriters

func CloseGzipWriters(writers ...*gzip.Writer)

func Count

func Count(filename string) (count int64)

Count returns the total number of records found in an audit record file

func CreateFile

func CreateFile(name, ext string) *os.File

CreateFile is a wrapper to create new audit record file

func Dump

func Dump(c DumpConfig)

Dump reads the specified netcap file and dumps the output according to the configuration to stdout

func FlushWriters

func FlushWriters(writers ...flushableWriter)

func InitRecord

func InitRecord(typ types.Type) (record proto.Message)

InitRecord initializes a new record of the given type that conforms to the proto.Message interface if netcap is extended with new audit records they need to be added here as well

func NewHeader

func NewHeader(t types.Type, source, version string, includesPayloads bool) *types.Header

NewHeader creates and returns a new netcap audit file header

func PrintBuildInfo

func PrintBuildInfo()

PrintBuildInfo displays build information related to netcap

func PrintLogo()

PrintLogo prints the netcap logo

func RemoveAuditRecordFileIfEmpty

func RemoveAuditRecordFileIfEmpty(name string) (size int64)

RemoveAuditRecordFileIfEmpty removes the audit record file if it does not contain audit records

Types

type DumpConfig

type DumpConfig struct {
	Path          string
	Separator     string
	TabSeparated  bool
	Structured    bool
	Table         bool
	Selection     string
	UTC           bool
	Fields        bool
	JSON          bool
	MemBufferSize int
}

DumpConfig contains all possible settings for dumping an audit records

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader implements reading netcap files

func Open

func Open(file string, memBufSize int) (*Reader, error)

Open a file

func (*Reader) Close

func (r *Reader) Close() error

Close the file

func (*Reader) Next

func (r *Reader) Next(msg proto.Message) error

Next Message

func (*Reader) ReadHeader

func (r *Reader) ReadHeader() *types.Header

ReadHeader reads the file header

type Writer

type Writer struct {

	// Name of the associated audit record type
	Name string

	IsChanWriter bool
	// contains filtered or unexported fields
}

Writer is a structure that supports writing audit records to disk

func NewWriter

func NewWriter(name string, buffer, compress, csv bool, out string, writeChan bool, memBufferSize int) *Writer

NewWriter initializes and configures a new Writer

func (*Writer) Close

func (w *Writer) Close() (name string, size int64)

func (*Writer) GetChan

func (w *Writer) GetChan() <-chan []byte

GetChan returns a channel for receiving bytes

func (*Writer) IsCSV

func (w *Writer) IsCSV() bool

func (*Writer) Write

func (w *Writer) Write(msg proto.Message) error

func (*Writer) WriteCSV

func (w *Writer) WriteCSV(msg proto.Message) (int, error)

WriteCSV writes a csv record

func (*Writer) WriteCSVHeader

func (w *Writer) WriteCSVHeader(msg proto.Message) (int, error)

WriteCSVHeader writes a CSV record

func (*Writer) WriteHeader

func (w *Writer) WriteHeader(t types.Type, source string, version string, includesPayloads bool) error

func (*Writer) WriteProto

func (w *Writer) WriteProto(msg proto.Message) error

WriteProto writes a protobuf message

Directories

Path Synopsis
cmd
Provides a mechanism to collect network packets from a network interface on macOS, linux and windows
Provides a mechanism to collect network packets from a network interface on macOS, linux and windows
This package implements a simple reader and writer for streams of length-delimited byte records.
This package implements a simple reader and writer for streams of length-delimited byte records.
Implements encoders to transform network packets into protocol buffers for various protocols
Implements encoders to transform network packets into protocol buffers for various protocols
Implements IO primitives
Implements IO primitives
Implements mapping alerts from suricata to netcap audit records
Implements mapping alerts from suricata to netcap audit records
Provides primitives for instrumentation via prometheus
Provides primitives for instrumentation via prometheus
Contains the type definitions for the supported network protocols
Contains the type definitions for the supported network protocols