picap

README

picap is an example use case which collects network traffic and indexes it in Pilosa. It uses tools from the PDK to do this, and so might be a nice example if you want to use the PDK in your own architecture.

Note: a version of this functionality used to be included in the PDK and invoked as pdk net. It has been separated because of dependencies which made use of the PDK as a whole somewhat cumbersome. Namely, it requires libpcap development headers and doesn't cross-compile easily due to reliance on the gopacket library.

Pre-requisites

Install Go, dep, and Pilosa.

Install

go get github.com/pilosa/picap cd $GOPATH/src/github.com/pilosa/picap dep ensure go install ./cmd/picap

Use

You must be running a Pilosa cluster.

See picap -h for command line usage.

Functionality

When invoked, picap reads network packet data, either from an interface, or a pcap file. It extracts information from each packet and indexes that information in Pilosa. Each packet is assigned a new column in Pilosa, and picap extracts a variety of fields, not all of which are necessarily present. See the picap.Packet struct for the most up to date description of what data is extracted.

Picap uses a PDK Translator to maintain mappings between values and their Pilosa IDs. It also starts a proxy server which may be queried in place of Pilosa and will map back and forth between values and Pilosa IDs. For example:

12:52:04~$ curl -XPOST localhost:11000/index/net/query -d'TopN(frame=http-hostname, n=3)' | jq
{
  "results": [
    [
      {
        "Key": "pilosa.com",
        "Count": 1
      },
      {
        "Key": "example.com",
        "Count": 1
      },
      {
        "Key": "readthedocs.org",
        "Count": 1
      }
    ]
  ]
}

Normally the "Key" values in a TopN response are integers, but the proxy has mapped them back to the hostnames that they represent. Similarly on the query side:

curl -XPOST localhost:11000/index/net/query -d'Count(Bitmap(frame=http-hostname, rowID="pilosa.com"))' | jq
{
  "results": [
    1
  ]
}

Documentation

Index

• type Main
  • func NewMain() *Main
  • func (m *Main) NewNetSource() (*NetSource, error)
  • func (m *Main) Run() error
• type NetSource
  • func (n *NetSource) Record() (interface{}, error)
• type Packet

Types

type Main

type Main struct {
	Iface       string        `help:"Interface on which to listen."`
	Filename    string        `help:"File containing pcap data to read."`
	Snaplen     int32         `help:"Maximum number of bytes to capture per packet."`
	Promisc     bool          `help:"Put capture interface into promiscuous mode."`
	Timeout     time.Duration `help:"Timeout for capturing packets."`
	Concurrency int           `help:"Number of goroutines parsing packets."`
	PilosaHosts []string      `help:"Comma separated list of pilosa host:port"`
	Filter      string        `help:"BPF style filter for packet capture."`
	Index       string        `help:"Pilosa index name."`
	BindAddr    string        `help:"Local address for mapping proxy to bind."`
	BufSize     int           `help:"Buffer size for Pilosa importer."`
	MappingDir  string        `help:"Directory to store mapping data. Empty string uses a temp dir."`
	Debug       bool          `help:"Turn on debug logging."`
	Translator  string        `help:"How to store mappings. In memory(mem) or LevelDB(level)."`
}

func NewMain

func NewMain() *Main

NewMain constructs a Main with default values.

func (*Main) NewNetSource

func (m *Main) NewNetSource() (*NetSource, error)

func (*Main) Run

func (m *Main) Run() error

type NetSource

type NetSource struct {
	// contains filtered or unexported fields
}

func (*NetSource) Record

func (n *NetSource) Record() (interface{}, error)

type Packet

type Packet struct {
	Length   int
	NetProto string
	NetSrc   string
	NetDst   string

	TransProto string
	TransSrc   string
	TransDst   string

	TCP struct {
		FIN bool
		SYN bool
		RST bool
		PSH bool
		ACK bool
		URG bool
		ECE bool
		CWR bool
		NS  bool
	}

	AppProto string

	HTTP struct {
		Hostname  string
		UserAgent string
		Method    string
	}
}