dedup

Documentation

Index

• func HashMD5(pkt gopacket.Packet) []byte
• type CircularDedup
  • func NewCircularDedup(max int, duration time.Duration) *CircularDedup
  • func (cd *CircularDedup) Drop(pkt gopacket.Packet) (found bool)
• type Dedupper
• type TrivialDedup
  • func (d *TrivialDedup) Drop(pkt gopacket.Packet) (found bool)

Functions

func HashMD5

func HashMD5(pkt gopacket.Packet) []byte

HashMD5 is inspired by packet deduplication in Arkime https://github.com/arkime/arkime/blob/main/capture/dedup.c#L57

Types

type CircularDedup

type CircularDedup struct {
	// circular array of hash sets
	Buckets []map[string]bool
	// max number of buckets to be kept
	MaxBuckets int
	// max duration of each bucket
	Duration time.Duration
	// timestamp of latest bucket
	Bucket time.Time
}

CircularDedup maintains N buckets of hash sets hash lookup is done from all if hash is not found, add to latest rotate buckets if bucket duration has passed

func NewCircularDedup

func NewCircularDedup(max int, duration time.Duration) *CircularDedup

func (*CircularDedup) Drop

func (cd *CircularDedup) Drop(pkt gopacket.Packet) (found bool)

type Dedupper

type Dedupper interface {
	// Drop implements Dedupper
	Drop(gopacket.Packet) bool
}

Dedupper is a subsystem that accepts a gopacket type and reports if it has been already seen

type TrivialDedup

type TrivialDedup struct {
	Set  map[string]bool
	Hits uint64
}

TrivialDedup is only a minimal prototype for simple testing and SHOULD NOT BE USED It will leak memory and will likely see a high hash collision rate

func (*TrivialDedup) Drop

func (d *TrivialDedup) Drop(pkt gopacket.Packet) (found bool)

Drop implements Dedupper