merger

Documentation

Overview

Package merger implements streaming network merge algorithm using nested iteration.

The merger processes networks from multiple MMDB databases, resolving overlaps by selecting the smallest network at each point. Adjacent networks with identical data are automatically merged for compact output. The streaming accumulator ensures O(1) memory usage regardless of database size.

Index

• type AccumulatedRange
• type Accumulator
  • func NewAccumulator(writer RowWriter, includeEmptyRows bool, pool *slicePool) *Accumulator
  • func (a *Accumulator) Flush() error
  • func (a *Accumulator) Process(prefix netip.Prefix, data []mmdbtype.DataType) error
• type Merger
  • func NewMerger(readers *mmdb.Readers, cfg *config.Config, writer RowWriter) (*Merger, error)
  • func (m *Merger) Merge() error
• type RangeRowWriter
• type RowWriter

Types

type AccumulatedRange

type AccumulatedRange struct {
	StartIP netip.Addr
	EndIP   netip.Addr
	Data    []mmdbtype.DataType // column values ordered by config.Columns
}

AccumulatedRange represents a continuous IP range with associated data.

type Accumulator

type Accumulator struct {
	// contains filtered or unexported fields
}

Accumulator accumulates adjacent networks with identical data and flushes them as CIDRs when data changes. This enables O(1) memory usage.

func NewAccumulator

func NewAccumulator(writer RowWriter, includeEmptyRows bool, pool *slicePool) *Accumulator

NewAccumulator creates a new streaming accumulator.

func (*Accumulator) Flush

func (a *Accumulator) Flush() error

Flush writes the current accumulated range as one or more CIDR rows. An accumulated range may produce multiple CIDRs if it doesn't align perfectly.

func (*Accumulator) Process

func (a *Accumulator) Process(prefix netip.Prefix, data []mmdbtype.DataType) error

Process handles an incoming network with its data. If the network is adjacent to the current accumulated range and has identical data, it extends the range. Otherwise, it flushes the current range and starts a new accumulation.

type Merger

type Merger struct {
	// contains filtered or unexported fields
}

Merger handles merging multiple MMDB databases into a single output stream.

func NewMerger

func NewMerger(readers *mmdb.Readers, cfg *config.Config, writer RowWriter) (*Merger, error)

NewMerger creates a new merger instance. Returns an error if database readers are missing or path normalization fails.

func (*Merger) Merge

func (m *Merger) Merge() error

Merge performs the streaming merge of all databases. It uses nested NetworksWithin iteration to find the smallest overlapping networks across all databases, then extracts data and streams to accumulator.

type RangeRowWriter

type RangeRowWriter interface {
	WriteRange(start, end netip.Addr, data []mmdbtype.DataType) error
}

RangeRowWriter can accept full start/end ranges instead of prefixes.

type RowWriter

type RowWriter interface {
	// WriteRow writes a single row with network prefix and column data.
	WriteRow(prefix netip.Prefix, data []mmdbtype.DataType) error
}

RowWriter defines the interface for writing output rows.