components

package
v0.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 20, 2026 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Overview

Package components defines the protocol-layer decomposition of caphouse's columnar storage model.

Overview

Caphouse parses each frame into its constituent protocol layers and stores each layer in its own ClickHouse table. This makes packet data queryable at the column level while still allowing lossless PCAP reconstruction via [ReconstructFrame].

The central abstraction is the Component interface, which every protocol layer must implement. A Component knows how to:

- Encode a gopacket layer into ClickHouse-ready fields (INSERT path). - Reconstruct its bytes into a gopacket SerializableLayer (export path). - Describe its ClickHouse schema, column list, and scan order.

The PacketNucleus struct is the primary row in pcap_packets. It holds per-packet metadata (timestamps, lengths, component bitmask) shared by all layer components. Each component receives a copy of the nucleus via [LayerDecoder.ApplyNucleus] so it can store the (capture_id, packet_id) key.

Component kinds and order

Each component has a numeric kind constant (ComponentEthernet, ComponentIPv4, etc.) and an order constant (OrderL2Base, OrderL3Base, …) that determines decode order. The bitmask in PacketNucleus.Components records which kinds are present for a given packet. Use ComponentHas to test the mask.

Some orders are repeatable (e.g. OrderL2Tag for 802.1Q stacking, OrderL3Ext for IPv6 extension headers); see OrderRepeatable.

Registries

ComponentFactories maps each kind constant to a zero-value constructor. It is the authoritative list of all registered components and is used throughout caphouse to build schema, drive batch inserts, and generate SQL.

LayerEncoders maps gopacket LayerType values to the Component that handles encoding for that layer. It is used during ingest to select the right encoder for each layer returned by gopacket.

Creating a new component

To add support for a new protocol layer:

1. Define a struct with exported fields tagged with `ch:"column_name"`. Embed CaptureID, PacketID, and CodecVersion as the first three fields — these are required by the schema conventions:

type MyComponent struct { CaptureID uuid.UUID `ch:"capture_id"` PacketID uint64 `ch:"packet_id"` CodecVersion uint16 `ch:"codec_version"` // ... protocol fields }

2. Implement the Component interface. The ClickHouse column/value methods can delegate to GetClickhouseColumnsFrom and GetClickhouseValuesFrom, which use struct field tags via reflection:

func (c *MyComponent) ClickhouseColumns() ([]string, error) { return GetClickhouseColumnsFrom(c) } func (c *MyComponent) ClickhouseValues() ([]any, error) { return GetClickhouseValuesFrom(c) }

3. Write the ClickHouse schema as an embedded SQL file (e.g. my_schema.sql) and expose it via Schema():

//go:embed my_schema.sql var mySchemaSQL string

func (c *MyComponent) Schema(table string) string { return applySchema(mySchemaSQL, table) }

4. Register the component in ComponentFactories and LayerEncoders:

// In types.go, add a kind constant: ComponentMyProto uint = iota

// In registry.go, add to LayerEncoders: layers.LayerTypeXxx: &MyComponent{},

// In types.go, add to ComponentFactories: ComponentMyProto: func() Component { return &MyComponent{} },

The new component will be automatically picked up by schema initialisation, batch ingest, SQL generation, and packet reconstruction.

Compression

Splitting frames into narrow, homogeneous columns is what makes ClickHouse compression effective: each column contains values of the same type and similar range, so delta-encoding and LZ4/ZSTD can compress them far more aggressively than a raw frame blob. When designing a new component, prefer narrow integer fields and avoid storing redundant data that is already present in another component (e.g. do not re-store the Ethernet type in an IPv4 component).

Compression ratios per table and column can be measured with the `compression` build tag:

go test -v -tags compression ./...

See the "Working with Compression" section of the development docs for the ClickHouse query used to inspect per-column ratios and a worked example for the pcap_ethernet table.

Index

Constants

View Source 
const (
	ComponentRawFrame uint = iota
	ComponentEthernet
	ComponentDot1Q
	ComponentLinuxSLL
	ComponentIPv4
	ComponentIPv6
	ComponentIPv6Ext
	ComponentHash
	ComponentTruncated
	ComponentTCP
	ComponentUDP
	ComponentDNS
	ComponentNTP
	ComponentARP
	ComponentICMPv4
	ComponentICMPv6
	ComponentGRE
)
View Source 
const (
	TCPFlagFIN uint16 = 1 << iota
	TCPFlagSYN
	TCPFlagRST
	TCPFlagPSH
	TCPFlagACK
	TCPFlagURG
	TCPFlagECE
	TCPFlagCWR
	TCPFlagNS
)

TCP flag bit positions within the stored UInt16. Bits 0–7 map to byte 13 of the TCP header (CWR…FIN). Bit 8 is the NS flag from the low bit of byte 12.

View Source 
const (
	CodecVersionV1 uint16 = 1
	CodecProfileV1        = "nested-v1"
)
View Source 
const (
	OrderL2Base uint = iota
	OrderL2Tag
	OrderL3Base
	OrderL3Ext
	OrderL4Base
	OrderL7Base
)

Variables

View Source 
var ComponentFactories = map[uint]func() Component{
	ComponentEthernet: func() Component { return &EthernetComponent{} },
	ComponentDot1Q:    func() Component { return &Dot1QComponent{} },
	ComponentLinuxSLL: func() Component { return &LinuxSLLComponent{} },
	ComponentIPv4:     func() Component { return &IPv4Component{} },
	ComponentIPv6:     func() Component { return &IPv6Component{} },
	ComponentIPv6Ext:  func() Component { return &IPv6ExtComponent{} },
	ComponentTCP:      func() Component { return &TCPComponent{} },
	ComponentUDP:      func() Component { return &UDPComponent{} },
	ComponentDNS:      func() Component { return &DNSComponent{} },
	ComponentNTP:      func() Component { return &NTPComponent{} },
	ComponentARP:      func() Component { return &ARPComponent{} },
	ComponentICMPv4:   func() Component { return &ICMPv4Component{} },
	ComponentICMPv6:   func() Component { return &ICMPv6Component{} },
	ComponentGRE:      func() Component { return &GREComponent{} },
}

ComponentFactories maps component kind constants to zero-value constructors.

View Source 
var ErrShortFrame = errors.New("frame too short")
View Source 
var KnownComponentKinds = []uint{
	ComponentEthernet,
	ComponentDot1Q,
	ComponentLinuxSLL,
	ComponentIPv4,
	ComponentIPv6,
	ComponentIPv6Ext,
	ComponentTCP,
	ComponentUDP,
	ComponentDNS,
	ComponentNTP,
	ComponentARP,
	ComponentICMPv4,
	ComponentICMPv6,
	ComponentGRE,
}
View Source 
var LayerEncoders = map[gopacket.LayerType]Component{
	layers.LayerTypeEthernet:        &EthernetComponent{},
	layers.LayerTypeDot1Q:           &Dot1QComponent{},
	layers.LayerTypeLinuxSLL:        &LinuxSLLComponent{},
	layers.LayerTypeIPv4:            &IPv4Component{},
	layers.LayerTypeIPv6:            &IPv6Component{},
	layers.LayerTypeIPv6HopByHop:    &IPv6ExtComponent{},
	layers.LayerTypeIPv6Routing:     &IPv6ExtComponent{},
	layers.LayerTypeIPv6Fragment:    &IPv6ExtComponent{},
	layers.LayerTypeIPv6Destination: &IPv6ExtComponent{},
	layers.LayerTypeTCP:             &TCPComponent{},
	layers.LayerTypeUDP:             &UDPComponent{},
	layers.LayerTypeDNS:             &DNSComponent{},
	layers.LayerTypeNTP:             &NTPComponent{},
	layers.LayerTypeARP:             &ARPComponent{},
	layers.LayerTypeICMPv4:          &ICMPv4Component{},
	layers.LayerTypeICMPv6:          &ICMPv6Component{},
	layers.LayerTypeGRE:             &GREComponent{},
}
View Source 
var OrderRepeatable = map[uint]bool{
	OrderL2Tag: true,
	OrderL3Ext: true,
}

Functions

func ComponentHas 

func ComponentHas(mask *big.Int, bit uint) bool

func ComponentTable 

func ComponentTable(c Component) string

ComponentTable returns the ClickHouse table name for c ("pcap_" + c.Name()).

func CreateBatch 

func CreateBatch(
	ctx context.Context,
	conn clickhouse.Conn,
	components []clickhouseWriter,
) (chdriver.Batch, error)

CreateBatch creates a batch for an array of components _of single type_ into a ClickHouse table. Unknown behavior for components of different types. (probably very bad)

func GetClickhouseColumnsFrom 

func GetClickhouseColumnsFrom(v any) ([]string, error)

func GetClickhouseValuesFrom 

func GetClickhouseValuesFrom(v any) ([]any, error)

func GetDataColumnsFrom 

func GetDataColumnsFrom(v any, tableAlias string) ([]string, error)

GetDataColumnsFrom derives SELECT column expressions from a struct's ch: tags. capture_id and codec_version are always excluded.

When tableAlias is empty, plain column names are returned (packet_id included), suitable for ScanRow queries.

When tableAlias is non-empty, packet_id is also excluded and each column is expressed as "alias.col AS alias_col". If the column name already begins with alias+"_" the output alias is kept as the column name itself (avoiding double prefixes like arp_arp_op).

func LayerSupported 

func LayerSupported(layerType gopacket.LayerType) bool

func NewComponentMask 

func NewComponentMask(bits ...uint) *big.Int

Types

type ARPComponent 

type ARPComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`

	// HwType    [4]byte
	// AddrType  [4]byte
	ArpOp     uint16  `ch:"arp_op"`
	SenderMAC [6]byte `ch:"sender_mac"`
	SenderIP  net.IP  `ch:"sender_ip"`
	TargetMAC [6]byte `ch:"target_mac"`
	TargetIP  net.IP  `ch:"target_ip"`
}

ARPComponent stores parsed ARP header fields.

func (*ARPComponent) ApplyNucleus 

func (c *ARPComponent) ApplyNucleus(nucleus PacketNucleus)

func (*ARPComponent) ClickhouseColumns 

func (c *ARPComponent) ClickhouseColumns() ([]string, error)

func (*ARPComponent) ClickhouseValues 

func (c *ARPComponent) ClickhouseValues() ([]any, error)

func (*ARPComponent) DataColumns 

func (c *ARPComponent) DataColumns(tableAlias string) ([]string, error)

func (*ARPComponent) Encode 

func (c *ARPComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*ARPComponent) FetchOrderBy 

func (c *ARPComponent) FetchOrderBy() string

func (*ARPComponent) Index 

func (c *ARPComponent) Index() uint16

func (*ARPComponent) Kind 

func (c *ARPComponent) Kind() uint

func (*ARPComponent) LayerSize 

func (c *ARPComponent) LayerSize() int

func (*ARPComponent) Name 

func (c *ARPComponent) Name() string

func (*ARPComponent) Order 

func (c *ARPComponent) Order() uint

func (*ARPComponent) Reconstruct 

func (c *ARPComponent) Reconstruct(ctx *DecodeContext) error

func (*ARPComponent) Schema 

func (c *ARPComponent) Schema(table string) string

func (*ARPComponent) SetIndex 

func (c *ARPComponent) SetIndex(i uint16)

type ClickhouseMapper 

type ClickhouseMapper interface {
	ClickhouseColumns() ([]string, error)
	ClickhouseValues() ([]any, error)
	// DataColumns returns SELECT column expressions derived from the struct's
	// ch: tags, always excluding capture_id and codec_version.
	//
	// When tableAlias is empty, plain column names are returned (packet_id
	// included) — suitable for ScanRow SELECT queries.
	//
	// When tableAlias is non-empty, packet_id is also excluded and each
	// column is expressed as "alias.col AS alias_col" (or "alias.col AS col"
	// when the column name already begins with alias+"_"). The result is
	// suitable for use in a multi-component SELECT with LEFT JOINs.
	DataColumns(tableAlias string) ([]string, error)
}

ClickhouseMapper covers generic ClickHouse INSERT and SELECT column concerns for any type that is stored in ClickHouse.

type Component 

type Component interface {
	ClickhouseMapper
	LayerDecoder
	LayerEncoder
	// Name returns the short component alias (e.g. "ethernet", "ipv4").
	// The ClickHouse table name is "pcap_" + Name().
	Name() string
	Schema(table string) string
	FetchOrderBy() string
}

Component is the full interface for a registered component: it can encode a gopacket layer and be decoded/stored in ClickHouse.

func ExpandOne 

func ExpandOne(v Component, sessionID uint64, packetID uint32) []Component

ExpandOne copies v, sets SessionID and PacketID, and returns it as a single-element Component slice. Used by non-repeatable components after a NewScanBuf scan so that the scanner can be reused for the next row.

type DNSComponent 

type DNSComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`
	RawLen       uint16 `ch:"raw_len"`

	TransactionID uint16 `ch:"transaction_id"`
	QR            uint8  `ch:"qr"` // 0=query 1=response
	Opcode        uint8  `ch:"opcode"`
	RCode         uint8  `ch:"rcode"`
	Flags         uint8  `ch:"flags"` // Z<<4 | AA<<3 | TC<<2 | RD<<1 | RA

	ANCount uint16 `ch:"an_count"`
	NSCount uint16 `ch:"ns_count"`
	ARCount uint16 `ch:"ar_count"`

	QuestionsName  []string `ch:"questions_name"`
	QuestionsType  []uint16 `ch:"questions_type"`
	QuestionsClass []uint16 `ch:"questions_class"`

	// Answer section — one entry per RR.
	AnswersName  []string `ch:"answers_name"`
	AnswersType  []uint16 `ch:"answers_type"`
	AnswersClass []uint16 `ch:"answers_class"`
	AnswersTTL   []uint32 `ch:"answers_ttl"`
	AnswersRdata []string `ch:"answers_rdata"` // raw RDATA bytes stored as string
	AnswersIP    []string `ch:"answers_ip"`    // dotted IP for A/AAAA, "" otherwise

	// Authority section.
	AuthorityName  []string `ch:"authority_name"`
	AuthorityType  []uint16 `ch:"authority_type"`
	AuthorityClass []uint16 `ch:"authority_class"`
	AuthorityTTL   []uint32 `ch:"authority_ttl"`
	AuthorityRdata []string `ch:"authority_rdata"`

	// Additional section.
	AdditionalName  []string `ch:"additional_name"`
	AdditionalType  []uint16 `ch:"additional_type"`
	AdditionalClass []uint16 `ch:"additional_class"`
	AdditionalTTL   []uint32 `ch:"additional_ttl"`
	AdditionalRdata []string `ch:"additional_rdata"`
}

DNSComponent stores parsed DNS header fields, questions, and all answer/authority/additional resource records as flat parallel arrays. Reconstruction uses a custom DNS wire encoder so no raw frame bytes need to be stored. Reconstructed packets are semantically identical to the originals but will not use DNS name compression (RFC 1035 §4.1.4).

func (*DNSComponent) ApplyNucleus 

func (c *DNSComponent) ApplyNucleus(nucleus PacketNucleus)

func (*DNSComponent) ClickhouseColumns 

func (c *DNSComponent) ClickhouseColumns() ([]string, error)

func (*DNSComponent) ClickhouseValues 

func (c *DNSComponent) ClickhouseValues() ([]any, error)

func (*DNSComponent) DataColumns 

func (c *DNSComponent) DataColumns(tableAlias string) ([]string, error)

func (*DNSComponent) Encode 

func (c *DNSComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*DNSComponent) FetchOrderBy 

func (c *DNSComponent) FetchOrderBy() string

func (*DNSComponent) Index 

func (c *DNSComponent) Index() uint16

func (*DNSComponent) Kind 

func (c *DNSComponent) Kind() uint

func (*DNSComponent) LayerSize 

func (c *DNSComponent) LayerSize() int

func (*DNSComponent) Name 

func (c *DNSComponent) Name() string

func (*DNSComponent) Order 

func (c *DNSComponent) Order() uint

func (*DNSComponent) Reconstruct 

func (c *DNSComponent) Reconstruct(ctx *DecodeContext) error

func (*DNSComponent) Schema 

func (c *DNSComponent) Schema(table string) string

func (*DNSComponent) SetIndex 

func (c *DNSComponent) SetIndex(i uint16)

type DecodeContext 

type DecodeContext struct {
	Nucleus PacketNucleus
	Layers  []gopacket.SerializableLayer
	Offset  int
}

DecodeContext carries shared state while reconstructing a frame.

type Dot1QComponent 

type Dot1QComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`
	Priority     uint8  `ch:"priority"`
	DropEligible uint8  `ch:"drop_eligible"`
	VLANID       uint16 `ch:"vlan_id"`
	EtherType    uint16 `ch:"type"`
	// contains filtered or unexported fields
}

Dot1QComponent stores one vlan tag (repeatable).

func (*Dot1QComponent) ApplyNucleus 

func (c *Dot1QComponent) ApplyNucleus(nucleus PacketNucleus)

func (*Dot1QComponent) ClickhouseColumns 

func (c *Dot1QComponent) ClickhouseColumns() ([]string, error)

func (*Dot1QComponent) ClickhouseValues 

func (c *Dot1QComponent) ClickhouseValues() ([]any, error)

func (*Dot1QComponent) DataColumns 

func (c *Dot1QComponent) DataColumns(tableAlias string) ([]string, error)

func (*Dot1QComponent) Encode 

func (c *Dot1QComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*Dot1QComponent) ExportExpand 

func (c *Dot1QComponent) ExportExpand(sessionID uint64, packetID uint32) []Component

func (*Dot1QComponent) ExportScanTargets 

func (c *Dot1QComponent) ExportScanTargets() []any

func (*Dot1QComponent) FetchOrderBy 

func (c *Dot1QComponent) FetchOrderBy() string

func (*Dot1QComponent) Index 

func (c *Dot1QComponent) Index() uint16

func (*Dot1QComponent) Kind 

func (c *Dot1QComponent) Kind() uint

func (*Dot1QComponent) LayerSize 

func (c *Dot1QComponent) LayerSize() int

func (*Dot1QComponent) Name 

func (c *Dot1QComponent) Name() string

func (*Dot1QComponent) Order 

func (c *Dot1QComponent) Order() uint

func (*Dot1QComponent) Reconstruct 

func (c *Dot1QComponent) Reconstruct(ctx *DecodeContext) error

func (*Dot1QComponent) Schema 

func (c *Dot1QComponent) Schema(table string) string

func (*Dot1QComponent) SetIndex 

func (c *Dot1QComponent) SetIndex(index uint16)

type EthernetComponent 

type EthernetComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`
	SrcMAC       []byte `ch:"src"`
	DstMAC       []byte `ch:"dst"`
	EtherType    uint16 `ch:"type"`
	Length       uint16 `ch:"len"`
}

EthernetComponent stores raw ethernet header bytes.

func (*EthernetComponent) ApplyNucleus 

func (c *EthernetComponent) ApplyNucleus(nucleus PacketNucleus)

func (*EthernetComponent) ClickhouseColumns 

func (c *EthernetComponent) ClickhouseColumns() ([]string, error)

func (*EthernetComponent) ClickhouseValues 

func (c *EthernetComponent) ClickhouseValues() ([]any, error)

func (*EthernetComponent) DataColumns 

func (c *EthernetComponent) DataColumns(tableAlias string) ([]string, error)

func (*EthernetComponent) Encode 

func (c *EthernetComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*EthernetComponent) FetchOrderBy 

func (c *EthernetComponent) FetchOrderBy() string

func (*EthernetComponent) Index 

func (c *EthernetComponent) Index() uint16

func (*EthernetComponent) Kind 

func (c *EthernetComponent) Kind() uint

func (*EthernetComponent) LayerSize 

func (c *EthernetComponent) LayerSize() int

func (*EthernetComponent) Name 

func (c *EthernetComponent) Name() string

func (*EthernetComponent) Order 

func (c *EthernetComponent) Order() uint

func (*EthernetComponent) Reconstruct 

func (c *EthernetComponent) Reconstruct(ctx *DecodeContext) error

func (*EthernetComponent) Schema 

func (c *EthernetComponent) Schema(table string) string

func (*EthernetComponent) SetIndex 

func (c *EthernetComponent) SetIndex(i uint16)

type GREComponent 

type GREComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`

	Protocol uint16 `ch:"protocol"` // encapsulated EthernetType
	Flags    uint8  `ch:"flags"`
	Version  uint8  `ch:"version"`
	Checksum uint16 `ch:"checksum"`
	Key      uint32 `ch:"key"`
	Seq      uint32 `ch:"seq"`

	LayerIndex uint16 `ch:"layer_index"`
}

GREComponent stores parsed GRE header fields. Flags encodes optional-field presence as a bitmask: 

bit 0 = ChecksumPresent (+4 bytes: checksum + reserved)
bit 1 = KeyPresent      (+4 bytes)
bit 2 = SeqPresent      (+4 bytes)

func (*GREComponent) ApplyNucleus 

func (c *GREComponent) ApplyNucleus(nucleus PacketNucleus)

func (*GREComponent) ClickhouseColumns 

func (c *GREComponent) ClickhouseColumns() ([]string, error)

func (*GREComponent) ClickhouseValues 

func (c *GREComponent) ClickhouseValues() ([]any, error)

func (*GREComponent) DataColumns 

func (c *GREComponent) DataColumns(tableAlias string) ([]string, error)

func (*GREComponent) Encode 

func (c *GREComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*GREComponent) FetchOrderBy 

func (c *GREComponent) FetchOrderBy() string

func (*GREComponent) Index 

func (c *GREComponent) Index() uint16

func (*GREComponent) Kind 

func (c *GREComponent) Kind() uint

func (*GREComponent) LayerSize 

func (c *GREComponent) LayerSize() int

func (*GREComponent) Name 

func (c *GREComponent) Name() string

func (*GREComponent) Order 

func (c *GREComponent) Order() uint

func (*GREComponent) Reconstruct 

func (c *GREComponent) Reconstruct(ctx *DecodeContext) error

func (*GREComponent) Schema 

func (c *GREComponent) Schema(table string) string

func (*GREComponent) SetIndex 

func (c *GREComponent) SetIndex(i uint16)

type ICMPv4Component 

type ICMPv4Component struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`

	Type     uint8  `ch:"type"`
	Code     uint8  `ch:"code"`
	Checksum uint16 `ch:"checksum"`
	Id       uint16 `ch:"id"`
	Seq      uint16 `ch:"seq"`
}

ICMPv4Component stores the fixed 8-byte ICMPv4 header. The id and seq fields correspond to bytes 4–7 of the header; for non-echo message types these bytes carry type-specific data but are preserved exactly.

func (*ICMPv4Component) ApplyNucleus 

func (c *ICMPv4Component) ApplyNucleus(nucleus PacketNucleus)

func (*ICMPv4Component) ClickhouseColumns 

func (c *ICMPv4Component) ClickhouseColumns() ([]string, error)

func (*ICMPv4Component) ClickhouseValues 

func (c *ICMPv4Component) ClickhouseValues() ([]any, error)

func (*ICMPv4Component) DataColumns 

func (c *ICMPv4Component) DataColumns(tableAlias string) ([]string, error)

func (*ICMPv4Component) Encode 

func (c *ICMPv4Component) Encode(layer gopacket.Layer) ([]Component, error)

func (*ICMPv4Component) FetchOrderBy 

func (c *ICMPv4Component) FetchOrderBy() string

func (*ICMPv4Component) Index 

func (c *ICMPv4Component) Index() uint16

func (*ICMPv4Component) Kind 

func (c *ICMPv4Component) Kind() uint

func (*ICMPv4Component) LayerSize 

func (c *ICMPv4Component) LayerSize() int

func (*ICMPv4Component) Name 

func (c *ICMPv4Component) Name() string

func (*ICMPv4Component) Order 

func (c *ICMPv4Component) Order() uint

func (*ICMPv4Component) Reconstruct 

func (c *ICMPv4Component) Reconstruct(ctx *DecodeContext) error

func (*ICMPv4Component) Schema 

func (c *ICMPv4Component) Schema(table string) string

func (*ICMPv4Component) SetIndex 

func (c *ICMPv4Component) SetIndex(i uint16)

type ICMPv6Component 

type ICMPv6Component struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`

	Type     uint8  `ch:"type"`
	Code     uint8  `ch:"code"`
	Checksum uint16 `ch:"checksum"`
}

ICMPv6Component stores the fixed 4-byte ICMPv6 header (TypeCode + Checksum). The variable-length body (e.g. echo Id/Seq, NDP fields) is stored in the packet nucleus payload.

func (*ICMPv6Component) ApplyNucleus 

func (c *ICMPv6Component) ApplyNucleus(nucleus PacketNucleus)

func (*ICMPv6Component) ClickhouseColumns 

func (c *ICMPv6Component) ClickhouseColumns() ([]string, error)

func (*ICMPv6Component) ClickhouseValues 

func (c *ICMPv6Component) ClickhouseValues() ([]any, error)

func (*ICMPv6Component) DataColumns 

func (c *ICMPv6Component) DataColumns(tableAlias string) ([]string, error)

func (*ICMPv6Component) Encode 

func (c *ICMPv6Component) Encode(layer gopacket.Layer) ([]Component, error)

func (*ICMPv6Component) FetchOrderBy 

func (c *ICMPv6Component) FetchOrderBy() string

func (*ICMPv6Component) Index 

func (c *ICMPv6Component) Index() uint16

func (*ICMPv6Component) Kind 

func (c *ICMPv6Component) Kind() uint

func (*ICMPv6Component) LayerSize 

func (c *ICMPv6Component) LayerSize() int

func (*ICMPv6Component) Name 

func (c *ICMPv6Component) Name() string

func (*ICMPv6Component) Order 

func (c *ICMPv6Component) Order() uint

func (*ICMPv6Component) Reconstruct 

func (c *ICMPv6Component) Reconstruct(ctx *DecodeContext) error

func (*ICMPv6Component) Schema 

func (c *ICMPv6Component) Schema(table string) string

func (*ICMPv6Component) SetIndex 

func (c *ICMPv6Component) SetIndex(i uint16)

type IPv4Component 

type IPv4Component struct {
	SessionID uint64 `ch:"session_id"`
	PacketID  uint32 `ch:"packet_id"`

	CodecVersion uint16 `ch:"codec_version"`

	Protocol uint8 `ch:"protocol"`

	SrcIP4 netip.Addr `ch:"src"`
	DstIP4 netip.Addr `ch:"dst"`

	IPv4IHL         uint8  `ch:"ihl"`
	IPv4TOS         uint8  `ch:"tos"`
	IPv4TotalLen    uint16 `ch:"total_len"`
	IPv4ID          uint16 `ch:"id"`
	IPv4Flags       uint8  `ch:"flags"`
	IPv4FragOffset  uint16 `ch:"frag_offset"`
	IPv4TTL         uint8  `ch:"ttl"`
	IPv4HdrChecksum uint16 `ch:"hdr_checksum"`

	OptionsRaw []byte `ch:"options_raw"`

	LayerIndex uint16 `ch:"layer_index"`
}

IPv4Component stores parsed IPv4 fields, including any option bytes.

func (*IPv4Component) ApplyNucleus 

func (c *IPv4Component) ApplyNucleus(nucleus PacketNucleus)

func (*IPv4Component) ClickhouseColumns 

func (c *IPv4Component) ClickhouseColumns() ([]string, error)

func (*IPv4Component) ClickhouseValues 

func (c *IPv4Component) ClickhouseValues() ([]any, error)

func (*IPv4Component) DataColumns 

func (c *IPv4Component) DataColumns(tableAlias string) ([]string, error)

func (*IPv4Component) Encode 

func (c *IPv4Component) Encode(layer gopacket.Layer) ([]Component, error)

func (*IPv4Component) FetchOrderBy 

func (c *IPv4Component) FetchOrderBy() string

func (*IPv4Component) Index 

func (c *IPv4Component) Index() uint16

func (*IPv4Component) Kind 

func (c *IPv4Component) Kind() uint

func (*IPv4Component) LayerSize 

func (c *IPv4Component) LayerSize() int

func (*IPv4Component) Name 

func (c *IPv4Component) Name() string

func (*IPv4Component) Order 

func (c *IPv4Component) Order() uint

func (*IPv4Component) Reconstruct 

func (c *IPv4Component) Reconstruct(ctx *DecodeContext) error

func (*IPv4Component) Schema 

func (c *IPv4Component) Schema(table string) string

func (*IPv4Component) SetIndex 

func (c *IPv4Component) SetIndex(i uint16)

type IPv6Component 

type IPv6Component struct {
	SessionID uint64 `ch:"session_id"`
	PacketID  uint32 `ch:"packet_id"`

	CodecVersion uint16 `ch:"codec_version"`

	Protocol uint8 `ch:"protocol"`

	SrcIP6 netip.Addr `ch:"src"`
	DstIP6 netip.Addr `ch:"dst"`

	IPv6PayloadLen   uint16 `ch:"payload_len"`
	IPv6HopLimit     uint8  `ch:"hop_limit"`
	IPv6FlowLabel    uint32 `ch:"flow_label"`
	IPv6TrafficClass uint8  `ch:"traffic_class"`

	LayerIndex uint16 `ch:"layer_index"`
}

IPv6Component stores parsed IPv6 fields.

func (*IPv6Component) ApplyNucleus 

func (c *IPv6Component) ApplyNucleus(nucleus PacketNucleus)

func (*IPv6Component) ClickhouseColumns 

func (c *IPv6Component) ClickhouseColumns() ([]string, error)

func (*IPv6Component) ClickhouseValues 

func (c *IPv6Component) ClickhouseValues() ([]any, error)

func (*IPv6Component) DataColumns 

func (c *IPv6Component) DataColumns(tableAlias string) ([]string, error)

func (*IPv6Component) Encode 

func (c *IPv6Component) Encode(layer gopacket.Layer) ([]Component, error)

func (*IPv6Component) FetchOrderBy 

func (c *IPv6Component) FetchOrderBy() string

func (*IPv6Component) Index 

func (c *IPv6Component) Index() uint16

func (*IPv6Component) Kind 

func (c *IPv6Component) Kind() uint

func (*IPv6Component) LayerSize 

func (c *IPv6Component) LayerSize() int

func (*IPv6Component) Name 

func (c *IPv6Component) Name() string

func (*IPv6Component) Order 

func (c *IPv6Component) Order() uint

func (*IPv6Component) Reconstruct 

func (c *IPv6Component) Reconstruct(ctx *DecodeContext) error

func (*IPv6Component) Schema 

func (c *IPv6Component) Schema(table string) string

func (*IPv6Component) SetIndex 

func (c *IPv6Component) SetIndex(i uint16)

type IPv6ExtComponent 

type IPv6ExtComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`
	ExtType      uint16 `ch:"ext_type"`
	ExtRaw       []byte `ch:"ext_raw"`
	// contains filtered or unexported fields
}

IPv6ExtComponent stores one raw IPv6 extension header (repeatable).

func (*IPv6ExtComponent) ApplyNucleus 

func (c *IPv6ExtComponent) ApplyNucleus(nucleus PacketNucleus)

func (*IPv6ExtComponent) ClickhouseColumns 

func (c *IPv6ExtComponent) ClickhouseColumns() ([]string, error)

func (*IPv6ExtComponent) ClickhouseValues 

func (c *IPv6ExtComponent) ClickhouseValues() ([]any, error)

func (*IPv6ExtComponent) DataColumns 

func (c *IPv6ExtComponent) DataColumns(tableAlias string) ([]string, error)

func (*IPv6ExtComponent) Encode 

func (c *IPv6ExtComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*IPv6ExtComponent) ExportExpand 

func (c *IPv6ExtComponent) ExportExpand(sessionID uint64, packetID uint32) []Component

func (*IPv6ExtComponent) ExportScanTargets 

func (c *IPv6ExtComponent) ExportScanTargets() []any

func (*IPv6ExtComponent) FetchOrderBy 

func (c *IPv6ExtComponent) FetchOrderBy() string

func (*IPv6ExtComponent) Index 

func (c *IPv6ExtComponent) Index() uint16

func (*IPv6ExtComponent) Kind 

func (c *IPv6ExtComponent) Kind() uint

func (*IPv6ExtComponent) LayerSize 

func (c *IPv6ExtComponent) LayerSize() int

func (*IPv6ExtComponent) Name 

func (c *IPv6ExtComponent) Name() string

func (*IPv6ExtComponent) Order 

func (c *IPv6ExtComponent) Order() uint

func (*IPv6ExtComponent) Reconstruct 

func (c *IPv6ExtComponent) Reconstruct(ctx *DecodeContext) error

func (*IPv6ExtComponent) Schema 

func (c *IPv6ExtComponent) Schema(table string) string

func (*IPv6ExtComponent) SetIndex 

func (c *IPv6ExtComponent) SetIndex(i uint16)

type LayerDecoder 

type LayerDecoder interface {
	Kind() uint
	Order() uint
	Index() uint16
	SetIndex(uint16)
	LayerSize() int
	ApplyNucleus(PacketNucleus)
	Reconstruct(*DecodeContext) error
}

LayerDecoder defines the reconstruct contract for decoded layers.

type LayerEncoder 

type LayerEncoder interface {
	Encode(gopacket.Layer) ([]Component, error)
}

LayerEncoder defines the encoding contract for a gopacket layer.

type LinuxSLLComponent 

type LinuxSLLComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`
	L2Len        uint16 `ch:"l2_len"`
	L2HdrRaw     []byte `ch:"l2_hdr_raw"`
}

LinuxSLLComponent stores raw SLL header bytes.

func (*LinuxSLLComponent) ApplyNucleus 

func (c *LinuxSLLComponent) ApplyNucleus(nucleus PacketNucleus)

func (*LinuxSLLComponent) ClickhouseColumns 

func (c *LinuxSLLComponent) ClickhouseColumns() ([]string, error)

func (*LinuxSLLComponent) ClickhouseValues 

func (c *LinuxSLLComponent) ClickhouseValues() ([]any, error)

func (*LinuxSLLComponent) DataColumns 

func (c *LinuxSLLComponent) DataColumns(tableAlias string) ([]string, error)

func (*LinuxSLLComponent) Encode 

func (c *LinuxSLLComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*LinuxSLLComponent) FetchOrderBy 

func (c *LinuxSLLComponent) FetchOrderBy() string

func (*LinuxSLLComponent) Index 

func (c *LinuxSLLComponent) Index() uint16

func (*LinuxSLLComponent) Kind 

func (c *LinuxSLLComponent) Kind() uint

func (*LinuxSLLComponent) LayerSize 

func (c *LinuxSLLComponent) LayerSize() int

func (*LinuxSLLComponent) Name 

func (c *LinuxSLLComponent) Name() string

func (*LinuxSLLComponent) Order 

func (c *LinuxSLLComponent) Order() uint

func (*LinuxSLLComponent) Reconstruct 

func (c *LinuxSLLComponent) Reconstruct(ctx *DecodeContext) error

func (*LinuxSLLComponent) Schema 

func (c *LinuxSLLComponent) Schema(table string) string

func (*LinuxSLLComponent) SetIndex 

func (c *LinuxSLLComponent) SetIndex(i uint16)

type NTPComponent 

type NTPComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`

	LeapIndicator  uint8  `ch:"leap_indicator"`
	Version        uint8  `ch:"version"`
	Mode           uint8  `ch:"mode"`
	Stratum        uint8  `ch:"stratum"`
	Poll           int8   `ch:"poll"`
	Precision      int8   `ch:"precision"`
	RootDelay      uint32 `ch:"root_delay"`
	RootDispersion uint32 `ch:"root_dispersion"`
	ReferenceID    uint32 `ch:"reference_id"`

	ReferenceTS uint64 `ch:"reference_ts"`
	OriginTS    uint64 `ch:"origin_ts"`
	ReceiveTS   uint64 `ch:"receive_ts"`
	TransmitTS  uint64 `ch:"transmit_ts"`

	NTPRaw []byte `ch:"ntp_raw"`
}

NTPComponent stores parsed NTP header fields. ntp_raw holds the full wire bytes for lossless reconstruction.

func (*NTPComponent) ApplyNucleus 

func (c *NTPComponent) ApplyNucleus(nucleus PacketNucleus)

func (*NTPComponent) ClickhouseColumns 

func (c *NTPComponent) ClickhouseColumns() ([]string, error)

func (*NTPComponent) ClickhouseValues 

func (c *NTPComponent) ClickhouseValues() ([]any, error)

func (*NTPComponent) DataColumns 

func (c *NTPComponent) DataColumns(tableAlias string) ([]string, error)

func (*NTPComponent) Encode 

func (c *NTPComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*NTPComponent) FetchOrderBy 

func (c *NTPComponent) FetchOrderBy() string

func (*NTPComponent) Index 

func (c *NTPComponent) Index() uint16

func (*NTPComponent) Kind 

func (c *NTPComponent) Kind() uint

func (*NTPComponent) LayerSize 

func (c *NTPComponent) LayerSize() int

func (*NTPComponent) Name 

func (c *NTPComponent) Name() string

func (*NTPComponent) Order 

func (c *NTPComponent) Order() uint

func (*NTPComponent) Reconstruct 

func (c *NTPComponent) Reconstruct(ctx *DecodeContext) error

func (*NTPComponent) Schema 

func (c *NTPComponent) Schema(table string) string

func (*NTPComponent) SetIndex 

func (c *NTPComponent) SetIndex(i uint16)

type PacketNucleus 

type PacketNucleus struct {
	SessionID uint64
	PacketID  uint32
	Timestamp time.Time
	InclLen   uint32
	OrigLen   uint32

	Components *big.Int
	TailOffset uint16

	Payload []byte
}

PacketNucleus represents the primary row in pcap_packets, holding per-packet metadata and the component bitmask.

func (PacketNucleus) ClickhouseColumns 

func (p PacketNucleus) ClickhouseColumns() ([]string, error)

func (PacketNucleus) ClickhouseValues 

func (p PacketNucleus) ClickhouseValues() ([]any, error)

func (PacketNucleus) Table 

func (PacketNucleus) Table() string

type RepeatableExporter 

type RepeatableExporter interface {
	// ExportScanTargets returns slice pointers that receive groupArray results.
	ExportScanTargets() []any
	// ExportExpand expands the scanned arrays into one Component per entry.
	ExportExpand(sessionID uint64, packetID uint32) []Component
}

RepeatableExporter is implemented by components that can appear multiple times per packet (Dot1Q, IPv6Ext). Their wide-JOIN scan targets are []T slices produced by groupArray CTEs, so they cannot be derived from the scalar ch-tagged struct fields via NewScanBuf.

type ScanBuf 

type ScanBuf struct {
	Targets []any
	Apply   func()
}

ScanBuf holds Scan() pointers and a post-scan type-conversion function. Targets must be passed to rows.Scan; call Apply after a successful scan to convert string intermediates into the struct's typed fields.

func NewScanBuf 

func NewScanBuf(v any, withMeta bool) (*ScanBuf, error)

NewScanBuf returns a ScanBuf for the ch-tagged fields of v (must be a non-nil pointer to a struct). If withMeta is false, the fields "session_id", "packet_id", and "codec_version" are omitted.

Fields whose Go type cannot be scanned directly from ClickHouse are given string intermediates; Apply converts them after Scan:

  • []byte → scan as *string, apply: []byte(*s)
  • netip.Addr → scan as *string, apply: netip.ParseAddr(*s)
  • [N]byte → scan as *string, apply: copy into array

net.IP and all other types are scanned directly.

type TCPComponent 

type TCPComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`

	SrcPort    uint16 `ch:"src"`
	DstPort    uint16 `ch:"dst"`
	Seq        uint32 `ch:"seq"`
	Ack        uint32 `ch:"ack"`
	DataOffset uint8  `ch:"data_offset"`
	Flags      uint16 `ch:"flags"`
	Window     uint16 `ch:"window"`
	Checksum   uint16 `ch:"checksum"`
	Urgent     uint16 `ch:"urgent"`
	OptionsRaw []byte `ch:"options_raw"`
}

TCPComponent stores parsed TCP header fields. options_raw holds raw bytes 20..data_offset*4 from the original header, enabling bit-perfect frame reconstruction without checksum recomputation.

func (*TCPComponent) ApplyNucleus 

func (c *TCPComponent) ApplyNucleus(nucleus PacketNucleus)

func (*TCPComponent) ClickhouseColumns 

func (c *TCPComponent) ClickhouseColumns() ([]string, error)

func (*TCPComponent) ClickhouseValues 

func (c *TCPComponent) ClickhouseValues() ([]any, error)

func (*TCPComponent) DataColumns 

func (c *TCPComponent) DataColumns(tableAlias string) ([]string, error)

func (*TCPComponent) Encode 

func (c *TCPComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*TCPComponent) FetchOrderBy 

func (c *TCPComponent) FetchOrderBy() string

func (*TCPComponent) Index 

func (c *TCPComponent) Index() uint16

func (*TCPComponent) Kind 

func (c *TCPComponent) Kind() uint

func (*TCPComponent) LayerSize 

func (c *TCPComponent) LayerSize() int

func (*TCPComponent) Name 

func (c *TCPComponent) Name() string

func (*TCPComponent) Order 

func (c *TCPComponent) Order() uint

func (*TCPComponent) Reconstruct 

func (c *TCPComponent) Reconstruct(ctx *DecodeContext) error

func (*TCPComponent) Schema 

func (c *TCPComponent) Schema(table string) string

func (*TCPComponent) SetIndex 

func (c *TCPComponent) SetIndex(i uint16)

type UDPComponent 

type UDPComponent struct {
	SessionID    uint64 `ch:"session_id"`
	PacketID     uint32 `ch:"packet_id"`
	CodecVersion uint16 `ch:"codec_version"`
	LayerIndex   uint16 `ch:"layer_index"`

	SrcPort  uint16 `ch:"src"`
	DstPort  uint16 `ch:"dst"`
	Length   uint16 `ch:"length"`
	Checksum uint16 `ch:"checksum"`
}

UDPComponent stores parsed UDP header fields.

func (*UDPComponent) ApplyNucleus 

func (c *UDPComponent) ApplyNucleus(nucleus PacketNucleus)

func (*UDPComponent) ClickhouseColumns 

func (c *UDPComponent) ClickhouseColumns() ([]string, error)

func (*UDPComponent) ClickhouseValues 

func (c *UDPComponent) ClickhouseValues() ([]any, error)

func (*UDPComponent) DataColumns 

func (c *UDPComponent) DataColumns(tableAlias string) ([]string, error)

func (*UDPComponent) Encode 

func (c *UDPComponent) Encode(layer gopacket.Layer) ([]Component, error)

func (*UDPComponent) FetchOrderBy 

func (c *UDPComponent) FetchOrderBy() string

func (*UDPComponent) Index 

func (c *UDPComponent) Index() uint16

func (*UDPComponent) Kind 

func (c *UDPComponent) Kind() uint

func (*UDPComponent) LayerSize 

func (c *UDPComponent) LayerSize() int

func (*UDPComponent) Name 

func (c *UDPComponent) Name() string

func (*UDPComponent) Order 

func (c *UDPComponent) Order() uint

func (*UDPComponent) Reconstruct 

func (c *UDPComponent) Reconstruct(ctx *DecodeContext) error

func (*UDPComponent) Schema 

func (c *UDPComponent) Schema(table string) string

func (*UDPComponent) SetIndex 

func (c *UDPComponent) SetIndex(i uint16)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.