etherconn

package module
v0.6.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 5, 2023 License: BSD-2-Clause Imports: 30 Imported by: 6

README

etherconn

Build Status PkgGoDev

Package etherconn is a golang pkg that allow user to send/receive Ethernet payload (like IP pkt) or UDP packet ,with custom Ethernet encapsulation like MAC address, VLAN tags, without creating corresponding interface in OS;

For example, with etherconn, a program could send/recive a UDP or IP packet with a source MAC address and VLAN tags don't exists/provisioned in any of OS interfaces;

Another benefit is since etherconn bypasses "normal" OS kernel routing and IP stack, in scale setup like tens of thousands conns no longer subject to kernel limitation like # of socket/fd limitations, UDP buffer size...etc;

Lastly etherconn.RUDPConn implements the net.PacketConn interface, so it could be easily integrated into existing code;

etherconn supports following types of fowarding engines:

  • RawSocketRelay: uses AF_PACKET socket, linux only
  • XDPRelay: uses xdp socket, linux only
  • RawSocketRelayPcap: uses libpcap, windows and linux

XDPRelay could achieve higher performance than RawSocketRelay, specially in multi-queue, multi-core enviroment.

Performance

Tested in a KVM VM with 8 hyperthreading cores, and Intel 82599ES 10GE NIC, achieves 1Mpps with XDPRelay (1000B packet).

What's New

  1. add RawSocketRelayPcap, supports both windows and linux

Dependencies

etherconn require libpcap on linux, npcap on windows.

Usage

interface <---> PacketRelay <----> EtherConn <---> RUDPConn
                            <----> EtherConn <---> RUDPConn
                            <----> EtherConn <---> RUDPConn

  1. Create a PacketRelay instance and bound to an interface.PacketRelay is the "forward engine" that does actual packet sending/receiving for all EtherConn instances registered with it; PacketRelay send/receive Ethernet packet;

  2. Create one EtherConn for each source MAC+VLAN(s)+EtherType(s) combination needed, and register with the PacketRelay instance. EtherConn send/receive Ethernet payload like IP packet;

  3. Create one RUDPConn instance for each UDP endpoint (IP+Port) needed, with a EtherConn. RUDPConn send/receive UDP payload.

  4. RUDPConn and EtherConn is 1:1 mapping, while EtherConn and PacketRelay is N:1 mapping; since EtherConn and RUDPConn is 1:1 mapping, which means EtherConn will forward all received UDP pkts to RUDPConn even when its IP/UDP port is different from RUDPConn's endpoint, and RUDPConn could either only accept correct pkt or accept any UDP packet;

Egress direction:

UDP_payload -> RUDPConn(add UDP&IP header) -> EtherConn(add Ethernet header) -> PacketRelay

Ingress direction:

Ethernet_pkt -> (BPFilter) PacketRelay (parse pkt) --- EtherPayload(e.g IP_pkt) --> EtherConn
Ethernet_pkt -> (BPFilter) PacketRelay (parse pkt) --- UDP_payload --> RUDPConn (option to accept any UDP pkt)

Note: PacketRelay parse pkt for Ethernet payload based on following rules:

  • PacketRelay has default BPFilter set to only allow IPv4/ARP/IPv6 packet
  • If Ethernet pkt doesn't have VLAN tag, dstMAC + EtherType in Ethernet header is used to locate registered EtherConn
  • else, dstMAC + VLANs + EtherType in last VLAN tag is used
SharedEtherConn and SharingRUDPConn

EtherConn and RUDPConn are 1:1 mapping,which means two RUDPConn can't share same MAC+VLAN+EtherType combination;

SharedEtherConn and SharingRUDPConn solve this issue:

                                    L2Endpointkey-1
interface <---> PacketRelay <----> SharedEtherConn <---> SharingRUDPConn (L4Recvkey-1)
                                                   <---> SharingRUDPConn (L4Recvkey-2)
                                                   <---> SharingRUDPConn (L4Recvkey-3)
                                    L2Endpointkey-2
                            <----> SharedEtherConn <---> SharingRUDPConn (L4Recvkey-4)
                                                   <---> SharingRUDPConn (L4Recvkey-5)
                                                   <---> SharingRUDPConn (L4Recvkey-6)

Example:

	// This is an example of using RUDPConn, a DHCPv4 client
	// it also uses "github.com/insomniacslk/dhcp/dhcpv4/nclient4" for dhcpv4 client part

	// create PacketRelay for interface "enp0s10"
	relay, err := etherconn.NewRawSocketRelay(context.Background(), "enp0s10")
	if err != nil {
		log.Fatalf("failed to create PacketRelay,%v", err)
	}
	defer relay.Stop()
	mac, _ := net.ParseMAC("aa:bb:cc:11:22:33")
	vlanLlist := []*etherconn.VLAN{
		&etherconn.VLAN{
			ID:        100,
			EtherType: 0x8100,
		},
	}
	// create EtherConn, with src mac "aa:bb:cc:11:22:33" , VLAN 100 and DefaultEtherTypes,
	// with DOT1Q EtherType 0x8100, the mac/vlan doesn't need to be provisioned in OS
	econn := etherconn.NewEtherConn(mac, relay, etherconn.WithVLANs(vlanLlist))
	// create RUDPConn to use 0.0.0.0 and UDP port 68 as source, with option to accept any UDP packet
	// since DHCP server will send reply to assigned IP address
	rudpconn, err := etherconn.NewRUDPConn("0.0.0.0:68", econn, etherconn.WithAcceptAny(true))
	if err != nil {
		log.Fatalf("failed to create RUDPConn,%v", err)
	}
	// create DHCPv4 client with the RUDPConn
	clnt, err := nclient4.NewWithConn(rudpconn, mac, nclient4.WithDebugLogger())
	if err != nil {
		log.Fatalf("failed to create dhcpv4 client for %v", err)
	}
	// do DORA
	_, _, err = clnt.Request(context.Background())
	if err != nil {
		log.Fatalf("failed to finish DORA,%v", err)
	}

There is a more complicated example in example folder

Limitations:

* linux and windows only
* since etherconn bypassed OS IP stack, it is user's job to provide functions like:
    * routing next-hop lookup
    * IP -> MAC address resolution
* no IP packet fragementation/reassembly support
* using of etherconn requires root privileges on linux

Built-in XDP Kernel Program

etherconn includes a built-in XDP kernel program binary, its source is in etherconnkern

Documentation

Overview

Package etherconn is a golang pkg that allow user to send/receive Ethernet payload (like IP pkt) or UDP packet ,with custom Ethernet encapsulation like MAC address, VLAN tags, without creating corresponding interface in OS;

For example, with etherconn, a program could send/recive a UDP or IP packet with a source MAC address and VLAN tags don't exists/provisioned in any of OS interfaces;

Another benefit is since etherconn bypasses "normal" OS kernel routing and IP stack, in scale setup like tens of thousands conns no longer subject to linux kernel limitation like # of socket/fd limitations, UDP buffer size...etc;

Lastly etherconn.RUDPConn implements the net.PacketConn interface, so it could be easily integrated into existing code;

Usage: 

interface <---> PacketRelay <----> EtherConn <---> RUDPConn
                            <----> EtherConn <---> RUDPConn
                            <----> EtherConn <---> RUDPConn

1. Create a PacketRelay instance and bound to an interface.PacketRelay is the "forward engine" that does actual packet sending/receiving for all EtherConn instances registered with it; PacketRelay send/receive Ethernet packet; PacketRelay is a GO interface, currently there are two implementations:

  • RawSocketRelay: uses AF_PACKET socket, linux only
  • XDPRelay: uses xdp socket, linux only
  • RawSocketRelayPcap: uses libpcap, windows and linux

2. Create one EtherConn for each source MAC+VLAN(s)+EtherType(s) combination needed, and register with the PacketRelay instance. EtherConn send/receive Ethernet payload like IP packet;

3. Create one RUDPConn instance for each UDP endpoint (IP+Port) needed, with a EtherConn. RUDPConn send/receive UDP payload.

4. RUDPConn and EtherConn is 1:1 mapping, while EtherConn and PacketRelay is N:1 mapping; since EtherConn and RUDPConn is 1:1 mapping, which means EtherConn will forward all received UDP pkts to RUDPConn even when its IP/UDP port is different from RUDPConn's endpoint, and RUDPConn could either only accept correct pkt or accept any UDP packet;

Egress direction: 

UDP_payload -> RUDPConn(add UDP&IP header) -> EtherConn(add Ethernet header) -> PacketRelay

Ingress direction: 

Ethernet_pkt -> (BPFilter) PacketRelay (parse pkt) --- EtherPayload(e.g IP_pkt) --> EtherConn
Ethernet_pkt -> (BPFilter) PacketRelay (parse pkt) --- UDP_payload --> RUDPConn (option to accept any UDP pkt)

Note: PacketRelay parse pkt for Ethernet payload based on following rules: * PacketRelay has default BPFilter set to only allow IPv4/ARP/IPv6 packet * If Ethernet pkt doesn't have VLAN tag, dstMAC + EtherType in Ethernet header is used to locate registered EtherConn * else, dstMAC + VLANs + EtherType in last VLAN tag is used

Limitations:

  • linux only
  • since etherconn bypassed linux IP stack, it is user's job to provide functions like:
  • routing next-hop lookup
  • IP -> MAC address resolution
  • no IP packet fragementation/reassembly support
  • using of etherconn requires root privileges

Example: 

// This is an example of using RUDPConn, a DHCPv4 client
// it also uses "github.com/insomniacslk/dhcp/dhcpv4/nclient4" for dhcpv4 client part

// create PacketRelay for interface "enp0s10"
relay, err := etherconn.NewRawSocketRelay(context.Background(), "enp0s10")
if err != nil {
	log.Fatalf("failed to create PacketRelay,%v", err)
}
defer relay.Stop()
mac, _ := net.ParseMAC("aa:bb:cc:11:22:33")
vlanLlist := []*etherconn.VLAN{
	&etherconn.VLAN{
		ID:        100,
		EtherType: 0x8100,
	},
}
// create EtherConn, with src mac "aa:bb:cc:11:22:33" , VLAN 100 and DefaultEtherTypes,
// with DOT1Q EtherType 0x8100, the mac/vlan doesn't need to be provisioned in OS
econn := etherconn.NewEtherConn(mac, relay, etherconn.WithVLANs(vlanLlist))
// create RUDPConn to use 0.0.0.0 and UDP port 68 as source, with option to accept any UDP packet
// since DHCP server will send reply to assigned IP address
rudpconn, err := etherconn.NewRUDPConn("0.0.0.0:68", econn, etherconn.WithAcceptAny(true))
if err != nil {
	log.Fatalf("failed to create RUDPConn,%v", err)
}
// create DHCPv4 client with the RUDPConn
clnt, err := nclient4.NewWithConn(rudpconn, mac, nclient4.WithDebugLogger())
if err != nil {
	log.Fatalf("failed to create dhcpv4 client for %v", err)
}
// do DORA
_, _, err = clnt.Request(context.Background())
if err != nil {
	log.Fatalf("failed to finish DORA,%v", err)
}

XDPRelay uses Linux AF_XDP socket as the underlying forwarding mechinism, so it achives higher performance than RawSocketRelay in multi-core setup, XDPRelay usage notes:

  1. for virtio interface, the number of queues provisioned needs to be 2x of number CPU cores VM has, binding will fail otherwise.
  2. AF_XDP is still relative new, see XDP kernel&driver support status: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md#xdp
  3. For best performance: a) use NIC multiple queues and multiple routine(with runtime.LockOSThread()) to drive the traffic b) the number of routines >= number of NIC queues

Index

Constants

View Source 
const (
	// DefaultSendChanDepth is the default value for PacketRelay send channel depth, e.g. send buffer
	DefaultSendChanDepth = 1024
	// DefaultPerClntRecvChanDepth is the defaul value for per registered client(EtherConn)'s receive channel depth. e.g. recv buffer
	DefaultPerClntRecvChanDepth = 1024
	// DefaultMaxEtherFrameSize is the deafult max size of Ethernet frame that PacketRelay could receive from the interface
	DefaultMaxEtherFrameSize = 2048
	// DefaultRelayRecvTimeout is the default value for PacketReceive receiving timeout
	DefaultRelayRecvTimeout = time.Second
	DefaultTTL              = 255
)
View Source 
const (
	// MaxNumVLAN specifies max number vlan this pkg supports
	MaxNumVLAN = 2
	// L2EndpointKeySize is the size of L2EndpointKey in bytes
	L2EndpointKeySize = 6 + 2*MaxNumVLAN + 2 //must be 6+2*n
)
View Source 
const DefaultVLANEtype = 0x8100

DefaultVLANEtype is the default Ethernet type for vlan tags, used by function GetVLANs()

View Source 
const (
	//DefaultXDPChunkSize is the default size for XDP UMEM chunk
	DefaultXDPChunkSize = 4096
)
View Source 
const (
	// DefaultXDPUMEMNumOfTrunk is the default number of UMEM trunks
	DefaultXDPUMEMNumOfTrunk = 16384
)
View Source 
const (
	// NOVLANTAG is the value to represents NO vlan tag in L2EndpointKey
	NOVLANTAG = 0xffff
)

Variables

View Source 
var (
	// ErrTimeOut is the error returned when opeartion timeout
	ErrTimeOut = fmt.Errorf("timeout")
	// ErrRelayStopped is the error returned when relay already stopped
	ErrRelayStopped = fmt.Errorf("relay stopped")
	// BroadCastMAC is the broadcast MAC address
	BroadCastMAC = net.HardwareAddr{0xff, 0xff, 0xff, 0xff, 0xff, 0xff}
)
View Source 
var DefaultEtherTypes = []uint16{
	uint16(layers.EthernetTypeARP),
	uint16(layers.EthernetTypeIPv4),
	uint16(layers.EthernetTypeIPv6)}

DefaultEtherTypes is the default list of Ethernet types for RawPacketRelay and EtherConn

Functions

func GetIFQueueNum added in v0.4.0 

func GetIFQueueNum(ifname string) (int, error)

GetIFQueueNum use ethtool to get number of combined queue of the interface, return 1 if failed to get the info

func GetIfNameViaDesc added in v0.6.0 

func GetIfNameViaDesc(desc string) (string, error)

GetIfNameViaDesc returns interface name via its description, this could be used on windows to get the interface name

func ResolveNexhopMACWithBrodcast 

func ResolveNexhopMACWithBrodcast(ip net.IP) net.HardwareAddr

ResolveNexhopMACWithBrodcast is the default resolve function that always return broadcast mac

func SetIfVLANOffloading added in v0.4.1 

func SetIfVLANOffloading(ifname string, enable bool) error

SetIfVLANOffloading set the HW VLAN offloading feature on/off for the interface, turning the feautre off is needed when using XDPRelay and can't get expected vlan tags in received packet.

func SetPromisc added in v0.6.0 

func SetPromisc(ifname string) error

SetPromisc put the interface in Promisc mode

Types

type AncillaryVLAN added in v0.6.0 

type AncillaryVLAN struct {
	// The VLAN VID provided by the kernel.
	VLAN int
}

type ChanMap added in v0.4.3 

type ChanMap struct {
	// contains filtered or unexported fields
}

ChanMap is an GO routine safe map, key is interfce{}, val is a chan *RelayReceival;

func NewChanMap added in v0.4.3 

func NewChanMap() *ChanMap

NewChanMap creates a new instance of ChanMap

func (*ChanMap) CloseAll added in v0.4.3 

func (cm *ChanMap) CloseAll()

CloseAll close all channel in cm

func (*ChanMap) Del added in v0.4.3 

func (cm *ChanMap) Del(k interface{})

Del delete entry with key as k

func (*ChanMap) DelList added in v0.4.3 

func (cm *ChanMap) DelList(ks []interface{})

DelList deletes entries with key as k in ks

func (*ChanMap) Get added in v0.4.3 

func (cm *ChanMap) Get(k interface{}) chan *RelayReceival

Get return the channel map to k

func (*ChanMap) GetList added in v0.4.3 

func (cm *ChanMap) GetList() []chan *RelayReceival

GetList return all channels in cm

func (*ChanMap) Set added in v0.4.3 

func (cm *ChanMap) Set(k interface{}, v chan *RelayReceival)

Set (k,v) into cm

func (*ChanMap) SetList added in v0.4.3 

func (cm *ChanMap) SetList(ks []interface{}, v chan *RelayReceival)

SetList set a (k,v) into cm for each k in ks

type EtherConn 

type EtherConn struct {
	// contains filtered or unexported fields
}

EtherConn send/recv Ethernet payload like IP packet with customizable Ethernet encapsualtion like MAC and VLANs without provisioning them in OS. it needs to be registed with a PacketRelay instance to work.

func NewEtherConn 

func NewEtherConn(mac net.HardwareAddr, relay PacketRelay, options ...EtherConnOption) *EtherConn

NewEtherConn creates a new EtherConn instance, mac is used as part of EtherConn's L2Endpoint; relay is the PacketRelay that EtherConn instance register with; options specifies EtherConnOption(s) to use;

func (*EtherConn) Close 

func (ec *EtherConn) Close() error

Close implements net.PacketConn interface, deregister itself from PacketRelay

func (*EtherConn) GetEtherTypes added in v0.5.0 

func (ec *EtherConn) GetEtherTypes() []uint16

GetEtherTypes returns list of EtherType ec recevies

func (*EtherConn) LocalAddr 

func (ec *EtherConn) LocalAddr() *L2Endpoint

LocalAddr return EtherConn's L2Endpoint

func (*EtherConn) ReadPkt 

func (ec *EtherConn) ReadPkt() ([]byte, *L2Endpoint, error)

ReadPkt return received Ethernet payload bytes with an already allocated byte slice, along with remote L2Endpoint ReadPkt only return payload that matches one of underlying PacketRelay's configured EtherTypes

func (*EtherConn) ReadPktFrom 

func (ec *EtherConn) ReadPktFrom(p []byte) (int, *L2Endpoint, error)

ReadPktFrom copies the received Ethernet payload to p; it calls ReadPkt to get the payload, it return number bytes of IP packet, remote MAC address

func (*EtherConn) SetDeadline 

func (ec *EtherConn) SetDeadline(t time.Time) error

SetDeadline implements net.PacketConn interface

func (*EtherConn) SetReadDeadline 

func (ec *EtherConn) SetReadDeadline(t time.Time) error

SetReadDeadline implements net.PacketConn interface

func (*EtherConn) SetWriteDeadline 

func (ec *EtherConn) SetWriteDeadline(t time.Time) error

SetWriteDeadline implements net.PacketConn interface

func (*EtherConn) WriteIPPktTo 

func (ec *EtherConn) WriteIPPktTo(p []byte, dstmac net.HardwareAddr) (int, error)

WriteIPPktTo sends an IPv4/IPv6 packet, the pkt will be sent to dstmac, along with EtherConn.L2EP.VLANs.

func (*EtherConn) WriteIPPktToFrom added in v0.2.0 

func (ec *EtherConn) WriteIPPktToFrom(p []byte,
	srcmac, dstmac net.HardwareAddr, vlans VLANs) (int, error)

WriteIPPktToFrom is same as WriteIPPktTo beside send pkt with srcmac

func (*EtherConn) WritePktTo 

func (ec *EtherConn) WritePktTo(p []byte, etype uint16, dstmac net.HardwareAddr) (int, error)

WritePktTo sends an Ethernet payload, along with specified EtherType, the pkt will be sent to dstmac, along with EtherConn.L2EP.VLANs.

func (*EtherConn) WritePktToFrom added in v0.2.0 

func (ec *EtherConn) WritePktToFrom(p []byte, etype uint16, srcmac,
	dstmac net.HardwareAddr, vlans VLANs) (int, error)

WritePktToFrom is same as WritePktTo except with srcmac

type EtherConnOption 

type EtherConnOption func(ec *EtherConn)

EtherConnOption is a function use to provide customized option when creating EtherConn

func WithDefault added in v0.2.0 

func WithDefault() EtherConnOption

WithDefault will register the EtherConn to be the default EtherConn for received traffic, see PacketRelay.RegisterDefault for details. if relay is created with mirroring to default, then the etherconn will get a copy of all received pkt by the relay

func WithEtherTypes added in v0.2.0 

func WithEtherTypes(ets []uint16) EtherConnOption

WithEtherTypes specifies a list of Ethernet types that this EtherConn is interested in, the specified Ethernet types is the types of inner payload, the default list is DefaultEtherTypes

func WithRecvMulticast added in v0.1.2 

func WithRecvMulticast(recv bool) EtherConnOption

WithRecvMulticast allow/disallow EtherConn to receive multicast/broadcast Ethernet traffic

func WithVLANs 

func WithVLANs(vlans VLANs) EtherConnOption

WithVLANs specifies VLAN(s) as part of EtherConn's L2Endpoint. by default, there is no VLAN.

type L2Endpoint 

type L2Endpoint struct {
	HwAddr net.HardwareAddr
	VLANs  []uint16
	Etype  uint16 //inner most EtherType (e.g payload type)
}

L2Endpoint represents a layer2 endpoint that send/receives Ethernet frame

func NewL2EndpointFromMACVLAN 

func NewL2EndpointFromMACVLAN(mac net.HardwareAddr, vlans VLANs) *L2Endpoint

NewL2EndpointFromMACVLAN creates a new L2Endpoint from mac and vlans; its Etype is set to any

func NewL2EndpointFromMACVLANEtype added in v0.2.0 

func NewL2EndpointFromMACVLANEtype(mac net.HardwareAddr, vlans VLANs, etype uint16) *L2Endpoint

NewL2EndpointFromMACVLANEtype creates a new L2Endpoint from mac, vlans and etype

func (*L2Endpoint) GetKey 

func (l2e *L2Endpoint) GetKey() (key L2EndpointKey)

GetKey returns the key of the L2Endpoint

func (*L2Endpoint) GetVLANsWithDefaultEtype added in v0.2.0 

func (l2e *L2Endpoint) GetVLANsWithDefaultEtype() (r VLANs)

GetVLANsWithDefaultEtype return an instance of VLANs with VLAN ethernet type set as DefaultVLANEtype

func (*L2Endpoint) Network 

func (l2e *L2Endpoint) Network() string

Network implements net.Addr interface, return "l2ep"

func (*L2Endpoint) String 

func (l2e *L2Endpoint) String() (s string)

String implements net.Addr interface, return a string with format: l2ep&<l2EndpointKey_str>, see L2EndpointKey.String for format of <l2EndpointKey_str>

type L2EndpointKey 

type L2EndpointKey [L2EndpointKeySize]byte

L2EndpointKey is key identify a L2EndPoint,first 6 bytes are MAC address, VLAN Ids in order (from outside to inner), each VLAN id are two bytes in network endian, if VLAN id is NOVLANTAG,then it means no such tag; last two bytes are inner most EtherType.

func (L2EndpointKey) String 

func (l2epkey L2EndpointKey) String() string

type L4RecvKey added in v0.3.0 

type L4RecvKey [19]byte

L4RecvKey resprsents a Layer4 recv endpoint: [0:15] bytes is the IP address, [16] is the IP protocol, [17:18] is the port number, in big endian

func NewL4RecvKeyViaUDPAddr added in v0.3.0 

func NewL4RecvKeyViaUDPAddr(uaddr *net.UDPAddr) (r L4RecvKey)

NewL4RecvKeyViaUDPAddr returns a L4RecvKey from a net.UDPAddr

func (L4RecvKey) String added in v0.3.0 

func (l4k L4RecvKey) String() string

String returns string representation of the l4k; in format of "Addr:protocol:Port"

type LogFunc added in v0.4.0 

type LogFunc func(fmt string, a ...interface{})

type PacketRelay 

type PacketRelay interface {
	// Register register a list of L2EndpointKey of a EtherConn, PacketRely send/recv pkt on its behalf,
	// it returns following channels:
	// recv is the channel used to recive;
	// send is the channel used to send;
	// stop is a channel that will be closed when PacketRelay stops sending;
	// if recvMulticast is true, then multicast ethernet traffic will be recvied as well;
	// if one of key is already registered, then existing key will be overriden;
	Register(ks []L2EndpointKey, recvMulticast bool) (recv chan *RelayReceival, send chan []byte, stop chan struct{})
	// RegisterDefault return default receive channel,
	// meaning a received pkt doesn't match any other specific EtherConn registered with L2Endpointkey will be send to this channel;
	// multicast traffic will be also sent to this channel;
	RegisterDefault() (recv chan *RelayReceival, send chan []byte, stop chan struct{})
	// Deregister removes L2EndpointKey from registration
	Deregister(ks []L2EndpointKey)
	// Stop stops the forwarding of PacketRelay
	Stop()
	// IfName return binding interface name
	IfName() string
	// GetStats returns stats
	GetStats() *RelayPacketStats
	// Type returns relay type
	Type() RelayType
}

PacketRelay is a interface for the packet forwarding engine, RawSocketRelay implements this interface;

type PcapConn added in v0.6.0 

type PcapConn struct {
	*pcap.Handle
}

func NewPcapConn added in v0.6.0 

func NewPcapConn(ifname string) (*PcapConn, error)

NewPcapConn creates a new PcapRelay instances for specified ifname. Note: on windows, the ifname is the "\Device\NPF_xxxx"

func (*PcapConn) CloseMe added in v0.6.0 

func (pconn *PcapConn) CloseMe()

type RUDPConn 

type RUDPConn struct {
	// contains filtered or unexported fields
}

RUDPConn implement net.PacketConn interface; it used to send/recv UDP payload, using a underlying EtherConn for pkt forwarding.

func NewRUDPConn 

func NewRUDPConn(src string, c *EtherConn, options ...RUDPConnOption) (*RUDPConn, error)

NewRUDPConn creates a new RUDPConn, with specified EtherConn, and, optionally RUDPConnOption(s). src is the string represents its UDP Address as format supported by net.ResolveUDPAddr(). note the src UDP address could be any IP address, even address not provisioned in OS, like 0.0.0.0

func (*RUDPConn) Close 

func (ruc *RUDPConn) Close() error

Close implements net.PacketConn interface, it closes underlying EtherConn

func (*RUDPConn) LocalAddr 

func (ruc *RUDPConn) LocalAddr() net.Addr

LocalAddr implements net.PacketConn interface, it returns its UDPAddr

func (*RUDPConn) ReadFrom 

func (ruc *RUDPConn) ReadFrom(p []byte) (int, net.Addr, error)

ReadFrom implements net.PacketConn interface, it copy UDP payload to p; note: the underlying EtherConn will send all received pkts as *RelayReceival to RUDPConn, RUDPConn will ignore pkts that is not destined to its UDPAddr, unless WithAcceptAny(true) is specified when creating the RUDPConn, in that case, RUDPConn will accept any UDP packet;

func (*RUDPConn) SetAddr 

func (ruc *RUDPConn) SetAddr(src *net.UDPAddr)

SetAddr set RUDPConn's UDP address to src

func (*RUDPConn) SetDeadline 

func (ruc *RUDPConn) SetDeadline(t time.Time) error

SetDeadline implements net.PacketConn interface

func (*RUDPConn) SetReadDeadline 

func (ruc *RUDPConn) SetReadDeadline(t time.Time) error

SetReadDeadline implements net.PacketConn interface

func (*RUDPConn) SetWriteDeadline 

func (ruc *RUDPConn) SetWriteDeadline(t time.Time) error

SetWriteDeadline implements net.PacketConn interface

func (*RUDPConn) WriteTo 

func (ruc *RUDPConn) WriteTo(p []byte, addr net.Addr) (int, error)

WriteTo implements net.PacketConn interface, it sends UDP payload; This function adds UDP and IP header, and uses RUDPConn's resolve function to get nexthop's MAC address, and use underlying EtherConn to send IP packet, with EtherConn's Ethernet encapsulation, to nexthop MAC address; by default ResolveNexhopMACWithBrodcast is used for nexthop mac resolvement

func (*RUDPConn) WriteToFrom added in v0.2.0 

func (ruc *RUDPConn) WriteToFrom(p []byte, srcaddr, dstaddr net.Addr) (int, error)

WriteToFrom is same as WriteTo except sending payload p to dst with source address as src

type RUDPConnOption 

type RUDPConnOption func(rudpc *RUDPConn)

RUDPConnOption is a function use to provide customized option when creating RUDPConn

func WithAcceptAny 

func WithAcceptAny(accept bool) RUDPConnOption

WithAcceptAny allows RUDPConn to accept any UDP pkts, even it is not destinated to its address

func WithResolveNextHopMacFunc 

func WithResolveNextHopMacFunc(f func(net.IP) net.HardwareAddr) RUDPConnOption

WithResolveNextHopMacFunc specifies a function to resolve a destination IP address to next-hop MAC address; by default, ResolveNexhopMACWithBrodcast is used.

type RawSocketRelay 

type RawSocketRelay struct {
	// contains filtered or unexported fields
}

RawSocketRelay implements PacketRelay interface, using AF_PACKET socket

func NewRawSocketRelay 

func NewRawSocketRelay(parentctx context.Context, ifname string, options ...RelayOption) (*RawSocketRelay, error)

func NewRawSocketRelayPcap added in v0.6.0 

func NewRawSocketRelayPcap(parentctx context.Context, ifname string, options ...RelayOption) (*RawSocketRelay, error)

func NewRawSocketRelayWithRelayConn added in v0.6.0 

func NewRawSocketRelayWithRelayConn(parentctx context.Context, ifname string, conn rawRelayConn, options ...RelayOption) (*RawSocketRelay, error)

NewRawSocketRelay creates a new RawSocketRelay instance, bound to the interface ifname, optionally along with RelayOption functions. This function will put the interface in promisc mode, which means it requires root privilage

func (*RawSocketRelay) Deregister 

func (rsr *RawSocketRelay) Deregister(ks []L2EndpointKey)

Deregister implements PacketRelay interface;

func (*RawSocketRelay) GetStats 

func (rsr *RawSocketRelay) GetStats() *RelayPacketStats

GetStats return pkt forwarding stats as *RelayPacketStats

func (*RawSocketRelay) IfName added in v0.1.3 

func (rsr *RawSocketRelay) IfName() string

IfName returns the name of the binding interface

func (*RawSocketRelay) Register 

func (rsr *RawSocketRelay) Register(ks []L2EndpointKey, recvMulticast bool) (chan *RelayReceival, chan []byte, chan struct{})

Register implements PacketRelay interface;

func (*RawSocketRelay) RegisterDefault added in v0.2.0 

func (rsr *RawSocketRelay) RegisterDefault() (chan *RelayReceival, chan []byte, chan struct{})

RegisterDefault implements PacketRelay interface

func (*RawSocketRelay) Stop 

func (rsr *RawSocketRelay) Stop()

Stop implements PacketRelay interface

func (*RawSocketRelay) Type added in v0.6.0 

func (rsr *RawSocketRelay) Type() RelayType

type RelayOption 

type RelayOption func(*RawSocketRelay)

RelayOption is a function use to provide customized option when creating RawSocketRelay

func WithBPFFilter added in v0.1.2 

func WithBPFFilter(filter string) RelayOption

WithBPFFilter set BPF filter, which is a pcap filter string; if filter is an empty string, then it means no filter; by default, Relay will have a filter only allow traffic with specified EtherType.

func WithDebug 

func WithDebug(debug bool) RelayOption

WithDebug enable/disable debug log output

func WithDefaultReceival added in v0.2.0 

func WithDefaultReceival(mirroring bool) RelayOption

WithDefaultReceival creates a default receiving channel, all received pkt doesn't match any explicit EtherConn, will be sent to this channel; using RegisterDefault to get the default receiving channel. if mirroring is true, then every received pkt will be sent to this channel.

func WithMaxEtherFrameSize 

func WithMaxEtherFrameSize(size uint) RelayOption

WithMaxEtherFrameSize specifies the max Ethernet frame size the RawSocketRelay could receive

func WithMultiEngine added in v0.4.0 

func WithMultiEngine(count uint) RelayOption

WithMultiEngine specifies the number of internal send/recv routine, count must >=1, default value is 1

func WithPerClntChanRecvDepth 

func WithPerClntChanRecvDepth(depth uint) RelayOption

WithPerClntChanRecvDepth specifies the per Client(EtherConn) receive channel depth, By default, DefaultPerClntRecvChanDepth is used

func WithRecvTimeout 

func WithRecvTimeout(t time.Duration) RelayOption

WithRecvTimeout specifies the receive timeout for RawSocketRelay

func WithSendChanDepth 

func WithSendChanDepth(depth uint) RelayOption

WithSendChanDepth specifies the send channel depth, by default, DefaultSendChanDepth is used

type RelayPacketStats 

type RelayPacketStats struct {
	// Tx is number of pkts sent successfully
	Tx *uint64
	// RxOffered is number of pkts relay get from interface
	RxOffered *uint64
	// RxInvalid is nunber of pkts relay get but ignored due to failed valid check
	RxInvalid *uint64
	// RxBufferFull is the number of pkts can't send to receiver's channel right away due to it is full
	RxBufferFull *uint64
	// RxMiss is the number of pkts relay can't find receiver
	RxMiss *uint64
	// Rx is the number of pkts relay successfully deliver to receiver, not including pkt sent to default channel
	Rx *uint64
	// RxDefault is the number of pkts relay deliver to the default rcv channel
	RxDefault *uint64
	// RxNonHitMulticast is the number of multicast pkts that doesn't have direct receiver, but deliver to a multicast recevier
	RxNonHitMulticast *uint64
	// RxMulticastIgnored is the number of multicast pkts ignored
	RxMulticastIgnored *uint64
}

RelayPacketStats is the PacketRelay's forwding stats; use atomic.LoadUint64 to read the values

func (RelayPacketStats) String 

func (rps RelayPacketStats) String() string

type RelayReceival 

type RelayReceival struct {
	//LocalEndpoint/RemoteEndpoint is the local/remote L2Endpoint
	LocalEndpoint, RemoteEndpoint *L2Endpoint
	// EtherBytes is the Ethernet frame bytes
	EtherBytes []byte
	// EtherPayloadBytes is the Ethernet payload bytes within the EtherBytes,
	// where payload belongs to the specified EtherTypes,
	// default are 0x0800 (IPv4) and 0x86dd (IPv6),
	// nil if there is no payload with specified EtherTypes;
	EtherPayloadBytes []byte
	// TransportPayloadBytes is the transport layer(UDP/TCP/SCTP) payload bytes within the IPBytes,nil for unsupport transport layer
	TransportPayloadBytes []byte
	// RemoteIP is the remote IP address
	RemoteIP net.IP
	// RemotePort is the remote transport layer port, 0 for unsupport transport layer
	RemotePort uint16
	// Protocol is the IP protocol number
	Protocol uint8
	// LocalIP is the local IP address
	LocalIP net.IP
	// LocalPort is the local transport layer port, 0 for unsupport transport layer
	LocalPort uint16
}

RelayReceival is the what PacketRelay received and parsed

func (*RelayReceival) GetL4Key added in v0.3.0 

func (rr *RelayReceival) GetL4Key() (r L4RecvKey)

type RelayType added in v0.6.0 

type RelayType string
const (
	RelayTypeAFP  RelayType = "afpkt"
	RelayTypePCAP RelayType = "pcap"
	RelayTypeXDP  RelayType = "xdp"
	UnknownRelay  RelayType = "unknown"
)

func (RelayType) MarshalText added in v0.6.1 

func (rt RelayType) MarshalText() (text []byte, err error)

MarshalText implements encoding.TextMarshaler interface

func (*RelayType) UnmarshalText added in v0.6.1 

func (rt *RelayType) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler interface

type SharedEconn added in v0.4.3 

type SharedEconn interface {
	Register(k L4RecvKey) (torecvch chan *RelayReceival)
	WriteIPPktTo(p []byte, dstmac net.HardwareAddr) (int, error)
	SetWriteDeadline(t time.Time) error
}

type SharedEtherConn added in v0.3.0 

type SharedEtherConn struct {
	// contains filtered or unexported fields
}

SharedEtherConn could be mapped to multiple RUDPConn

func NewSharedEtherConn added in v0.3.0 

func NewSharedEtherConn(parentctx context.Context,
	mac net.HardwareAddr, relay PacketRelay,
	ecopts []EtherConnOption, options ...SharedEtherConnOption) *SharedEtherConn

NewSharedEtherConn creates a new SharedEtherConn; mac is the SharedEtherConn's own MAC address; relay is the underlying PacketRelay; ecopts is a list of EtherConnOption that could be used to customized new SharedEtherConnOption, all currently defined EtherConnOption could also be used for SharedEtherConn. options is a list of SharedEtherConnOption, not used currently;

func (*SharedEtherConn) Close added in v0.3.0 

func (sec *SharedEtherConn) Close()

Close stop the SharedEtherConn

func (*SharedEtherConn) Register added in v0.3.0 

func (sec *SharedEtherConn) Register(k L4RecvKey) (torecvch chan *RelayReceival)

Register register a key, return following channels: torecvch is the channel which is used to store received packets has one of registered key in keys;

func (*SharedEtherConn) RegisterList added in v0.3.0 

func (sec *SharedEtherConn) RegisterList(keys []L4RecvKey) (torecvch chan *RelayReceival)

RegisterList register a set of keys, return following channels: torecvch is the channel which is used to store received packets has one of registered key in keys;

func (*SharedEtherConn) SetWriteDeadline added in v0.4.3 

func (sec *SharedEtherConn) SetWriteDeadline(t time.Time) error

func (*SharedEtherConn) WriteIPPktTo added in v0.3.0 

func (sec *SharedEtherConn) WriteIPPktTo(p []byte, dstmac net.HardwareAddr) (int, error)

WriteIPPktTo sends an IP packet to dstmac, with EtherConn's vlan encapsualtaion, if any;

func (*SharedEtherConn) WriteIPPktToFrom added in v0.3.0 

func (sec *SharedEtherConn) WriteIPPktToFrom(p []byte, srcmac, dstmac net.HardwareAddr, vlans VLANs) (int, error)

WriteIPPktToFrom is same as WriteIPPktTo beside send pkt with srcmac

func (*SharedEtherConn) WritePktTo added in v0.3.0 

func (sec *SharedEtherConn) WritePktTo(p []byte, etype uint16, dstmac net.HardwareAddr) (int, error)

WritePktTo sends an Ethernet payload, along with specified EtherType, the pkt will be sent to dstmac, along with EtherConn.L2EP.VLANs.

func (*SharedEtherConn) WritePktToFrom added in v0.3.0 

func (sec *SharedEtherConn) WritePktToFrom(p []byte, etype uint16, srcmac, dstmac net.HardwareAddr, vlans VLANs) (int, error)

WritePktToFrom is same as WritePktTo except with srcmac

type SharedEtherConnOption added in v0.3.0 

type SharedEtherConnOption func(sec *SharedEtherConn)

SharedEtherConnOption is the option to customize new SharedEtherConnOption

func WithSharedEConnPerClntRecvChanDepth added in v0.4.0 

func WithSharedEConnPerClntRecvChanDepth(depth uint) SharedEtherConnOption

type SharingRUDPConn added in v0.3.0 

type SharingRUDPConn struct {
	// contains filtered or unexported fields
}

SharingRUDPConn is the UDP connection could share same SharedEtherConn;

func NewSharingRUDPConn added in v0.3.0 

func NewSharingRUDPConn(src string, c SharedEconn, roptions []RUDPConnOption, options ...SharingRUDPConnOptions) (*SharingRUDPConn, error)

NewSharingRUDPConn creates a new SharingRUDPConn, src is the string represents its UDP Address as format supported by net.ResolveUDPAddr(). c is the underlying SharedEtherConn, roptions is a list of RUDPConnOptions that use for customization, supported are: WithResolveNextHopMacFunc; note unlike RUDPConn, SharingRUDPConn doesn't support acceptting pkt is not destinated to own address

func (*SharingRUDPConn) Close added in v0.3.0 

func (sruc *SharingRUDPConn) Close() error

Close implements net.PacketConn interface, it closes underlying EtherConn

func (*SharingRUDPConn) LocalAddr added in v0.3.0 

func (sruc *SharingRUDPConn) LocalAddr() net.Addr

LocalAddr implements net.PacketConn interface, it returns its UDPAddr

func (*SharingRUDPConn) ReadFrom added in v0.3.0 

func (sruc *SharingRUDPConn) ReadFrom(p []byte) (int, net.Addr, error)

ReadFrom implment net.PacketConn interface, it returns UDP payload;

func (*SharingRUDPConn) SetDeadline added in v0.3.0 

func (sruc *SharingRUDPConn) SetDeadline(t time.Time) error

SetDeadline implements net.PacketConn interface

func (*SharingRUDPConn) SetReadDeadline added in v0.3.0 

func (sruc *SharingRUDPConn) SetReadDeadline(t time.Time) error

SetReadDeadline implements net.PacketConn interface

func (*SharingRUDPConn) SetWriteDeadline added in v0.3.0 

func (sruc *SharingRUDPConn) SetWriteDeadline(t time.Time) error

SetWriteDeadline implements net.PacketConn interface

func (*SharingRUDPConn) WriteTo added in v0.3.0 

func (sruc *SharingRUDPConn) WriteTo(p []byte, addr net.Addr) (int, error)

WriteTo implements net.PacketConn interface, it sends UDP payload; This function adds UDP and IP header, and uses sruc's resolve function to get nexthop's MAC address, and use underlying SharedEtherConn to send IP packet, with SharedEtherConn's Ethernet encapsulation, to nexthop MAC address; by default ResolveNexhopMACWithBrodcast is used for nexthop mac resolvement

type SharingRUDPConnOptions added in v0.3.0 

type SharingRUDPConnOptions func(srudpc *SharingRUDPConn)

SharingRUDPConnOptions is is the option to customize new SharingRUDPConn

type VLAN 

type VLAN struct {
	ID        uint16
	EtherType uint16
}

VLAN reprents a VLAN tag

type VLANs added in v0.1.1 

type VLANs []*VLAN

VLANs is a slice of VLAN

func (VLANs) Clone added in v0.1.1 

func (vlans VLANs) Clone() VLANs

Clone return a deep copy

func (*VLANs) Copy added in v0.1.1 

func (vlans *VLANs) Copy(vlans2 VLANs)

Copy copy value of lvans2 to vlans

func (VLANs) Equal added in v0.1.1 

func (vlans VLANs) Equal(vlans2 VLANs) bool

Equal returns true if vlans == vlans2

func (VLANs) IDs added in v0.1.1 

func (vlans VLANs) IDs() []uint16

IDs return a VLAN IDs as a slice of uint16

func (VLANs) MarshalText added in v0.6.1 

func (vlans VLANs) MarshalText() (text []byte, err error)

MarshalText implements encoding.TextMarshaler interface

func (VLANs) SetIDs added in v0.1.1 

func (vlans VLANs) SetIDs(ids []uint16) error

SetIDs set VLAN ID with the specified uint16 slice

func (VLANs) String added in v0.1.1 

func (vlans VLANs) String() string

String return a string representation

func (*VLANs) UnmarshalText added in v0.6.1 

func (vlans *VLANs) UnmarshalText(text []byte) error

UnmarshalText implements encoding.TextUnmarshaler interface, it accepts two formats: either "x.y" or "x|y", and it uses the DefaultVLANEtype

type XDPRelay added in v0.3.0 

type XDPRelay struct {
	// contains filtered or unexported fields
}

XDPRelay is a PacketRelay implementation that uses AF_XDP Socket

func NewXDPRelay added in v0.3.0 

func NewXDPRelay(parentctx context.Context, ifname string, options ...XDPRelayOption) (*XDPRelay, error)

NewXDPRelay creates a new instance of XDPRelay, by default, the XDPRelay binds to all queues of the specified interface

func (*XDPRelay) Deregister added in v0.3.0 

func (xr *XDPRelay) Deregister(ks []L2EndpointKey)

Deregister implements PacketRelay interface;

func (*XDPRelay) GetStats added in v0.3.0 

func (xr *XDPRelay) GetStats() *RelayPacketStats

GetStats returns the stats

func (*XDPRelay) IfName added in v0.3.0 

func (xr *XDPRelay) IfName() string

IfName implements PacketRelay interface;

func (*XDPRelay) NumSocket added in v0.4.0 

func (xr *XDPRelay) NumSocket() int

NumSocket returns number of XDP socket

func (*XDPRelay) Register added in v0.3.0 

func (xr *XDPRelay) Register(ks []L2EndpointKey, recvMulticast bool) (chan *RelayReceival, chan []byte, chan struct{})

Register implements PacketRelay interface;

func (*XDPRelay) RegisterDefault added in v0.3.0 

func (xr *XDPRelay) RegisterDefault() (chan *RelayReceival, chan []byte, chan struct{})

RegisterDefault implements PacketRelay interface

func (*XDPRelay) Stop added in v0.3.0 

func (xr *XDPRelay) Stop()

Stop implements PacketRelay interface

func (*XDPRelay) Type added in v0.6.0 

func (xr *XDPRelay) Type() RelayType

type XDPRelayOption added in v0.3.0 

type XDPRelayOption func(xr *XDPRelay)

XDPRelayOption could be used in NewXDPRelay to customize XDPRelay upon creation

func WithQueueID added in v0.3.0 

func WithQueueID(qidlist []int) XDPRelayOption

WithQueueID specifies a list of interface queue id (start from 0) that the XDPRelay binds to; by default, XDPRelay will use all queues. note: only use this option if you know what you are doing, since this could cause lower performance or XDPRelay unable to receive some of packets.

func WithSendingMode added in v0.4.0 

func WithSendingMode(m XDPSendingMode) XDPRelayOption

WithSendingMode set the XDPRelay's sending mode to m

func WithXDPDebug added in v0.3.0 

func WithXDPDebug(debug bool) XDPRelayOption

WithXDPDebug enable/disable debug log output

func WithXDPDefaultReceival added in v0.3.0 

func WithXDPDefaultReceival(mirroring bool) XDPRelayOption

WithXDPDefaultReceival creates a default receiving channel, all received pkt doesn't match any explicit EtherConn, will be sent to this channel; using RegisterDefault to get the default receiving channel. if mirroring is true, then every received pkt will be sent to this channel.

func WithXDPEtherTypes added in v0.4.2 

func WithXDPEtherTypes(ets []uint16) XDPRelayOption

WithXDPEtherTypes specifies a list of EtherType that the relay accepts, if a rcvd packet doesn't have a expected EtherType, then it will be passed to kernel. the EtherType is the inner most EtherType in case there is vlan tag. the default accept EtherTypes is DefaultEtherTypes. Note: this requires the builtin XDP kernel program.

func WithXDPExtProg added in v0.4.0 

func WithXDPExtProg(fname, prog, qmap, xskmap, etypemap string) XDPRelayOption

WithXDPExtProg loads an external XDP kernel program iso using the built-in one

func WithXDPPerClntRecvChanDepth added in v0.4.0 

func WithXDPPerClntRecvChanDepth(depth uint) XDPRelayOption

WithXDPPerClntRecvChanDepth set the depth in recving channel for each registered

func WithXDPRXPktHandler added in v0.4.0 

func WithXDPRXPktHandler(h XDPSocketPktHandler) XDPRelayOption

WithXDPRXPktHandler sets h as the rx packet handler

func WithXDPSendChanDepth added in v0.4.0 

func WithXDPSendChanDepth(depth uint) XDPRelayOption

WithXDPSendChanDepth set the dep th in sending channel

func WithXDPTXPktHandler added in v0.4.0 

func WithXDPTXPktHandler(h XDPSocketPktHandler) XDPRelayOption

WithXDPTXPktHandler sets h as the tx packet handler

func WithXDPUMEMChunkSize added in v0.4.0 

func WithXDPUMEMChunkSize(fsize uint) XDPRelayOption

WithXDPUMEMChunkSize specifies the XDP UMEM size, which implicitly set the max packet size could be handled by XDPRelay, must be either 4096 or 2048 (kernel XDP limitation)

func WithXDPUMEMNumOfTrunk added in v0.3.0 

func WithXDPUMEMNumOfTrunk(num uint) XDPRelayOption

WithXDPUMEMNumOfTrunk specifies the number of UMEM trunks, must be power of 2. the Fill/Completion/TX/RX ring size is half of specified value;

type XDPSendingMode added in v0.4.0 

type XDPSendingMode string

XDPSendingMode is the TX mode of XDPRelay 

const (
	//XDPSendingModeSingle is the TX mode send a packet a time, this is the default mode;
	XDPSendingModeSingle XDPSendingMode = "single"
	//XDPSendingModeBatch is the TX mode sends a batch of packet a time, only use this mode when needed TX pps is high;
	XDPSendingModeBatch XDPSendingMode = "batch"
)

type XDPSocketPktHandler added in v0.4.0 

type XDPSocketPktHandler func(pbytes []byte, sockid int) error

XDPSocketPktHandler is a handler function could be used for rx/tx packet of a XDP socket

type XdpSockStats added in v0.4.0 

type XdpSockStats struct {
	Sent, Recv uint64
}

XdpSockStats hold per XDP socket/queue stats

type XdpSockStatsList added in v0.4.0 

type XdpSockStatsList map[int]*XdpSockStats

func (XdpSockStatsList) String added in v0.4.0 

func (list XdpSockStatsList) String() string

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.