balancer

README

Layer-3 Direct Server Return load balancer prototype

note this is proof-of-concept project, do not use this in production

This is a prototype for a L3-DSR hash based load balancer. For a client I was researching if we could improve its CDN (for streaming content) by using a hash-based loadbalancer with the capability of using Layer-3 based Direct Server Return (so that returning traffic is bypassing the loadbalancer).

This codebase only implements a naive balancer application which:

1. creates a handshake with the client
2. inspects the first packet to determine which server to route the traffic to
3. syncs the TCP handshake with the packetbridge application
4. forwards all further TCP traffic to the packetbridge

By using the DSCP field in the IPv4 header, the loadbalancer identifies itself to the packetbride.

The packetbridge application:

1. receives TCP packets from the balancer application
2. creates (for new connections) a handshake with the backend (e.g. NGINX)
3. modifies the incoming traffic so that it matches with the TCP handshake of the backend
4. modifies outgoing traffic so that it matches with the TCP handshake of the client.

The packetbride will use the loadbalancer IP address (it knows because of the DSCP field) for outgoing traffic, so that outgoing traffic bypasses the loadbalancer.

How to use

The easiest way to play with this project is to setup a Vagrant environment. Running vagrant up will setup two boxes, one for the balancer, the other for the backend.

Note that all cli argument have defaults that matches the Vagrant environment.

starting the balancer

Run the following commands to start the balancer (within the Vagrant environment).

vagrant ssh balancer
cd src/github.com/brocaar/l3dsr-hash-balancer
go get ./...
make
sudo ./bin/balancer

The balancer box has one interface 192.168.33.10 on which it listens for incoming requests.

starting the packetbridge / backend

Run the following commands to start the packetbridge (within the Vagrant environment).

vagrant ssh backend
cd src/github.com/brocaar/l3dsr-hash-balancer
sudo ./bin/packetbridge

The backend box has two interfaces. On 192.168.33.20 it listens for incoming packets from the balancer. On 192.168.33.30 NGINX is running.

making requests

Now that both applications are running, you can make a request to http://192.168.33.10/. This will:

• Create a TCP handshake between you and the balancer (.10)
• The balancer (.10) will sync your TCP handshake with the packetbridge (.20)
• The packetbridge (.20) will create a TCP handshake with the backend (NGINX on .30).
• The packetbridge will start forwarding your packets to NGIXN (.30) and the packets from NGINX to you (by using the .10 source ip). Your HTTP client will think that all packets came from the balancer :-)

Documentation

Index

• func BalancePackets(packetsIn chan gopacket.Packet, packetsOut chan *EthPacket, ...)
• func GetAddrByName(name string) (net.IP, error)
• func HandleBackendPackets(conn net.PacketConn, dstIP, srcIP net.IP, srcPort layers.TCPPort, ...)
• func HandleBalancerPackets(packetsIn chan gopacket.Packet, backendPackets chan *TCPPacket, ...)
• func SendToBackend(conn net.PacketConn, backendPackets chan *TCPPacket, srcIP, dstIP net.IP)
• func SendToClient(handle *pcap.Handle, ethPackets chan *EthPacket)
• type DummyPool
  • func (b *DummyPool) AddServer(s *Server)
  • func (b *DummyPool) RouteToServer(i int64) (*Server, error)
• type EthPacket
  • func NewEthPacket(eth *layers.Ethernet, ip *layers.IPv4, tcp *layers.TCP) *EthPacket
  • func (p *EthPacket) MarshalBinary() ([]byte, error)
  • func (p *EthPacket) SetTOS(tos uint8)
  • func (p *EthPacket) String() string
• type PacketBridgeState
• type PacketBridgeStateTable
  • func NewPacketBridgeStateTable() *PacketBridgeStateTable
  • func (s *PacketBridgeStateTable) GetByIP(ip net.IP, port layers.TCPPort) (*PacketBridgeState, bool)
  • func (s *PacketBridgeStateTable) GetByPort(port layers.TCPPort) (*PacketBridgeState, bool)
  • func (s *PacketBridgeStateTable) NewState(ip net.IP, mac net.HardwareAddr, port layers.TCPPort, lbIndex uint8, ...) *PacketBridgeState
• type PoolBalancer
  • func NewDummyBalancer() PoolBalancer
• type Server
• type State
• type StateTable
  • func NewStateTable() *StateTable
  • func (s *StateTable) GetState(ip net.IP, port layers.TCPPort) (*State, bool)
  • func (s *StateTable) NewState(ip net.IP, port layers.TCPPort) *State
• type TCPPacket
  • func NewTCPPacket(ip *layers.IPv4, tcp *layers.TCP) *TCPPacket
  • func (p *TCPPacket) MarshalBinary() ([]byte, error)
  • func (p *TCPPacket) SetDstIP(dst net.IP)
  • func (p *TCPPacket) SetSrcIP(src net.IP)
  • func (p *TCPPacket) String() string
• type TCPState

Functions

func BalancePackets

func BalancePackets(packetsIn chan gopacket.Packet, packetsOut chan *EthPacket, stateTable *StateTable, pool PoolBalancer)

BalancePacket implements the actual load-balance logic on packet level.

func GetAddrByName

func GetAddrByName(name string) (net.IP, error)

GetInterfaceAndAddrByName returns the net.IP for a given interface name.

func HandleBackendPackets

func HandleBackendPackets(conn net.PacketConn, dstIP, srcIP net.IP, srcPort layers.TCPPort, pbIface *net.Interface, backendTCPPackets chan *TCPPacket, ethPackets chan *EthPacket, stateTable *PacketBridgeStateTable, balancers map[uint8]net.IP)

HandleBackendPackets handles incoming packets from the backend. If the connection is known, it will forward these packets to the client.

func HandleBalancerPackets

func HandleBalancerPackets(packetsIn chan gopacket.Packet, backendPackets chan *TCPPacket, stateTable *PacketBridgeStateTable)

HandleBalancerPackets handles the incoming packets from the balancer app. When the connection is known, it will forward it to the backend. If not, it will first start a TCP handshake with the backend.

func SendToBackend

func SendToBackend(conn net.PacketConn, backendPackets chan *TCPPacket, srcIP, dstIP net.IP)

SendToBackend sends packets from the packetbridge to the backend.

func SendToClient

func SendToClient(handle *pcap.Handle, ethPackets chan *EthPacket)

SendToClient sends packets to the client who started the request at the balancer.

Types

type DummyPool

type DummyPool struct {
	// contains filtered or unexported fields
}

DummyPool provides a PoolBalancer for a single server.

func (*DummyPool) AddServer

func (b *DummyPool) AddServer(s *Server)

AddServer sets the (single) server.

func (*DummyPool) RouteToServer

func (b *DummyPool) RouteToServer(i int64) (*Server, error)

RouteToServer returns the single server (or an error when no server is set).

type EthPacket

type EthPacket struct {
	// contains filtered or unexported fields
}

EthPacket represents an ethernet packet.

func NewEthPacket

func NewEthPacket(eth *layers.Ethernet, ip *layers.IPv4, tcp *layers.TCP) *EthPacket

NewEthPacket creates and initializes a new EthPacket.

func (*EthPacket) MarshalBinary

func (p *EthPacket) MarshalBinary() ([]byte, error)

MarshalBinary returns the binary representation of the packet.

func (*EthPacket) SetTOS

func (p *EthPacket) SetTOS(tos uint8)

SetTOS sets the IP TOS field.

func (*EthPacket) String

func (p *EthPacket) String() string

type PacketBridgeState

type PacketBridgeState struct {
	State        TCPState
	IP           net.IP
	HardwareAddr net.HardwareAddr
	RandPort     layers.TCPPort
	Port         layers.TCPPort
	LBIndex      uint8
	SeqOffset    uint32
	PayloadBuf   []byte
}

PacketBridgeState represents a single connection state at the packet bridge.

type PacketBridgeStateTable

type PacketBridgeStateTable struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

PacketBridgeStateTable represents a table of tcp connection states.

func NewPacketBridgeStateTable

func NewPacketBridgeStateTable() *PacketBridgeStateTable

NewPacketBridgeStateTable creates and initializes a new PacketBridgeStateTable.

func (*PacketBridgeStateTable) GetByIP

func (s *PacketBridgeStateTable) GetByIP(ip net.IP, port layers.TCPPort) (*PacketBridgeState, bool)

func (*PacketBridgeStateTable) GetByPort

func (s *PacketBridgeStateTable) GetByPort(port layers.TCPPort) (*PacketBridgeState, bool)

func (*PacketBridgeStateTable) NewState

func (s *PacketBridgeStateTable) NewState(ip net.IP, mac net.HardwareAddr, port layers.TCPPort, lbIndex uint8, seqOffset uint32, payload []byte) *PacketBridgeState

type PoolBalancer

type PoolBalancer interface {
	AddServer(*Server)
	RouteToServer(int64) (*Server, error)
}

PoolBalancer specifies the interface for a balancer backend.

func NewDummyBalancer

func NewDummyBalancer() PoolBalancer

NewDummyBalancers returns a new DummyPool.

type Server

type Server struct {
	IP           net.IP
	HardwareAddr net.HardwareAddr
}

Server contains all the information for a backend server.

type State

type State struct {
	State  TCPState
	Server *Server
	Seq    uint32
}

State represents a single connection state

type StateTable

type StateTable struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

StateTable keeps track of the connection states.

func NewStateTable

func NewStateTable() *StateTable

NewStateTable creates and initializes a new StateTable.

func (*StateTable) GetState

func (s *StateTable) GetState(ip net.IP, port layers.TCPPort) (*State, bool)

func (*StateTable) NewState

func (s *StateTable) NewState(ip net.IP, port layers.TCPPort) *State

type TCPPacket

type TCPPacket struct {
	// contains filtered or unexported fields
}

TCPPacket represents a TCP packet.

func NewTCPPacket

func NewTCPPacket(ip *layers.IPv4, tcp *layers.TCP) *TCPPacket

NewTCPPacket creates and initializes a new TCP packet. The IP layer is needed to calculate the correct TCP checksum.

func (*TCPPacket) MarshalBinary

func (p *TCPPacket) MarshalBinary() ([]byte, error)

MarshalBinary returns the binary representation of the packet.

func (*TCPPacket) SetDstIP

func (p *TCPPacket) SetDstIP(dst net.IP)

SetDstIP sets the destination IP.

func (*TCPPacket) SetSrcIP

func (p *TCPPacket) SetSrcIP(src net.IP)

SetSrcIP sets the source IP.

func (*TCPPacket) String

func (p *TCPPacket) String() string

type TCPState

type TCPState uint8

TCPState defines the state a TCP connection has.

const (
	TCP_STATE_SYN_SENT TCPState = iota
	TCP_STATE_SYN_RECEIVED
	TCP_STATE_ESTABLISHED
	TCP_STATE_FIN_WAIT_1
	TCP_STATE_FIN_WAIT_2
	TCP_STATE_CLOSE_WAIT
	TCP_STATE_CLOSING
	TCP_STATE_LAST_ACK
	TCP_STATE_TIME_WAIT
	TCP_STATE_CLOSED
)