multipath

README

Multipath aggregates ordered and reliable connections over multiple paths together for throughput and resilience. It relies on existing dialers and listeners to create net.Conns and wrap them as subflows on which it basically does two things:

1. On the sender side, transmits data over the subflow with the lowest roundtrip time, and if it takes long to get an acknowledgement, retransmits data over other subflows one by one.
2. On the receiver side, reorders the data received from all subflows and delivers ordered byte stream (net.Conn) to the upper layer.

See docs in multipath.go for details.

This code is used in https://github.com/benjojo/bondcat, and hat's off to @benjojo for implementing many fixes.

At a high level, this concept is built on a similar notion as MultiPath TCP, but our "multipath" works at a higher level in the stack where it implements different protocols that each have their own subflow but are all running on their own TCP or reliable UDP transports underneath.

Documentation

Overview

Package multipath provides a simple way to aggregate multiple network paths between a pair of hosts to form a single connection from the upper layer perspective, for throughput and resilience.

The term connection, path and subflow used here are the same as mentioned in MP-TCP https://www.rfc-editor.org/rfc/rfc8684.html#name-terminology

Each subflow is a bidirectional byte stream each side in the following form until being disrupted or the connection ends. When establishing the very first subflow, the client sends an all-zero connnection ID (CID) and the server sends the assigned CID back. Subsequent subflows use the same CID.

 ----------------------------------------------------
|  version(1)  |  cid(16)  |  frames (...)  |
 ----------------------------------------------------

There are two types of frames. Data frame carries application data while ack frame carries acknowledgement to the frame just received. When one data frame is not acked in time, it is sent over another subflow, until all available subflows have been tried. Payload size and frame number uses variable-length integer encoding as described here: https://tools.ietf.org/html/draft-ietf-quic-transport-29#section-16

 --------------------------------------------------------
|  payload size(1-8)  |  frame number (1-8)  |  payload  |
 --------------------------------------------------------

 ---------------------------------------
|  00000000  |  ack frame number (1-8)  |
 ---------------------------------------

Ack frames with frame number < 10 are reserved for control. For now only 0 and 1 are used, for ping and pong frame respectively. They are for updating RTT on inactive subflows and detecting recovered subflows.

Ping frame:

 -------------------------
|  00000000  |  00000000  |
 -------------------------

Pong frame:

 -------------------------
|  00000000  |  00000001  |
 -------------------------

Index

• Variables
• func NewListener(listeners []net.Listener, stats []StatsTracker) net.Listener
• func ReadVarInt(b io.ByteReader) (uint64, error)
• func VarIntLen(i uint64) int
• func WriteVarInt(b *bytes.Buffer, i uint64)
• type Dialer
  • func NewDialer(dest string, dialers []Dialer) Dialer
• type NullTracker
  • func (st NullTracker) OnRecv(uint64)
  • func (st NullTracker) OnRetransmit(uint64)
  • func (st NullTracker) OnSent(uint64)
  • func (st NullTracker) UpdateRTT(time.Duration)
• type Stats
• type StatsTracker

Variables

var (
	ErrUnexpectedVersion = errors.New("unexpected version")
	ErrUnexpectedCID     = errors.New("unexpected connnection ID")
	ErrClosed            = errors.New("closed connection")
	ErrFailOnAllDialers  = errors.New("fail on all dialers")
)

Functions

func NewListener

func NewListener(listeners []net.Listener, stats []StatsTracker) net.Listener

func ReadVarInt

func ReadVarInt(b io.ByteReader) (uint64, error)

ReadVarInt reads a number in the QUIC varint format

func VarIntLen

func VarIntLen(i uint64) int

VarIntLen determines the number of bytes that will be needed to write a number

func WriteVarInt

func WriteVarInt(b *bytes.Buffer, i uint64)

WriteVarInt writes a number in the QUIC varint format

Types

type Dialer

type Dialer interface {
	DialContext(ctx context.Context) (net.Conn, error)
	Label() string
}

Dialer is the interface each subflow dialer needs to satisify. It is also the type of the multipath dialer.

func NewDialer

func NewDialer(dest string, dialers []Dialer) Dialer

type NullTracker

type NullTracker struct{}

func (NullTracker) OnRecv

func (st NullTracker) OnRecv(uint64)

func (NullTracker) OnRetransmit

func (st NullTracker) OnRetransmit(uint64)

func (NullTracker) OnSent

func (st NullTracker) OnSent(uint64)

func (NullTracker) UpdateRTT

func (st NullTracker) UpdateRTT(time.Duration)

type Stats

type Stats interface {
	FormatStats() (stats []string)
}

Stats is also provided by the multipath dialer so the caller can get and print the status of each path.

type StatsTracker

type StatsTracker interface {
	OnRecv(uint64)
	OnSent(uint64)
	OnRetransmit(uint64)
	UpdateRTT(time.Duration)
}

StatsTracker allows getting a sense of how the paths perform. Its methods are called when each subflow sends or receives a frame.