derp

README

DERP

This directory (and subdirectories) contain the DERP code. The server itself is in ../cmd/derper.

DERP is a packet relay system (client and servers) where peers are addressed using WireGuard public keys instead of IP addresses.

It relays two types of packets:

• "Disco" discovery messages (see ../disco) as the a side channel during NAT traversal.

• Encrypted WireGuard packets as the fallback of last resort when UDP is blocked or NAT traversal fails.

DERP Map

Each client receives a "DERP Map" from the coordination server describing the DERP servers the client should try to use.

The client picks its home "DERP home" based on latency. This is done to keep costs low by avoid using cloud load balancers (pricey) or anycast, which would necessarily require server-side routing between DERP regions.

Clients pick their DERP home and report it to the coordination server which shares it to all the peers in the tailnet. When a peer wants to send a packet and it doesn't already have a WireGuard session open, it sends disco messages (some direct, and some over DERP), trying to do the NAT traversal. The client will make connections to multiple DERP regions as needed. Only the DERP home region connection needs to be alive forever.

DERP Regions

Tailscale runs 1 or more DERP nodes (instances of cmd/derper) in various geographic regions to make sure users have low latency to their DERP home.

Regions generally have multiple nodes per region "meshed" (routing to each other) together for redundancy: it allows for cloud failures or upgrades without kicking users out to a higher latency region. Instead, clients will reconnect to the next node in the region. Each node in the region is required to to be meshed with every other node in the region and forward packets to the other nodes in the region. Packets are forwarded only one hop within the region. There is no routing between regions. The assumption is that the mesh TCP connections are over a VPC that's very fast, low latency, and not charged per byte. The coordination server assigns the list of nodes in a region as a function of the tailnet, so all nodes within a tailnet should generally be on the same node and not require forwarding. Only after a failure do clients of a particular tailnet get split between nodes in a region and require inter-node forwarding. But over time it balances back out. There's also an admin-only DERP frame type to force close the TCP connection of a particular client to force them to reconnect to their primary if the operator wants to force things to balance out sooner. (Using the (*derphttp.Client).ClosePeer method, as used by Tailscale's internal rarely-used cmd/derpprune maintenance tool)

We generally run a minimum of three nodes in a region not for quorum reasons (there's no voting) but just because two is too uncomfortably few for cascading failure reasons: if you're running two nodes at 51% load (CPU, memory, etc) and then one fails, that makes the second one fail. With three or more nodes, you can run each node a bit hotter.

Documentation

Overview

Package derp implements the Designated Encrypted Relay for Packets (DERP) protocol.

DERP routes packets to clients using curve25519 keys as addresses.

DERP is used by Tailscale nodes to proxy encrypted WireGuard packets through the Tailscale cloud servers when a direct path cannot be found or opened. DERP is a last resort. Both sides between very aggressive NATs, firewalls, no IPv6, etc? Well, DERP.

Index

• Constants
• func ReadFrameTypeHeader(br *bufio.Reader, wantType FrameType) (frameLen uint32, err error)
• func WriteFrame(bw *bufio.Writer, t FrameType, b []byte) error
• func WriteFrameHeader(bw *bufio.Writer, t FrameType, frameLen uint32) error
• type Client
  • func NewClient(privateKey key.NodePrivate, nc Conn, brw *bufio.ReadWriter, logf logger.Logf, ...) (*Client, error)
  • func (c *Client) ClosePeer(target key.NodePublic) error
  • func (c *Client) ForwardPacket(srcKey, dstKey key.NodePublic, pkt []byte) (err error)
  • func (c *Client) LocalAddr() (netip.AddrPort, error)
  • func (c *Client) NotePreferred(preferred bool) (err error)
  • func (c *Client) PublicKey() key.NodePublic
  • func (c *Client) Recv() (m ReceivedMessage, err error)
  • func (c *Client) Send(dstKey key.NodePublic, pkt []byte) error
  • func (c *Client) SendPing(data [8]byte) error
  • func (c *Client) SendPong(data [8]byte) error
  • func (c *Client) ServerPublicKey() key.NodePublic
  • func (c *Client) WatchConnectionChanges() error
• type ClientInfo
  • func (c *ClientInfo) Equal(other *ClientInfo) bool
• type ClientOpt
  • func CanAckPings(v bool) ClientOpt
  • func IsProber(v bool) ClientOpt
  • func MeshKey(k key.DERPMesh) ClientOpt
  • func ServerPublicKey(key key.NodePublic) ClientOpt
• type Conn
• type FrameType
  • func ReadFrameHeader(br *bufio.Reader) (t FrameType, frameLen uint32, err error)
• type HealthMessage
• type KeepAliveMessage
• type PeerGoneMessage
• type PeerGoneReasonType
• type PeerPresentFlags
• type PeerPresentMessage
• type PingMessage
• type PongMessage
• type ReceivedMessage
• type ReceivedPacket
• type ServerInfo
• type ServerInfoMessage
• type ServerRestartingMessage

Constants

const (
	NonceLen       = 24
	FrameHeaderLen = 1 + 4 // frameType byte + 4 byte length
	KeyLen         = 32
	MaxInfoLen     = 1 << 20
)

const (
	FrameServerKey     = FrameType(0x01) // 8B magic + 32B public key + (0+ bytes future use)
	FrameClientInfo    = FrameType(0x02) // 32B pub key + 24B nonce + naclbox(json)
	FrameServerInfo    = FrameType(0x03) // 24B nonce + naclbox(json)
	FrameSendPacket    = FrameType(0x04) // 32B dest pub key + packet bytes
	FrameForwardPacket = FrameType(0x0a) // 32B src pub key + 32B dst pub key + packet bytes
	FrameRecvPacket    = FrameType(0x05) // v0/1: packet bytes, v2: 32B src pub key + packet bytes
	FrameKeepAlive     = FrameType(0x06) // no payload, no-op (to be replaced with ping/pong)
	FrameNotePreferred = FrameType(0x07) // 1 byte payload: 0x01 or 0x00 for whether this is client's home node

	// framePeerGone is sent from server to client to signal that
	// a previous sender is no longer connected. That is, if A
	// sent to B, and then if A disconnects, the server sends
	// framePeerGone to B so B can forget that a reverse path
	// exists on that connection to get back to A. It is also sent
	// if A tries to send a CallMeMaybe to B and the server has no
	// record of B
	FramePeerGone = FrameType(0x08) // 32B pub key of peer that's gone + 1 byte reason

	// framePeerPresent is like framePeerGone, but for other members of the DERP
	// region when they're meshed up together.
	//
	// The message is at least 32 bytes (the public key of the peer that's
	// connected). If there are at least 18 bytes remaining after that, it's the
	// 16 byte IP + 2 byte BE uint16 port of the client. If there's another byte
	// remaining after that, it's a PeerPresentFlags byte.
	// While current servers send 41 bytes, old servers will send fewer, and newer
	// servers might send more.
	FramePeerPresent = FrameType(0x09)

	// frameWatchConns is how one DERP node in a regional mesh
	// subscribes to the others in the region.
	// There's no payload. If the sender doesn't have permission, the connection
	// is closed. Otherwise, the client is initially flooded with
	// framePeerPresent for all connected nodes, and then a stream of
	// framePeerPresent & framePeerGone has peers connect and disconnect.
	FrameWatchConns = FrameType(0x10)

	// frameClosePeer is a privileged frame type (requires the
	// mesh key for now) that closes the provided peer's
	// connection. (To be used for cluster load balancing
	// purposes, when clients end up on a non-ideal node)
	FrameClosePeer = FrameType(0x11) // 32B pub key of peer to close.

	FramePing = FrameType(0x12) // 8 byte ping payload, to be echoed back in framePong
	FramePong = FrameType(0x13) // 8 byte payload, the contents of the ping being replied to

	// frameHealth is sent from server to client to tell the client
	// if their connection is unhealthy somehow. Currently the only unhealthy state
	// is whether the connection is detected as a duplicate.
	// The entire frame body is the text of the error message. An empty message
	// clears the error state.
	FrameHealth = FrameType(0x14)

	// frameRestarting is sent from server to client for the
	// server to declare that it's restarting. Payload is two big
	// endian uint32 durations in milliseconds: when to reconnect,
	// and how long to try total. See ServerRestartingMessage docs for
	// more details on how the client should interpret them.
	FrameRestarting = FrameType(0x15)
)

Protocol flow:

Login: * client connects * server sends frameServerKey * client sends frameClientInfo * server sends frameServerInfo

Steady state: * server occasionally sends frameKeepAlive (or framePing) * client responds to any framePing with a framePong * client sends frameSendPacket * server then sends frameRecvPacket to recipient

const (
	PeerGoneReasonDisconnected  = PeerGoneReasonType(0x00) // is only sent when a peer disconnects from this server
	PeerGoneReasonNotHere       = PeerGoneReasonType(0x01) // server doesn't know about this peer
	PeerGoneReasonMeshConnBroke = PeerGoneReasonType(0xf0) // invented by Client.RunWatchConnectionLoop on disconnect; not sent on the wire
)

const (
	PeerPresentIsRegular  = 1 << 0
	PeerPresentIsMeshPeer = 1 << 1
	PeerPresentIsProber   = 1 << 2
	PeerPresentNotIdeal   = 1 << 3 // client said derp server is not its Region.Nodes[0] ideal node
)

PeerPresentFlags bits.

const FastStartHeader = "Derp-Fast-Start"

FastStartHeader is the header (with value "1") that signals to the HTTP server that the DERP HTTP client does not want the HTTP 101 response headers and it will begin writing & reading the DERP protocol immediately following its HTTP request.

const IdealNodeHeader = "Ideal-Node"

IdealNodeHeader is the HTTP request header sent on DERP HTTP client requests to indicate that they're connecting to their ideal (Region.Nodes[0]) node. The HTTP header value is the name of the node they wish they were connected to. This is an optional header.

const KeepAlive = 60 * time.Second

KeepAlive is the minimum frequency at which the DERP server sends keep alive frames. The server adds some jitter, so this timing is not exact, but 2x this value can be considered a missed keep alive.

const Magic = "DERP🔑" // 8 bytes: 0x44 45 52 50 f0 9f 94 91

Magic is the DERP Magic number, sent in the frameServerKey frame upon initial connection.

const MaxPacketSize = 64 << 10

MaxPacketSize is the maximum size of a packet sent over DERP. (This only includes the data bytes visible to magicsock, not including its on-wire framing overhead)

const ProtocolVersion = 2

ProtocolVersion is bumped whenever there's a wire-incompatible change.

• version 1 (zero on wire): consistent box headers, in use by employee dev nodes a bit
• version 2: received packets have src addrs in frameRecvPacket at beginning

Functions

func ReadFrameTypeHeader

func ReadFrameTypeHeader(br *bufio.Reader, wantType FrameType) (frameLen uint32, err error)

ReadFrameTypeHeader reads a frame header from br and verifies that the frame type matches wantType.

If it does, it returns the frame length (not including the 5 byte header) and a nil error.

If it doesn't, it returns an error and a zero length.

func WriteFrame

func WriteFrame(bw *bufio.Writer, t FrameType, b []byte) error

WriteFrame writes a complete frame & flushes it.

func WriteFrameHeader

func WriteFrameHeader(bw *bufio.Writer, t FrameType, frameLen uint32) error

WriteFrameHeader writes a frame header to bw.

The frame header is 5 bytes: a one byte frame type followed by a big-endian uint32 length of the remaining frame (not including the 5 byte header).

It does not flush bw.

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client is a DERP client.

func NewClient

func NewClient(privateKey key.NodePrivate, nc Conn, brw *bufio.ReadWriter, logf logger.Logf, opts ...ClientOpt) (*Client, error)

func (*Client) ClosePeer

func (c *Client) ClosePeer(target key.NodePublic) error

ClosePeer asks the server to close target's TCP connection. It's a fatal error if the client wasn't created using MeshKey.

func (*Client) ForwardPacket

func (c *Client) ForwardPacket(srcKey, dstKey key.NodePublic, pkt []byte) (err error)

func (*Client) LocalAddr

func (c *Client) LocalAddr() (netip.AddrPort, error)

LocalAddr returns the TCP connection's local address.

If the client is broken in some previously detectable way, it returns an error.

func (*Client) NotePreferred

func (c *Client) NotePreferred(preferred bool) (err error)

NotePreferred sends a packet that tells the server whether this client is the user's preferred server. This is only used in the server for stats.

func (*Client) PublicKey

func (c *Client) PublicKey() key.NodePublic

func (*Client) Recv

func (c *Client) Recv() (m ReceivedMessage, err error)

Recv reads a message from the DERP server.

The returned message may alias memory owned by the Client; it should only be accessed until the next call to Client.

Once Recv returns an error, the Client is dead forever.

func (*Client) Send

func (c *Client) Send(dstKey key.NodePublic, pkt []byte) error

Send sends a packet to the Tailscale node identified by dstKey.

It is an error if the packet is larger than 64KB.

func (*Client) SendPing

func (c *Client) SendPing(data [8]byte) error

func (*Client) SendPong

func (c *Client) SendPong(data [8]byte) error

func (*Client) ServerPublicKey

func (c *Client) ServerPublicKey() key.NodePublic

ServerPublicKey returns the server's public key.

func (*Client) WatchConnectionChanges

func (c *Client) WatchConnectionChanges() error

WatchConnectionChanges sends a request to subscribe to the peer's connection list. It's a fatal error if the client wasn't created using MeshKey.

type ClientInfo

type ClientInfo struct {
	// MeshKey optionally specifies a pre-shared key used by
	// trusted clients.  It's required to subscribe to the
	// connection list & forward packets. It's empty for regular
	// users.
	MeshKey key.DERPMesh `json:"meshKey,omitempty,omitzero"`

	// Version is the DERP protocol version that the client was built with.
	// See the ProtocolVersion const.
	Version int `json:"version,omitempty"`

	// CanAckPings is whether the client declares it's able to ack
	// pings.
	CanAckPings bool

	// IsProber is whether this client is a prober.
	IsProber bool `json:",omitempty"`
}

ClientInfo is the information a DERP client sends to the server about itself when it connects.

func (*ClientInfo) Equal

func (c *ClientInfo) Equal(other *ClientInfo) bool

Equal reports if two clientInfo values are equal.

type ClientOpt

type ClientOpt interface {
	// contains filtered or unexported methods
}

ClientOpt is an option passed to NewClient.

func CanAckPings

func CanAckPings(v bool) ClientOpt

CanAckPings returns a ClientOpt to set whether it advertises to the server that it's capable of acknowledging ping requests.

func IsProber

func IsProber(v bool) ClientOpt

IsProber returns a ClientOpt to pass to the DERP server during connect to declare that this client is a a prober.

func MeshKey

func MeshKey(k key.DERPMesh) ClientOpt

MeshKey returns a ClientOpt to pass to the DERP server during connect to get access to join the mesh.

An empty key means to not use a mesh key.

func ServerPublicKey

func ServerPublicKey(key key.NodePublic) ClientOpt

ServerPublicKey returns a ClientOpt to declare that the server's DERP public key is known. If key is the zero value, the returned ClientOpt is a no-op.

type Conn

type Conn interface {
	io.WriteCloser
	LocalAddr() net.Addr
	// The *Deadline methods follow the semantics of net.Conn.
	SetDeadline(time.Time) error
	SetReadDeadline(time.Time) error
	SetWriteDeadline(time.Time) error
}

Conn is the subset of the underlying net.Conn the DERP Server needs. It is a defined type so that non-net connections can be used.

type FrameType

type FrameType byte

FrameType is the one byte frame type at the beginning of the frame header. The second field is a big-endian uint32 describing the length of the remaining frame (not including the initial 5 bytes).

func ReadFrameHeader

func ReadFrameHeader(br *bufio.Reader) (t FrameType, frameLen uint32, err error)

ReadFrameHeader reads the header of a DERP frame, reading 5 bytes from br.

type HealthMessage

type HealthMessage struct {
	// Problem, if non-empty, is a description of why the connection
	// is unhealthy.
	//
	// The empty string means the connection is healthy again.
	//
	// The default condition is healthy, so the server doesn't
	// broadcast a HealthMessage until a problem exists.
	Problem string
}

HealthMessage is a one-way message from server to client, declaring the connection health state.

type KeepAliveMessage

type KeepAliveMessage struct{}

KeepAliveMessage is a one-way empty message from server to client, just to keep the connection alive. It's like a PingMessage, but doesn't solicit a reply from the client.

type PeerGoneMessage

type PeerGoneMessage struct {
	Peer   key.NodePublic
	Reason PeerGoneReasonType
}

PeerGoneMessage is a ReceivedMessage that indicates that the client identified by the underlying public key is not connected to this server.

It has only historically been sent by the server when the client connection count decremented from 1 to 0 and not from e.g. 2 to 1. See https://github.com/tailscale/tailscale/issues/13566 for details.

type PeerGoneReasonType

type PeerGoneReasonType byte

PeerGoneReasonType is a one byte reason code explaining why a server does not have a path to the requested destination.

type PeerPresentFlags

type PeerPresentFlags byte

PeerPresentFlags is an optional byte of bit flags sent after a framePeerPresent message.

For a modern server, the value should always be non-zero. If the value is zero, that means the server doesn't support this field.

type PeerPresentMessage

type PeerPresentMessage struct {
	// Key is the public key of the client.
	Key key.NodePublic
	// IPPort is the remote IP and port of the client.
	IPPort netip.AddrPort
	// Flags is a bitmask of info about the client.
	Flags PeerPresentFlags
}

PeerPresentMessage is a ReceivedMessage that indicates that the client is connected to the server. (Only used by trusted mesh clients)

It will be sent to client watchers for every new connection from a client, even if the client's already connected with that public key. See https://github.com/tailscale/tailscale/issues/13566 for PeerPresentMessage and PeerGoneMessage not being 1:1.

type PingMessage

type PingMessage [8]byte

PingMessage is a request from a client or server to reply to the other side with a PongMessage with the given payload.

type PongMessage

type PongMessage [8]byte

PongMessage is a reply to a PingMessage from a client or server with the payload sent previously in a PingMessage.

type ReceivedMessage

type ReceivedMessage interface {
	// contains filtered or unexported methods
}

ReceivedMessage represents a type returned by Client.Recv. Unless otherwise documented, the returned message aliases the byte slice provided to Recv and thus the message is only as good as that buffer, which is up to the caller.

type ReceivedPacket

type ReceivedPacket struct {
	Source key.NodePublic
	// Data is the received packet bytes. It aliases the memory
	// passed to Client.Recv.
	Data []byte
}

ReceivedPacket is a ReceivedMessage representing an incoming packet.

type ServerInfo

type ServerInfo struct {
	Version int `json:"version,omitempty"`

	TokenBucketBytesPerSecond int `json:",omitempty"`
	TokenBucketBytesBurst     int `json:",omitempty"`
}

ServerInfo is the message sent from the server to clients during the connection setup.

type ServerInfoMessage

type ServerInfoMessage struct {
	// TokenBucketBytesPerSecond is how many bytes per second the
	// server says it will accept, including all framing bytes.
	//
	// Zero means unspecified. There might be a limit, but the
	// client need not try to respect it.
	TokenBucketBytesPerSecond int

	// TokenBucketBytesBurst is how many bytes the server will
	// allow to burst, temporarily violating
	// TokenBucketBytesPerSecond.
	//
	// Zero means unspecified. There might be a limit, but the
	// client need not try to respect it.
	TokenBucketBytesBurst int
}

ServerInfoMessage is sent by the server upon first connect.

type ServerRestartingMessage

type ServerRestartingMessage struct {
	// ReconnectIn is an advisory duration that the client should wait
	// before attempting to reconnect. It might be zero.
	// It exists for the server to smear out the reconnects.
	ReconnectIn time.Duration

	// TryFor is an advisory duration for how long the client
	// should attempt to reconnect before giving up and proceeding
	// with its normal connection failure logic. The interval
	// between retries is undefined for now.
	// A server should not send a TryFor duration more than a few
	// seconds.
	TryFor time.Duration
}

ServerRestartingMessage is a one-way message from server to client, advertising that the server is restarting.