magictls

README

MagicTLS

A simple Go library that detects protocol automatically:

• Support for PROXY and PROXYv2 allows detecting the real user's IP when, for example, using AWS elastic load balancers. The fact the protocol is detected automatically allows the daemon to work even before ELB is properly configured, and avoid rejecting requests by mistake.
• Automatic TLS support allows using a single port for SSL and non-SSL traffic, and simplifies configuration.
• Allows creating listener for specific TLS negociated protocols, allowing a single port to be used for many things easily.

This library was used in some of my projects, I've cleaned it up and licensed it under the MIT License since it's small and useful. Pull requests welcome.

It is written to work with protocols where the client sends the first data, and it expects the client to send at least 16 bytes. This works nicely with HTTP (GET / HTTP/1.0\r\n is exactly 16 bytes), SSL, etc. but may not work with protocols such as POP3, IMAP or SMTP where the server is expected to send the first bytes unless TLS is required. In this case using the ForceTLS filter only allows to still benefit from the TLS NextProto routing.

Usage

Use magictls.Listen() to create sockets the same way you would use tls.Listen().

socket, err := magictls.Listen("tcp", ":8080", tlsConfig)
if err != nil {
	...
}
log.Fatal(http.Serve(socket, handler))

The created listener can receive various configurations. For example if you need to force all connections to be TLS and only want to use PROXY protocol detection:

socket, err := magictls.Listen("tcp", ":8443", tlsConfig)
if err != nil {
	...
}
socket.Filters = []magictls.Filter{magictls.DetectProxy, magictls.ForceTLS}
log.Fatal(http.Serve(socket, handler))

It is also possible to implement your own filters.

PROXY protocol allowed IPs

Depending on your provider, you may need to allow more than the local IPs for PROXY protocol.

For example Google Cloud's global load balancer uses a wider range of IPs that may come with PROXY requests and need this to be called:

magictls.AddAllowedProxies("35.191.0.0/16", "130.211.0.0/22", "2600:2d00:1:b029::/64", "2600:2d00:1:1::/64")
magictls.AddAllowedProxiesSpf("_cloud-eoips.googleusercontent.com")

autocert

This can be used with autocert too for automatic TLS certificates. Note that in this case you are required to have a listener on port 443.

// initialize autocert structure
m := &autocert.Manager{
	Prompt: autocert.AcceptTOS,
	HostPolicy: autocert.HostWhitelist("domain", "domain2"),
	Cache: autocert.DirCache("/tmp"), // use os.UserCacheDir() to find where to put that
}
// grab autocert TLS config
cfg := m.TLSConfig()
// you may want to add to cfg.NextProtos any protocol you want to handle with ProtoListener. Be careful to not overwrite it.
cfg.NextProtos = append(cfg.NextProtos, "my-proto")
// standard listen
socket, err := magictls.Listen("tcp", ":443", cfg)
if err != nil {
	...
}
...

Documentation

Index

• Variables
• func AddAllowedProxies(cidrs ...string) error
• func AddAllowedProxiesSpf(spfhosts ...string) error
• func DetectProxy(cw *Conn, srv *Listener) error
• func DetectTLS(conn *Conn, srv *Listener) error
• func ForceTLS(conn *Conn, srv *Listener) error
• func GetTlsConn(c net.Conn) *tls.Conn
• func HijackedConn(c net.Conn, io *bufio.ReadWriter, err error) (net.Conn, error)
• func SetAllowedProxies(cidrs ...string) error
• type Conn
  • func (c *Conn) Close() error
  • func (c *Conn) LocalAddr() net.Addr
  • func (c *Conn) PeekMore(count int) ([]byte, error)
  • func (c *Conn) PeekUntil(count int) ([]byte, error)
  • func (c *Conn) Read(b []byte) (int, error)
  • func (c *Conn) RemoteAddr() net.Addr
  • func (c *Conn) SetDeadline(t time.Time) error
  • func (c *Conn) SetLocalAddr(l net.Addr)
  • func (c *Conn) SetReadDeadline(t time.Time) error
  • func (c *Conn) SetRemoteAddr(r net.Addr)
  • func (c *Conn) SetWriteDeadline(t time.Time) error
  • func (c *Conn) SkipPeek(count int)
  • func (c *Conn) Unwrap() net.Conn
  • func (c *Conn) Write(b []byte) (int, error)
• type Filter
• type Listener
  • func Listen(network, laddr string, config *tls.Config) (*Listener, error)
  • func ListenNull() *Listener
  • func (r *Listener) Accept() (net.Conn, error)
  • func (r *Listener) Addr() net.Addr
  • func (r *Listener) Close() error
  • func (r *Listener) GetRunningThreads() uint32
  • func (r *Listener) HandleConn(c net.Conn, filterOverride []Filter)
  • func (r *Listener) Listen(network, laddr string) error
  • func (r *Listener) ListenFilter(network, laddr string, filters []Filter) error
  • func (r *Listener) ProtoListener(proto ...string) (net.Listener, error)
  • func (r *Listener) PushConn(c net.Conn)
  • func (r *Listener) SetThreads(count uint32)
  • func (p *Listener) String() string
• type Override
  • func (o *Override) Error() string

Variables

var ErrDuplicateProtocol = errors.New("protocol already has a listener")

Functions

func AddAllowedProxies

func AddAllowedProxies(cidrs ...string) error

AddAllowedProxies adds to the list of allowed proxies

func AddAllowedProxiesSpf

func AddAllowedProxiesSpf(spfhosts ...string) error

AddAllowedProxiesSpf will perform TXT lookup on the given hosts and add those IPs as allowed proxies. This is only performed once and may need to be refreshed from times to times.

func DetectProxy

func DetectProxy(cw *Conn, srv *Listener) error

DetectProxy is a magictls filter that will detect proxy protocol headers (both versions) and update local/remote addr based on these if the source is an allowed proxy (see SetAllowedProxies).

func DetectTLS

func DetectTLS(conn *Conn, srv *Listener) error

DetectTLS is a magictls filter that will attempt to detect if the connection is a TLS client. This best works with protocols where the first byte is expected to be an ASCII character, such as HTTP. This will not work well if the client is not sending the first message.

func ForceTLS

func ForceTLS(conn *Conn, srv *Listener) error

ForceTLS is a magictls filter that will engage TLS mode.

func GetTlsConn

func GetTlsConn(c net.Conn) *tls.Conn

GetTlsConn will attempt to unwrap the given connection in order to locate a TLS connection, or return nil if none found.

func HijackedConn

func HijackedConn(c net.Conn, io *bufio.ReadWriter, err error) (net.Conn, error)

HijackedConn allows returning a simple net.Conn from a Conn+ReadWriter as returned by http.Hijacker.Hijack()

func SetAllowedProxies

func SetAllowedProxies(cidrs ...string) error

SetAllowedProxies allows modifying the list of IP addresses allowed to use proxy protocol. Any host matching a CIDR listed in here will be trusted to provide the client's real IP.

By default all local IPs are allowed as these cannot appear on Internet.

SetAllowedProxies("127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "::1/128", "fd00::/8")

Types

type Conn

type Conn struct {
	// contains filtered or unexported fields
}

Conn is used to prepend data to the data stream when we need to unread what we've read. It can be used as a net.Conn.

func (*Conn) Close

func (c *Conn) Close() error

func (*Conn) LocalAddr

func (c *Conn) LocalAddr() net.Addr

func (*Conn) PeekMore

func (c *Conn) PeekMore(count int) ([]byte, error)

PeekMore will perform a single read from the socket, and return the data read so far. May return an error if the socket was closed (in which case data may still be returned if it was read before).

func (*Conn) PeekUntil

func (c *Conn) PeekUntil(count int) ([]byte, error)

PeekUntil will block until at least count bytes were read, or an error happens.

func (*Conn) Read

func (c *Conn) Read(b []byte) (int, error)

func (*Conn) RemoteAddr

func (c *Conn) RemoteAddr() net.Addr

func (*Conn) SetDeadline

func (c *Conn) SetDeadline(t time.Time) error

func (*Conn) SetLocalAddr

func (c *Conn) SetLocalAddr(l net.Addr)

func (*Conn) SetReadDeadline

func (c *Conn) SetReadDeadline(t time.Time) error

func (*Conn) SetRemoteAddr

func (c *Conn) SetRemoteAddr(r net.Addr)

func (*Conn) SetWriteDeadline

func (c *Conn) SetWriteDeadline(t time.Time) error

func (*Conn) SkipPeek

func (c *Conn) SkipPeek(count int)

SkipPeek will skip count bytes from the peek buffer, or strip the whole buffer if count is larger or equal to the buffer.

func (*Conn) Unwrap

func (c *Conn) Unwrap() net.Conn

func (*Conn) Write

func (c *Conn) Write(b []byte) (int, error)

type Filter

type Filter func(conn *Conn, srv *Listener) error

Filter is a generic magictls filter, to be used when accepting a connection. Default filters provided for convenience include DetectProxy, DetectTLS and ForceTLS.

type Listener

type Listener struct {
	TLSConfig *tls.Config
	Filters   []Filter
	*log.Logger
	Timeout time.Duration
	// contains filtered or unexported fields
}

Listener is a stream network listener supporting TLS and PROXY protocol automatically. It assumes no matter what the used protocol is, at least 16 bytes will always be initially sent (true for HTTP).

func Listen

func Listen(network, laddr string, config *tls.Config) (*Listener, error)

Listen creates a hybrid TCP/TLS listener accepting connections on the given network address using net.Listen. The configuration config must be non-nil and must include at least one certificate or else set GetCertificate. If not, then only PROXY protocol support will be available.

If the connection uses TLS protocol, then Accept() returned net.Conn will actually be a tls.Conn object.

func ListenNull

func ListenNull() *Listener

ListenNull creates a listener that is not actually listening to anything, but can be used to push connections via PushConn. This can be useful to use a http.Server with custom listeners.

func (*Listener) Accept

func (r *Listener) Accept() (net.Conn, error)

Accept blocks until a connection is available, then return said connection or an error if the listener was closed.

func (*Listener) Addr

func (r *Listener) Addr() net.Addr

Addr returns the address the socket is currently listening on, or nil for null listeners.

func (*Listener) Close

func (r *Listener) Close() error

Close() closes the socket.

func (*Listener) GetRunningThreads

func (r *Listener) GetRunningThreads() uint32

GetRunningThreads returns the current number of running threads.

func (*Listener) HandleConn

func (r *Listener) HandleConn(c net.Conn, filterOverride []Filter)

HandleConn will run detection on a given incoming connection and attempt to find if it should parse any kind of PROXY headers, or TLS handshake/etc.

func (*Listener) Listen

func (r *Listener) Listen(network, laddr string) error

Listen makes the given listener listen on an extra port. Each listener will spawn a new goroutine.

func (*Listener) ListenFilter

func (r *Listener) ListenFilter(network, laddr string, filters []Filter) error

ListenFilter listens on a given port with the selected filters used instead of the default ones.

func (*Listener) ProtoListener

func (r *Listener) ProtoListener(proto ...string) (net.Listener, error)

ProtoListener returns a net.Listener that will receive connections for which TLS is enabled and the specified protocol(s) have been negociated between client and server.

func (*Listener) PushConn

func (r *Listener) PushConn(c net.Conn)

PushConn allows pushing an existing connection to the queue as if it had just been accepted by the server. No auto-detection will be performed.

func (*Listener) SetThreads

func (r *Listener) SetThreads(count uint32)

SetThreads sets the number of threads (goroutines) magictls will spawn in parallel when handling incoming connections. Note that once a connection leaves Accept() it is not tracked anymore. Filters will however run in parallel for those connections, meaning that one connection's handshake taking time will not block other connections.

func (*Listener) String

func (p *Listener) String() string

type Override

type Override struct {
	Conn     net.Conn
	Protocol string
}

special type returned as error by filters to return a different Conn

func (*Override) Error

func (o *Override) Error() string