ugate

README

ugate

Minimal TCP/UDP gateway, optimized for capturing outbound connections using few common protocols (iptables, SOCKS, CONNECT), detecting traffic type and extracting metadata, and forwarding to a destination. TUN supported using LWIP in the costinm/tungate, used for android.

Design

The uGate can be used in few roles:

1. Egress sidecar - captures outbound traffic, using SOCKS, iptables or TUN
2. Ingress sidecar - forwards 'mesh' traffic to local app
3. Ingress gateway - receives 'regular' traffic, forwards to the mesh
4. Egress gateway - forwards 'mesh' traffic to the internet.
5. Legacy egress - forwards mesh traffic to local non-mesh devices.
6. Message gate - can proxy WebPush messages, typically for control plane.

It is intended for small devices - android, OpenWRT-like routers, very small containers, so dependencies are minimized.

It is also optimized for battery operation - control plane interacts with the gate via encrypted Webpush or GCM messages.

Mesh protocols

Sidecars send and receive mesh traffic using HTTP/2 or WebRTC. For HTTP/2, a custom SNI header is used.

Gateways use SNI to route, using splice after the header is parsed.

Since uGate is optimized for devices which may be in home nets or in p2p ad-hoc networks, inside the mesh it will support WebRTC communication. This is the main external dependency, used in the webrtc/ module.

ReadFrom/splice

By avoiding wrappers and abstractions we can detect if both input and output are TcpConn and use the 'splice' call, where the transfer between in and out is done in kernel space.

For example, current numbers on my server, using iperf3, in Gbps:

• direct/local (-c 5201): 29
• capture+proxy without splice: 11Gbps
• capture+proxy - splice: 21Gbps,
• capture + tlsOut + tlsIn + proxy (similar with Istio full path):
• capture + plainOut + plainIn + proxy (similar with Istio plain text):

This is useful for 'TURN'-style proxy, CONNECT and SNI, where the stream is already encrypted e2e and the gateway is not adding an encryption layer. It doesn't help for encrypting, only on the middle boxes where encryption is not needed, just routing.

Splice also helps when the app already encrypts, there is no need for a second encryption. This is handled by auto-detecting TLS.

TODO

• P1: register dialers ( webrtc, quic, etc) for muxed connections and streams

• P1: webrtc listener to create new peerconnection after one is used, dial to use a synth. SDP string.

• UDP

• P2: (separate repo) WebRTC/TURN/STUN compat - check perf against H2 and SNI routing

• P2: K8s compat (konectivity ?), KNative

• P3: raw H2 implementation - just forward the frames, without decoding or re-encoding.

  • per stream flow control will be end-to-end.
  • extended PUSH support

• P1: Relay-over-SNI: register N plain text conns, with a signed/encrypted header. SNI will proxy connections without multiplex or encryption - using splice. Alternative to H2R

• P0: iptable redirect 80 and 443, HTTP proxy with subset of K8S HTTPRoute

• P3: Cert signing, using Istio and/or simplified API (json-grpc?)

• P0: mangled hostname: KEYID.namespace.TRUST_DOMAIN in certs and SNI routes. Use pod ID as SAN, Istio Spiffe based on SA from JWT + namespace

• P2: OIDC auth (to support certs), VAPID extensions for OIDC compat (send cert)

Documentation

Index

• Variables
• type Duration
  • func (ms Duration) MarshalJSON() ([]byte, error)
  • func (ms *Duration) UnmarshalJSON(data []byte) error
• type Host
• type MeshCluster
  • func (n *MeshCluster) String() string
• type MeshSettings
• type Route
• type UDPHandler
• type UGate
  • func New(auth *meshauth.MeshAuth, ms *MeshSettings) *UGate
  • func (hb *UGate) AddCluster(c *MeshCluster, host ...*Host) *MeshCluster
  • func (ug *UGate) Close() error
  • func (hb *UGate) Cluster(ctx context.Context, addr string) (*MeshCluster, error)
  • func (hb *UGate) Dial(n, a string) (net.Conn, error)
  • func (hb *UGate) DialContext(ctx context.Context, network, addr string) (net.Conn, error)
  • func (ug *UGate) DialMUX(ctx context.Context, net string, node *MeshCluster, ...) (http.RoundTripper, error)
  • func (hb *UGate) GetCluster(addr string) *MeshCluster
  • func (hb *UGate) HandleTCPProxy(w io.Writer, r io.Reader, hostPort string) error
  • func (hb *UGate) HandleTUN(nc net.Conn, target *net.TCPAddr, la *net.TCPAddr)
  • func (hb *UGate) HandleUdp(dstAddr net.IP, dstPort uint16, localAddr net.IP, localPort uint16, ...)
  • func (hb *UGate) HttpClient(caCert []byte) *http.Client
  • func (*UGate) NewTLSConnOut(ctx context.Context, nc net.Conn, cfg *meshauth.MeshAuth, remotePeerID string, ...) (nio.Stream, error)
  • func (gw *UGate) OnHClose(s string, id string, san string, r *http.Request, since time.Duration)
  • func (gw *UGate) OnMuxClose(dm *MeshCluster)
  • func (gw *UGate) OnSClose(s nio.Stream, addr net.Addr)
  • func (ug *UGate) OnStream(s nio.Stream)
  • func (ug *UGate) OnStreamDone(str nio.Stream)
  • func (ug *UGate) RegisterProxyStream(s nio.Stream)
  • func (gw *UGate) RemoteID(s nio.Stream) string
  • func (ug *UGate) Start() error
  • func (ug *UGate) StartListener(ll *meshauth.PortListener) error
• type UdpWriter

Variables

var Debug = false

Debug for dev support, will log verbose info. Avoiding dependency on logging - eventually a trace interface will be provided so any logger can be used.

var LogClose = true

var Modules = map[string]func(gate *UGate){}

Modules are used with conditional compiled modules, to reduce deps and binary size. The function will be called when the Gate is created - they may initialize. gate.StartFunctions will be called during Start().

var ServiceUnavailable = errors.New("Service Unavailable 503")

EndpointCon (peer) is over capacity or unavailable.

Types

type Duration

type Duration struct {
	time.Duration
}

func (Duration) MarshalJSON

func (ms Duration) MarshalJSON() ([]byte, error)

func (*Duration) UnmarshalJSON

func (ms *Duration) UnmarshalJSON(data []byte) error

type Host

type Host struct {
	// Labels for the workload. Extracted from pod info - possibly TXT records
	//
	// 'hbone' can be used for a custom hbone endpoint (default 15008).
	//
	Labels map[string]string `json:"labels,omitempty"`

	// Address is an IP where the host can be reached.
	// It can be a real IP (in the mesh, direct) or a jump host.
	//
	Address string `json:"addr,omitempty"`

	// FQDN of the host. Used to check host cert.
	Hostname string
}

Host represents the properties of a single workload. By default, clusters resolve the endpoints dynamically, using DNS or EDS or other discovery mechanisms.

type MeshCluster

type MeshCluster struct {
	// Dest includes the address and auth-related info for the MeshCluster.
	// The meshauth package includes helpers around authentication and certificates, decoupled from mesh.
	meshauth.Dest

	// MeshCluster WorkloadID - the cluster name in kube config, hub, gke - cluster name in XDS
	// Defaults to Base addr - but it is possible to have multiple clusters for
	// same address ( ex. different users / token providers).
	//
	// Examples:
	// GKE cluster: gke_PROJECT_LOCATION_NAME
	//
	// For mesh nodes:
	// ID is the (best) primary id known for the node. Format is:
	//    base32(SHA256(EC_256_pub)) - 32 bytes binary, 52 bytes encoded
	//    base32(ED_pub) - same size, for nodes with ED keys.
	//
	// For non-mesh nodes, it is a (real) domain name or IP if unknown.
	// It may include port, or even be a URL - the external destinations may
	// have different public keys on different ports.
	//
	// The node may be a virtual IP ( ex. K8S/Istio service ) or name
	// of a virtual service.
	//
	// If IPs are used, they must be either truncated SHA or included
	// in the node cert or the control plane must return metadata and
	// secure low-level network is used (like wireguard)
	//
	// Required for secure communication.
	//
	// Examples:
	//  -  [B32_SHA]
	//  -  [B32_SHA].reviews.bookinfo.svc.example.com
	//  -  IP6 (based on SHA or 'trusted' IP)
	//  -  IP4 ('trusted' IP)
	//
	ID string `json:"id,omitempty"`

	// Hosts are workload addresses associated with the backend service.
	//
	// If empty, the MeshCluster Addr will be used directly - it is expected to be
	// a FQDN or VIP that is routable - either a service backed by an LB or handled by
	// ambient or K8S.
	//
	// This may be pre-configured or result of discovery (IPs, extra properties).
	Hosts []*Host
}

MeshCluster represents a set of endpoints, with a common configuration. Can be a K8S Service with VIP and DNS name, an external service, etc.

Similar with Envoy Cluster or K8S service, can also represent single endpoint with multiple paths/IPs. It includes node information, based on registration info or discovery.

Also used for 'mesh' nodes, where we have a public key and other info, as well as non-mesh nodes.

This struct includes statistics about the node and current active association/mux connections.

func (*MeshCluster) String

func (n *MeshCluster) String() string

Textual representation of the node registration data.

type MeshSettings

type MeshSettings struct {
	// SSHConfig includes MeshCfg - which defines the authentication.
	//
	// Current ugate mesh 'core' protocol is SSH, other protocols are bridged/gateway-ed
	// The config is shared with the ssh-mesh project.
	sshd.SSHConfig `json:inline`

	// Additional defaults for outgoing connections.
	// Probably belong to Dest.
	ConnectTimeout Duration `json:"connect_timeout,omitempty"`

	TCPUserTimeout time.Duration

	// Timeout used for TLS or SSH handshakes. If not set, 3 seconds is used.
	HandsahakeTimeout time.Duration

	// Clusters by hostname. The key is primarily a hostname:port, matching Istio/K8S Service name and ports.
	// TODO: do we need the port ? With ztunnel all endpoins can be reached, and the service selector applies
	// to all ports.
	//
	// Generally MeshClusters have different public keys/certs.
	// Includes Nodes, Pods and Services - the key can be the hash of the public key.
	Clusters map[string]*MeshCluster `json:clusters,omitempty"`

	// BasePort is the first port used for the virtual/control ports.
	// For Istio interop, it defaults to 15000 and uses same offsets.
	// This port is used for admin/debug/local MDS, bound to localhost, http protocol
	// Deprecated - use listeners
	BasePort int `json:"basePort,omitempty"`
}

MeshSettings holds the settings for a mesh node.

type Route

type Route struct {
}

type UDPHandler

type UDPHandler interface {
	HandleUdp(dstAddr net.IP, dstPort uint16, localAddr net.IP, localPort uint16, data []byte)
}

UDPHandler is used to abstract the handling of incoming UDP packets on a UDP listener or TUN.

type UGate

type UGate struct {
	*MeshSettings

	// Auth plugs-in mTLS support. The generated configs should perform basic mesh
	// authentication.
	// Typically a *meshauth.MeshAuth
	Auth *meshauth.MeshAuth `json:"-"`

	// AuthProviders - matching kubeconfig user.authProvider.name
	// It is expected to return tokens with the given audience - in case of GCP
	// returns access tokens. If not set the cluster can't be created.
	//
	// A number of pre-defined token sources are used:
	// - gcp - returns GCP access tokens using MDS or default credentials. Used for example by GKE clusters.
	// - k8s - return K8S WorkloadID tokens with the given audience for default K8S cluster.
	// - istio-ca - returns K8S tokens with istio-ca audience - used by Citadel and default Istiod
	// - sts - federated google access tokens associated with GCP identity pools.
	AuthProviders map[string]func(context.Context, string) (string, error)

	// ReverseProxy is used when UGate is used to proxy to a local http/1.1 server.
	ReverseProxy *httputil.ReverseProxy

	// Main HTTP handler - will perform auth, dispatch, etc
	H2Handler http.Handler

	// Mux is used for HTTP and gRPC handler exposed externally.
	//
	// It is the handler for "hbone" and "hbonec" protocol handlers.
	//
	// The HTTP server on localhost:15000 uses http.DefaultMux - which is used by pprof
	// and others by default.
	Mux *http.ServeMux

	// MuxDialers are used to create an association with a peer and multiplex connections.
	// HBone, SSH, etc can act as mux dialers.
	MuxDialers map[string]meshauth.ContextDialer

	ListenerProto map[string]func(gate *UGate, l *meshauth.PortListener) error

	Client *http.Client

	Http11Transport *http.Transport

	// Default dialer used to connect to host:port extracted from metadata.
	// Defaults to net.Dialer, making real connections.
	//
	// Can be replaced with a mux or egress dialer or router for
	// integration.
	NetDialer meshauth.ContextDialer

	// Used for udp proxy, when a captured packet is received.
	DNS        UDPHandler
	UDPHandler UDPHandler

	// Active connection by stream tuple, for MDS and debug.
	// This is primarily used for proxied connection, to allow the receiver to get metadata
	// (certs, real caller, etc)
	ActiveTcp map[string]nio.Stream

	TcpConActive *expvar.Int
	TcpConTotal  expvar.Int

	// template, used for TLS connections and the host WorkloadID
	TLSConfig *tls.Config

	StartFunctions []func(ug *UGate)
	// contains filtered or unexported fields
}

UGate represents a node using a HTTP/2 or HTTP/3 based overlay network environment. This can act as a minimal REST client and server - or can be used as a RoundTripper, Dialer and PortListener compatible with HBONE protocol and mesh security.

UGate by default uses mTLS, using spiffee identities encoding K8S namespace, KSA and a trust domain. Other forms of authentication can be supported - auth is handled via configurable interface, not part of the core package.

UGate can be used as a client, server or proxy/gateway.

func New

func New(auth *meshauth.MeshAuth, ms *MeshSettings) *UGate

New creates a new UGate node. It requires a workload identity, including mTLS certificates.

func (*UGate) AddCluster

func (hb *UGate) AddCluster(c *MeshCluster, host ...*Host) *MeshCluster

AddCluster will add a cluster to be used for Dial and RoundTrip. The 'Addr' field can be a host:port or IP:port. If id is set, it can be host:port or hostname - will be added as a destination. The service can be IP:port or URLs

func (*UGate) Close

func (ug *UGate) Close() error

func (*UGate) Cluster

func (hb *UGate) Cluster(ctx context.Context, addr string) (*MeshCluster, error)

Cluster will get an existing cluster or create a dynamic one. Dynamic clusters can be GC and loaded on-demand.

func (*UGate) Dial

func (hb *UGate) Dial(n, a string) (net.Conn, error)

Dial calls @See DialContext

func (*UGate) DialContext

func (hb *UGate) DialContext(ctx context.Context, network, addr string) (net.Conn, error)

DialContext dials a destination address (host:port). This can be used in applications as a TCP Dial replacement.

It will first attempt to look up the host config, and if it supports 'mesh' will use a secure, multiplexed connection.

func (*UGate) DialMUX

func (ug *UGate) DialMUX(ctx context.Context, net string, node *MeshCluster, ev func(t string, stream nio.Stream)) (http.RoundTripper, error)

DialMUX creates an association with the node, using one of the supported transports.

The node should have at least the address or public key or hash populated.

func (*UGate) GetCluster

func (hb *UGate) GetCluster(addr string) *MeshCluster

GetCluster returns a cluster for the given address, or nil if not found.

func (*UGate) HandleTCPProxy

func (hb *UGate) HandleTCPProxy(w io.Writer, r io.Reader, hostPort string) error

HandleTCPProxy connects and forwards r/w to the hostPort

func (*UGate) HandleTUN

func (hb *UGate) HandleTUN(nc net.Conn, target *net.TCPAddr, la *net.TCPAddr)

HanldeTUN is called when a TCP egress connection is intercepted via TProxy or TUN (gVisor or lwip) target is the destination address, la is the local address (the connection will have it reversed).

func (*UGate) HandleUdp

func (hb *UGate) HandleUdp(dstAddr net.IP, dstPort uint16,
	localAddr net.IP, localPort uint16,
	data []byte)

HandleUdp is the common entry point for UDP capture. - tproxy - gvisor/lwIP WIP

func (*UGate) HttpClient

func (hb *UGate) HttpClient(caCert []byte) *http.Client

HttpClient returns a http.Client configured with the specified root CA, and reasonable settings. The URest wrapper is added, for telemetry or other interceptors.

func (*UGate) NewTLSConnOut

func (*UGate) NewTLSConnOut(ctx context.Context, nc net.Conn,
	cfg *meshauth.MeshAuth,
	remotePeerID string, alpn []string) (nio.Stream, error)

DialTLS dials a TLS connection to addr and does the handshake. It opens a direct TLS connection using the dialer for TCP. No peer verification - the returned stream will have the certs. addr is a real internet address, not a mesh one.

Used internally to create the raw TLS connections to both mesh and non-mesh nodes. Do a TLS handshake on the plain text nc. Verify the server identity using a remotePeerID - based on public key. TODO: add syncthing style hash of cert, spiffee, DNS as alternative identities. TODO: add root CAs (including public) and SHA of root cert.

func (*UGate) OnHClose

func (gw *UGate) OnHClose(s string, id string, san string, r *http.Request, since time.Duration)

OnHClose called on http close

func (*UGate) OnMuxClose

func (gw *UGate) OnMuxClose(dm *MeshCluster)

func (*UGate) OnSClose

func (gw *UGate) OnSClose(s nio.Stream, addr net.Addr)

func (*UGate) OnStream

func (ug *UGate) OnStream(s nio.Stream)

All streams must call this method once a connection is made, and defer OnStreamDone

func (*UGate) OnStreamDone

func (ug *UGate) OnStreamDone(str nio.Stream)

Called at the end of the connection handling. After this point nothing should use or refer to the connection, both proxy directions should already be closed for write or fully closed.

func (*UGate) RegisterProxyStream

func (ug *UGate) RegisterProxyStream(s nio.Stream)

func (*UGate) RemoteID

func (gw *UGate) RemoteID(s nio.Stream) string

RemoteID returns the node WorkloadID based on authentication.

func (*UGate) Start

func (ug *UGate) Start() error

Start listening on all configured ports. This doesn't have to be called if ugate is used in client mode.

func (*UGate) StartListener

func (ug *UGate) StartListener(ll *meshauth.PortListener) error

StartListener and Start a real port listener on a port. Virtual listeners can be added to ug.Conf or the mux. Creates a raw (port) TCP listener. Accepts connections on a local port, forwards to a remote destination.

type UdpWriter

type UdpWriter interface {
	WriteTo(data []byte, dstAddr *net.UDPAddr, srcAddr *net.UDPAddr) (int, error)
}

UdpWriter is the interface implemented by the TunTransport, to send packets back to the virtual interface. TUN or TProxy raw support this. Required for 'transparent' capture of UDP - otherwise use STUN/TURN/etc. A UDP NAT does not need this interface.