package module
Version: v0.0.0-...-0cad6a9 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Oct 24, 2022 License: Apache-2.0 Imports: 21 Imported by: 11


HTTP Based Overlay Network Environment

There are already many VPN, VPC and tunneling protocols in broad use, using UDP, custom TCP protocols, SSH or even HTTP and Websocket.

This package attempts to implement a minimal mesh overlay network based on mTLS and compatible with existing HTTP infrastructure. It will mirror Istio implementation and model - but with few extra features to work with non-Istio load balancers and infra.

The core protocol is very simple and can be implemented in any language and part of almost any framework - we hope that core gRPC libraries and other proxies will support it natively, and multiple implementations will be available.


  • mTLS based identity and end to end security between workloads
  • zero trust - all middle boxes will be mutually authenticated and have independent policies
  • support metadata - 'baggage' header, source/destination info
  • compatible with plain/standard gateways and load balancers with core HTTP/2 support

The repository contains a library that can be used in go application using the protocol natively, and a server that can be used as:

  • uProxy with TPROXY, 'whitebox', SOCKS capture
  • minimal Egress or East-West gateway
  • minimal Ingress gateway


Basic CONNECT mode

All TCP streams are sent as HTTP/2 CONNECT requests, with few optional extra headers.

Both ends are expected to use workload identity certificates (optional DNS), using mTLS to authenticate.

By default, only 'same trust domain and namespace' communication is allowed. The API/config allows custom policies.

When used as an Ingress - the gateway terminates TCP, HTTP, HTTPS, TLS connections, applies policies and forwards to workloads using CONNECT.

When used as Egress, the gateway terminates mTLS CONNECT, applies policies and forwards to the destination, optionally adding TLS.

It can also be used as a PEP or 'policy enforcing' East-West gateway.

In all 'middle box' cases, the gateway has access to the original plain text data and may apply policies or modify it.

mTLS over JWT-authenticated POST/CONNECT

For compatibility with existing infra, POST is also supported as equivalent to CONNECT.

This mode is enabled using 'tun' label on an endpoint, with value 'POST' or 'CONNECT'.

The original stream from the user will be end-to-end encrypted and authenticated using mTLS.

The proxy infrastructure is expected to handle the client-to-proxy authentication and forward the inner stream as a HTTP/2 connection. The ":authority" header will be set as expected to the hostname of the proxy.

A new 'X-tun' header will include the original address - sent as :authority in the basic CONNECT mode.



  • regular HTTP CONNECT proxy support

SNI routing

This is compatible with Istio East-West gateway, accepting without handshake the mTLS connections on 15443 and using the ClientHello info to find the ServerName (SNI). The old Istio clients should treat it as any regular Istio gateway.

The HBONE SNI gateway will forward the mTLS connection using mtls-over-H2 to an external address, including JWT authentication if needed.

H2R - Reverse connections support (remote accept)

Execution environment




[] Docker and helm [] Callbacks for events, [] hook to k8s slice [] Timeouts/deadlines/keepalives [] Move to separate git repo


[] convert sshd to use h2r (still over mTLS, but not exposed on the public address)


It is extremely common for applications to make REST or gRPC requests - this package is focused on implementing the mesh and HBONE protocols, and provides some minimal support for gRPC and K8S calls without extra dependencies and using the same code.

It is based on/inspired from kelseyhightower/konfig - which uses the 'raw' K8S protocol to avoid a big dependency. Google, K8S and many other APIs have a raw representation and don't require complex client libraries and depdencies. No code generation or protos are used - raw JSON in []byte is used, caller can handle marshalling.

For gRPC only the basic framing and protocol are support - exposed as frames containing []byte. Caller or generated code can handle marshalling.

This is intended for apps that make few small requests and want to keep size small, or want to take advantage of HBONE and mesh auto-setup. Also for low-level testing.

Other projects


  • IPFS / libP2P - one of the supported transports is a modified H2, also Quic. Reinvents cert format.

  • Syncthing - reverse tunnels, custom protocol, certs

  • Tor - of course.

  • BitTorrent

  • Konectivity Narrow use case of 'reverse connections' for Nodes (agents) getting calls from the APIserver via proxy, when the APIserver doesn't have direct connectivity to the node.

    gRPC or HTTP CONNECT based on the agent-proxy connection, and gRPC for APIserver to proxy.

    service AgentService {
      // Agent Identifier?
      rpc Connect(stream Packet) returns (stream Packet) {}
    service ProxyService {
      rpc Proxy(stream Packet) returns (stream Packet) {}
    Packet can be Data, Dial Req/Res, Close Req/Res


gost provides multiple integration points, focuses on similar TCP proxy modes.


# socks+http proxy
gost -L=:8080

gRPC framing support

HBone provides a low-level H2-based overlay network. The gRPC protocol defines a framing (1 byte TAG 4 byte LEN) and a set of headers. This library does not make a distinction between unary or stream - just like http library doesn't make distinction between HEAD, GET and POST, input/output are a stream of zero, 1 or more frames.

To integrate with XDS servers we need at least minimal protobuf support. The micro XDS implementation is based on HBone raw protocol, but we need to understand a minimal set of config options.

Taking a full dependency on gRPC is another option - it may allow using the optimized H2 stack by tunneling over gRPC, still H2 but with the extra 5-byte framing.

The minimal gRPC implementation supports the basic protocol - without generated stubs or any high level feature. It is intended for proxy and sniffing - for future support for GRPCRoute and content filtering. Both proto and byte frames are suported.


At low level, gRPC frames are parsed as Buffers, using the nio model to minimize copy. This is useful for proxy or filtering without unmarshal.

For most practical uses gRPC needs translation to protos - this is done automatically to avoid buffer allocations. "" library is used - the old one is deprecated ( but still a dependency since the new library is using some).




This section is empty.


View Source
var Debug = false

Debug for dev support, will log verbose info. Avoiding dependency on logging - eventually a trace interface will be provided so any logger can be used.

View Source
var ProxyCnt atomic.Int32


func LocalForwardPort

func LocalForwardPort(localAddr, dest string, hb *HBone) error

LocalForwardPort is a helper for port forwarding, similar with -L localPort:dest:destPort

func Proxy

func Proxy(nc net.Conn, in io.Reader, w io.Writer, dest string) error

Proxy forwards from nc to in/w. nc is typically the result of DialContext


type Auth

type Auth interface {
	GenerateTLSConfigServer() *tls.Config
	GenerateTLSConfigClient(name string) *tls.Config
	GenerateTLSConfigClientRoots(name string, pool *x509.CertPool) *tls.Config

Auth is the interface expected by hbone for mTLS support.

type Cluster

type Cluster struct {
	// Cluster WorkloadID - the cluster name in kube config, hub, gke - cluster name in XDS
	// Defaults to Base addr - but it is possible to have multiple clusters for
	// same address ( ex. different users / token providers).
	// Examples:
	ID string `json:"id,omitempty"`

	// ServiceAddr is the TCP address of the cluster - VIP:port or DNS:port
	// For K8S service it should be in the form serviceName.namespace.svc:svcPort
	// This can also be an 'original dst' address for single-endpoint clusters.
	// Can be a DNS name that resolves, or some other node.
	// Individual IPs (relay, etc) will be in the info.addrs field
	// May also be a URL (webpush endpoint).
	Addr string `json:"addr,omitempty"`

	// VIP for the cluster. In case of workload, the IP of the workload.
	// May be empty - Addr will be used.
	VIP string

	// SNI to use when making the request. Defaults to hostname in Addr
	SNI string

	// Set if the destination cluster is a HTTP service with a URL path.
	// This is only used when the cluster is used for HTTP, as a prefix.
	Path string

	// Active connections to endpoints, each is a multiplexed H2 connection.
	EndpointCon []*EndpointCon

	// Endpoint addresses associated with the cluster.
	// If empty, the Cluster Addr will be used directly.
	Endpoints []*Endpoint

	// If empty, the cluster is using system certs or SPIFFE
	// Otherwise, it's the configured root certs list, in PEM format.
	// May include multiple concatenated roots.
	CACert string

	// Primary public key of the node.
	// EC256: 65 bytes, uncompressed format
	// RSA: DER
	// ED25519: 32B
	// Used for sending encryted webpush message
	// If not known, will be populated after the connection.
	PublicKey []byte `json:"pub,omitempty"`

	// If set, a token source with this name is used.
	// If not found, no tokens will be added. If found, errors getting tokens will result
	// in errors connecting.
	TokenSource string

	// Optional TokenProvider - not needed if client wraps google oauth
	// or mTLS is used.
	TokenProvider func(context.Context, string) (string, error)

	// Static token to use. May be a long lived K8S service account secret or other long-lived creds.
	Token string

	// For GKE K8S clusters - extracted from ID.
	// This is the default location for the endpoints.
	Location string

	// timeout for new network connections to endpoints in cluster
	ConnectTimeout           time.Duration
	TCPKeepAlive             time.Duration
	TCPUserTimeout           time.Duration
	MaxRequestsPerConnection int

	// Default values for initial window size, initial window, max frame size
	InitialConnWindowSize int32
	InitialWindowSize     int32
	MaxFrameSize          uint32

	// Client configured with the root CA of the K8S cluster, used
	// for HTTP/1.1 requests. If set, the cluster is not HBone/H2 but a fallback
	// or non-mesh destination.
	// TODO: customize Dialer to still use mesh LB
	// TODO: attempt ws for tunneling H2
	Client *http.Client

	// TLS config used when dialing using workload identity, shared
	TLSClientConfig *tls.Config

	LastUsed time.Time
	Dynamic  bool

	// contains filtered or unexported fields

Cluster represents a set of endpoints, with a common configuration. Can be a K8S Service with VIP and DNS name, an external service, etc.

Similar with Envoy Cluster or K8S service, can also represent single endpoint with multiple paths/IPs.

func (*Cluster) AddToken

func (c *Cluster) AddToken(req *http.Request, aut string) error

func (*Cluster) Dial

func (c *Cluster) Dial(ctx context.Context, r *http.Request) (net.Conn, error)

Similar with HBone dial, if you already have the Cluster (by hostname).

func (*Cluster) DoRequest

func (c *Cluster) DoRequest(req *http.Request) ([]byte, error)

func (*Cluster) RoundTrip

func (c *Cluster) RoundTrip(req *http.Request) (*http.Response, error)

func (*Cluster) UpdateEndpoints

func (c *Cluster) UpdateEndpoints(ep []*Endpoint)

type Endpoint

type Endpoint struct {
	Labels map[string]string

	LBWeight int
	Priority int

	// Address is the PodIP:port. Can be a hostname:port for external endpoints.
	// Will be used when dialing direct.
	// If Dialing via a proxy (east-west, PEP, SNI, etc) - the proxy address will
	// be dialed, but the endpoint Address will be included as a header.
	Address string

	// HBoneAddress is hostOrIP:port for hbone. If not set, default port 15008 will be used.
	// The server is expected to support HTTP/2 and mTLS for CONNECT, or HTTP/2 and JWT for POST.
	// It is expected to have a spiffee identity, and request client certs in CONNECT case.
	HBoneAddress string

	// SNIGate is the endpoint address of a SNI gate. It can be a normal Istio SNI, a SNI to HBone or other protocols,
	// or a H2R gate.
	// If empty, the endpoint will use the URL and HBone protocol directly.
	// If set, the endpoint will use the normal in-cluster Istio protocol.
	SNIGate string

	// SNI name to use - defaults to service name
	SNI string

	Secure bool

Endpoint represents a connection/association with a cluster. May be a separate pod, or another IP for an existing pod.

type EndpointCon

type EndpointCon struct {
	Cluster  *Cluster
	Endpoint *Endpoint

	ConnectionStart time.Time
	SSLEnd          time.Time
	// contains filtered or unexported fields

EndpointCon is a multiplexed H2 client for a specific destination instance. Should not be used directly.

func (*EndpointCon) Close

func (hc *EndpointCon) Close() error

func (*EndpointCon) DialTLS

func (hc *EndpointCon) DialTLS(ctx context.Context, addr string) (net.Conn, error)

DialTLS creates an outer layer TLS connection with the H2 CONNECT (or POST) address. Will initiate a TCP connection first - possibly using the SNI gate, and do the handshake.

type HBone

type HBone struct {

	// Event handlers will be copied to all created Mux and streams
	// It is possible to add more to each individual mux/stream

	// AuthProviders - matching kubeconfig
	// It is expected to return tokens with the given audience - in case of GCP
	// returns access tokens. If not set the cluster can't be created.
	// A number of pre-defined token sources are used:
	// - gcp - returns GCP access tokens using MDS or default credentials. Used for example by GKE clusters.
	// - k8s - return K8S WorkloadID tokens with the given audience for default K8S cluster.
	// - istio-ca - returns K8S tokens with istio-ca audience - used by Citadel and default Istiod
	// - sts - federated google access tokens associated with GCP identity pools.
	AuthProviders map[string]func(context.Context, string) (string, error)

	Mux http.ServeMux

	// EndpointResolver hooks into the Dial process and return the configured
	// EndpointCon object. This integrates with the XDS/config plane, with
	// additional local configs.
	EndpointResolver func(sni string) *EndpointCon

	H2RConn     map[*h2.H2Transport]*EndpointCon
	H2RCallback func(string, *h2.H2Transport)

	Client *http.Client

	Http11Transport *http.Transport
	// contains filtered or unexported fields

HBone represents a node using a HTTP/2 or HTTP/3 based overlay network environment. This can act as a minimal REST client and server - or can be used as a RoundTripper, Dialer and Listener compatible with HBONE protocol and mesh security.

HBone by default uses mTLS, using spiffee identities encoding K8S namespace, KSA and a trust domain. Other forms of authentication can be supported - auth is handled via configurable interface, not part of the core package.

HBone can be used as a client, server or proxy/gateway.

func New

func New(auth Auth, ms *MeshSettings) *HBone

New creates a new HBone node. It requires a workload identity, including mTLS certificates.

func NewMesh

func NewMesh(ms *MeshSettings) *HBone

NewMesh creates the mesh object. It requires an auth source. Configuring the auth source should also initialize the identity and basic settings.

func (*HBone) AddService

func (hb *HBone) AddService(c *Cluster, service ...*Endpoint) *Cluster

AddService will add a cluster to be used for Dial and RoundTrip. The 'Addr' field can be a host:port or IP:port. If id is set, it can be host:port or hostname - will be added as a destination. The service can be IP:port or URLs

func (*HBone) Cluster

func (hb *HBone) Cluster(ctx context.Context, addr string) (*Cluster, error)

func (*HBone) Dial

func (hb *HBone) Dial(n, a string) (net.Conn, error)

func (*HBone) DialContext

func (hb *HBone) DialContext(ctx context.Context, network, addr string) (net.Conn, error)

DialContext dials a destination address (host:port) using TCP-over-H2 protocol. This can be used in applications as a TCP Dial replacement.

The result is an instance of nio.HTTPConn (diret) or a tls.Conn (for untrusted proxy mode). In the later case, NetConn() returns the nio.HTTPConn to the proxy.

The HTTPConn represents the HBone connection to the peer or to a proxy.

func (*HBone) DialRequest

func (hb *HBone) DialRequest(ctx context.Context, req *http.Request) (net.Conn, error)

DialRequest connects to the host defined in req.Host or req.URL.Host, creates a H2 mux and opens a stream, returing a net.Conn object

Dialing sends and headers in req. If req.Body is set, it will be forwarded to remote, as expected. If it is not set, net.Conn Write will be sent to remote, as in a typical dialed connection.

The resulting connection implements HBoneConn interface, which provides access to response headers. Response headers are not available immediately, unlike RoundTrip() this does not wait for response headers - just connect and starts the stream. Response headers can be waited, or will be available after first Read() in the conn.

func (*HBone) GetCluster

func (hb *HBone) GetCluster(addr string) *Cluster

func (*HBone) HandleAcceptedH2

func (hb *HBone) HandleAcceptedH2(conn net.Conn)

HandleAcceptedH2 implements server-side handling of the conn - including TLS handshake. conn may be a wrapped connection.

func (*HBone) HandleAcceptedH2C

func (hb *HBone) HandleAcceptedH2C(conn net.Conn)

HandleAcceptedH2C handles a plain text H2 connection, for example in case of secure networks.

func (*HBone) HandleTCPProxy

func (hb *HBone) HandleTCPProxy(w io.Writer, r io.Reader, hostPort string) error

HandleTCPProxy connects and forwards r/w to the hostPort

func (*HBone) HttpClient

func (hb *HBone) HttpClient(caCert []byte) *http.Client

HttpClient returns a http.Client configured with the specified root CA, and reasonable settings. The URest wrapper is added, for telemetry or other interceptors.

func (*HBone) SecureConn

func (hb *HBone) SecureConn(ep *Endpoint) bool

SecureConn return true if the connection the the specific endpoint is over a secure network and doesn't need encryption.

type HBoneAcceptedConn

type HBoneAcceptedConn struct {
	// contains filtered or unexported fields

HBoneAcceptedConn keeps track of one accepted H2 connection.

func (*HBoneAcceptedConn) ServeHTTP

func (hac *HBoneAcceptedConn) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP implements the basic TCP-over-H2 and H2 proxy protocol. Requests that are not HBone will be handled by the mux in HBone, and if they don't match a handler may be forwarded by the reverse HTTP proxy.

type Listener

type Listener struct {

	// Address address (ex :8080). This is the requested address.
	// BTS, SOCKS, HTTP_PROXY and IPTABLES have default ports and bindings, don't
	// need to be configured here.
	Address string `json:"address,omitempty"`

	// Port can have multiple protocols:
	// If missing or other value, this is a dedicated port, specific to a single
	// destination.
	Protocol string `json:"proto,omitempty"`

	// ForwardTo where to forward the proxied connections.
	// Used for accepting on a dedicated port. Will be set as Dest in
	// the stream, can be mesh node.
	// host:port format.
	ForwardTo string `json:"forwardTo,omitempty"`

	// Must block until the connection is fully handled !
	Handler nio.Handler `json:-`

	// ALPN to announce, for TLS listeners
	ALPN []string

	// Certificates to use.
	// Key is a domain, *.domain or *.
	Certs map[string]string

	NetListener net.Listener `json:-`
	PortHandler nio.Handler  `json:-`

Listener represents the configuration for a real port listener. uGate has a set of special listeners that multiplex requests: - socks5 dest - iptables original dst ( may be combined with DNS interception ) - NAT dst address - SNI for TLS - :host header for HTTP - ALPN - after TLS handshake

Multiplexed channels do an additional lookup to find the listener based on the channel address.

func (*Listener) Accept

func (l *Listener) Accept() (net.Conn, error)

func (*Listener) Addr

func (l *Listener) Addr() net.Addr

func (*Listener) Close

func (l *Listener) Close() error

type MeshSettings

type MeshSettings struct {

	// Hub or user project WorkloadID. If set, will be used to lookup clusters.
	//ProjectId      string
	Namespace      string
	ServiceAccount string

	ConnectTimeout time.Duration
	TCPUserTimeout time.Duration

	// Clients by name. The key is primarily a hostname:port, matching Istio/K8S Service name and ports.
	// TODO: do we need the port ? With ztunnel all endpoins can be reached, and the service selector applies
	// to all ports.
	Clusters map[string]*Cluster

	// Ports is the equivalent of container ports in k8s.
	// Name follows the same conventions as Istio and should match the port name in the Service.
	// Port "*" means 'any' port - if set, allows connections to any port by number.
	// Currently this is loaded from env variables named PORT_name=value, with the default PORT_http=8080
	// TODO: refine the 'wildcard' to indicate http1/2 protocol
	// TODO: this can be populated from a WorkloadGroup object, loaded from XDS or mesh env.
	Ports map[string]string

	// Timeout used for TLS handshakes. If not set, 3 seconds is used.
	HandsahakeTimeout time.Duration

	// Auth plugs-in mTLS support. The generated configs should perform basic mesh
	// authentication.
	Auth Auth

	Env map[string]string

	// Default to
	HBone string

	// Reverse tunnel to this address if set
	RemoteTunnel string

	// If set, hbonec is enabled on this address
	// TrustedIPRanges should be used instead.
	HBoneC string

	// Default to localhost:1080
	SocksAddr string

	// SNI port, default to 15003
	SNI string

	AdminPort string

	LocalForward map[int]string

	// ServiceCluster is mapped to Istio canonical service and envoy --serviceCluster
	// It will show up in x-envoy-downstream-service-cluster if user_agent is true
	ServiceCluster string

	// Secure is the list of secure networks (IPSec, wireguard, etc).
	// If both client and server are on a secure network, tls is not used.
	// WIP - for now any string will cause the cluster to use plaintext.
	SecureCIDR []string

	// ServiceNode is mapped to node name and envoy --service-node
	// It will show up in x-envoy-downstream-service-node
	ServiceNode string

MeshSettings has common settings for all clients

func (*MeshSettings) GetEnv

func (ms *MeshSettings) GetEnv(k, def string) string


Path Synopsis
ext module
gcp Module
grpc Module
otel Module
transport Module
uxds Module
Package transport defines and implements message oriented communication channel to complete various transactions (e.g., an RPC).
Package transport defines and implements message oriented communication channel to complete various transactions (e.g., an RPC).
Package http2 implements the HTTP/2 protocol.
Package http2 implements the HTTP/2 protocol.
Package hpack implements HPACK, a compression format for efficiently representing HTTP header fields in the context of HTTP/2.
Package hpack implements HPACK, a compression format for efficiently representing HTTP header fields in the context of HTTP/2.
hboned module
Package syscall provides functionalities that grpc uses to get low-level operating system stats/info.
Package syscall provides functionalities that grpc uses to get low-level operating system stats/info.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL