turnstile

package module
v0.0.0-...-ee1aec3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 3, 2023 License: LGPL-3.0 Imports: 4 Imported by: 0

README

turnstile

pipeline status coverage report go report card Documentation

Turnstile is a very simple gateway / ingres component which provides the following functionality

  • transparent proxy for HTTP/HTTPS/REST microservices
  • transparent proxy for gRPC microservices including grpc-web
  • transparent proxy for TCP microservices
  • central authentication / authorisation handling for downstream HTTP, HTTPS and GPRC-WEB microservices
  • automatic generation of SSL certificates
  • automatic management of TLS (using let's encrypt)
  • allows custom routing to allow further customisation
  • turnstile supports consuming of the proxy protocol. It is then terminated and removed from connections to upstream consumers
  • supports compressing the content (credit to https://github.com/nytimes/gziphandler)

Getting started (using HTTP)

func main() {
  // Define basic config
  config := turnstile.Config{
    CompressResponse: true              // Gzip data coming out (http proxies)
    ListenerHTTP: ":8080",              // set http to listen on port 8080
    PublicURLs:   []string{"/healthz"}, // exclude healthz handler from authentication
    PublicDomains: []string{"public.domain.com", "*.public.com"},   // exclude public domain from authorisation
    Proxies: turnstile.ProxyList{
        &turnstile.HTTPProxy{
          Domain: "protected.domain.com", // requests to this domain will be routed
          Target: "https://httpbin.org",  // .. to this server
          Authoriser: nil,                      // you can provide custom authoriser
        },
        &turnstile.HTTPProxy{
          Domain: "public.domain.com",    // requests to this domain will be routed
          Target: "https://httpbin.org",  // .. to this server
		  Authoriser: nil                       // you can provide custom authoriser
        },
        &turnstile.HTTPProxy{
          Path:   "/http-bin",            // requests starting with this path will be routed
          Target: "https://httpbin.org",  // .. to this server
          StripPath: true,                // strip path before relaying downstream
          Authoriser: nil                       // you can provide custom authoriser
        },
        &turnstile.HTTPProxy{
          Domain: "public.domain.com",              // requests to this domain
          Path:   "/http-bin-status-500",           // .. and this path will be routed
          Target: "https://httpbin.org/status/500", // .. to this server, this path
          StripPath: true,                          // strip path before relaying downstream
          Authoriser: nil                           // you can provide custom authoriser
        },
    },
  }
  // Define some keys for authorisation
  apiKeyMap := map[string]string{
    "api-key-1": "client-id-1",
    "api-key-2": "client-id-2",
  }
  // Set up API key authoriser, expecting in requests to send key in my-api-key header
  // ... downstream microservices expecting to receive client id in out-client-id header
  auth := authoriser.NewAPIKeyAuthoriser("my-api-key", "out-client-id", apiKeyMap)
  // Initialise turnstile server
  srv := server.New(conf, nil)
  srv.AddAuthoriser(auth)
  // Start the server
  srv.Start()
  select {}
}

Using HTTPS

When it comes to HTTPS, turnstile offers multiple ways of how to obtain the certificate. Currently, turnstile supports following modes.

Auto-generated, self signed certificate

This is the simplest and fastest way to get started. Turnstile automatically generates both key and certificate which are then used to serve over HTTPS

func main() {
	config := turnstile.Config{
		ListenerHTTPS: ":8443",            // port where HTTPS will run
		PublicURLs:    []string{"/"},     // just for this example - disable authentication
	}
    // create, provide name of the organization 
    // ... and list of host names / ip addresses written to the certificate (can be empty)
	certProvider := certprovider.GenerateSelfSigned("myorganization", []{"my.organization.com"})
	srv := server.New(config, certProvider)
	run.IfErrorThenExit(srv.Start(), "error when starting turnstile")
}
Certificate and key loaded from files

If you need to provide certificate and key from a file, use this cert provider. Example:

func main() {
	config := turnstile.Config{
		ListenerHTTPS: ":8443",            // port where HTTPS will run
		PublicURLs:    []string{"/"},     // just for this example - disable authentication
	}
    certProvider := certprovider.FromFiles("cert.pem", "key.pem")
	srv := server.New(config, certProvider)
	run.IfErrorThenExit(srv.Start(), "error when starting turnstile")
}
Externally provided certificate and key as bytes

Use this for all other cases. The following example loads certificates from environment variables.

func main() {
	config := turnstile.Config{
		ListenerHTTPS: ":8443",            // port where HTTPS will run
		PublicURLs:    []string{"/"},     // just for this example - disable authentication
	}
    certProvider := certprovider.Provide([]byte(os.Getenv("CERT")), []byte(os.Getenv("KEY")))
	srv := server.New(config, certProvider)
	run.IfErrorThenExit(srv.Start(), "error when starting turnstile")
}
Let's encrypt, auto requested, auto renewed

The certificate and key is stored on local HD. If the certificate does not exist, turnstile automatically starts HTTP server (along with the HTTPS one) and reuqests let's encrypt for a certificate. Once it receives valid certificate, it is stored on local HD.

There are pre requisites needed to be able to run turnstile in this mode:

  1. the box must be accessible from the internet
  2. server must be listening on http(80) and https(443) ports
  3. both http and https must be accessible from outside (no firewall)
  4. the DNS you use must be pointing to that box - validation of the certificate is using web ACME challenge, so let's encrypt need to be able to reach your server

To run turnstile in this mode, you need to provide certManager object: golang.org/x/crypto/acme/autocert

func main() {
    config := turnstile.Config{
        ListenerHTTPS: ":443",
        PublicURLs:    []string{"/"},
    }
    certManager := &autocert.Manager{
        Prompt: autocert.AcceptTOS,
        Cache:  autocert.DirCache("/tmp/"),
    }
    certprovider.LetsEncryptCertProvider(certManager)
    srv := server.New(config, certprovider.LetsEncryptCertProvider(certManager))
    run.IfErrorThenExit(srv.Start(), "error when starting turnstile")
    select {}
}

Assuming that your domain is ts.mydomain.com, then once the server starts (you may need root permissions to expose things on port 80 and 443), navigate to the following URL:

https://ts.mydomain.com/healthz

If everything worked well, you should see Status 200 - healthy message after some time on page which is served over https. The certificates are stored in ./stored-certs/ or in a directory you specified in tls config.

Custom 401, 403, 404 handlers

If you want the server to return customised response (or redirect), turnstile exposes 401, 403 and 404 handlers for overriding. \

Example:
  // ...
  srv := server.New(conf, nil)
  // Custom 404 handler
  srv.Handle404(func(w http.ResponseWriter, req *http.Request) {
    w.WriteHeader(http.StatusNotFound)
    _, _ = w.Write([]byte("Custom 404 message"))
  })
  // Custom 401 handler
  srv.Handle401(func(w http.ResponseWriter, req *http.Request) {
    w.WriteHeader(http.StatusUnauthorised)
    _, _ = w.Write([]byte("Custom 401 message"))
  })
  // Custom 403 handler
  srv.Handle404(func(w http.ResponseWriter, req *http.Request) {
    w.WriteHeader(http.StatusForbidden)
    _, _ = w.Write([]byte("Custom 403 message"))
  })
  // ...

Custom authorisers

Aim is to leave bespoke authorisation mechanisms to the developer. To implement your bespoke logic, simply extend Authoriser interface.
Authoriser interface has only one method which is returning 2 parameters

  • authorised (bool): return true if the server should pass request downstream, return false if you want to return 403 - forbidden
  • error: if it returns error, then the server returns 401 - unauthorised (or runs srv.Handle401() if provided)
Example
type MyAuthoriser struct {}

func (a *MyAuthoriser) Authorise(r *http.Request) (authorised bool, resultErr error) {
    if r.Header.Get("my-header") == "" {
      return false, fmt.Errorf("my header is missing") // server returns 401 - unauthorised
    }
    if r.Header.Get("my-header") != "SecretPassword" {
      return false, nil // server returns 403 - forbidden
    }
    return true, nil // server passes request to the next handler and returns what that handler return
}

To use the authoriser, use the example above and just change authoriser

  ...
  auth := MyAuthoriser{}
  ...
Using multiple authorisers

If there is a situation you need to use multiple authorisers for the server or and individual end point, use authoriser.UseAny(auth1, auth2, ...). For example:

apiKeyAuth := authoriser.NewAPIKeyAuthoriser("my-api-key", "out-client-id", map[string]string{
    "api-key-1": "client-id-1",
    "api-key-2": "client-id-2",
})
myAuth := MyAuthoriser{}
auth := authoriser.UseAny(apiKeyAuth, myAuth)
...

Custom routes

Server provides AddRoutes() callback. If the route provided already exist in the server, the provided is used instead of existing. This is also true for proxied paths (i. e. if your router maps to matching proxied path, your router is used) AddRoutes callback provides router parameter, which is a subrouter created from the main router. This subrouter is mapped to root ("/") of the router.
Turnstile is using gorilla mux router under the hood - https://github.com/gorilla/mux

Example

(extending the example above)

...
srv := server.New(conf, nil)
srv.AddRoutes(func(router *mux.Router) {
  router.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("this is from the custom router"))
  })
  router.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("this is the custom http://my-server:8080/healthz handler"))
 })
 )
 ...

gRPC Proxy

If you need to proxy GRPC services, there is a gRPC proxy server built in. Underlying service utilizes https://github.com/mwitkow/grpc-proxy. To make things even simpler, the exposed grpc services can be wrapped so they work according to the grpc-web spec, so they can be called by javascript from the web browsers. The wrappers are utilizing https://github.com/improbable-eng/grpc-web/tree/master/go/grpcweb

If there is a certProvider loaded in turnstile, gRPC proxy will use the same certificates for securing gRPC proxies.

To proxy gRPC services, you need to provide gRPCProxyConfig object to the constructor function.

Example:

...
conf.GRPCConfig = &turnstile.GrpcConfig{
    Listener:   ":3009",                 // gRPC port which will the gRPC be listening on
    Targets: map[string]string{
        "exposed1.host.com": "internal-host-1:3009", // route requests for exposed1.host.com to internal-host-1:3009
        "exposed2.host.com": "internal-host-2:3009", // route requests for exposed2.host.com to internal-host-2:3009
    },
    EnableGrpcWeb: true, // whether to create grpc-web wrapper
    DisableGrpcListener: false, // if set to true, no standalone grpc is started. Still may be used with EnableGrpcWeb
}
srv := server.New(conf, nil)
...

TCP Proxy

If you need to proxy any other TCP service, there is a TCP proxy built in. To do this, simply add TCPProxy definition into the config.Proxies list

Example:

...
config.Proxies = turnstile.ProxyList{
  &turnstile.TCPProxy{Listener: ":3000", Target: "internal-host-1:1234"},
  &turnstile.TCPProxy{Listener: ":3010", Target: "internal-host-2:4567", ID: "proxy2"},
}
srv := server.New(conf, nil)
...

TCP proxies support authorisers as well. These are provided in form of a method as seen in the following example:

tcpProxy := &turnstile.TCPProxy{
  Listener: ":3000"
  Target: "internal-host-1:1234",
  Authoriser: func(conn net.Conn) {
    // ... do your logic, return true if should be authorised, false if rejected 	
  }
}

If you want to get proxy in the later stage, you must set ID property to it (as seen for proxy2). Then you can call the following to get it

if proxy2 := srv.GetTCPProxy("proxy2"); proxy2 != nil {
	// do something with proxy2
}

There are two prebuilt Conn authorisers. IPWhiteListAuthoriser and IPBlackListAuthoriser, both can consume IP addresses and CIDR blocks.

whiteListedProxy := &turnstile.TCPProxy{
  Listener: ":3000"
  Target: "internal-host-1:1234",
  Authoriser: authoriser.IPWhiteList("192.168.1.1", "10.0.0.0/16")
}
blackListedProxy := &turnstile.TCPProxy{
  Listener: ":3000"
  Target: "internal-host-1:1234",
  Authoriser: authoriser.IPBlackList("192.168.1.2", "88.1.1.1/24")
}

Shutter functionality

Sometimes, there is a need to 'shut' all communication and respond with a static message (when the site is under maintenance for example). This functionality can be achieved using shutter handler. Shutter is currently supported only for http trafic, both on the top turnstile level or in the individual HTTP proxies.

Turnstile level shutter handler example:
 // ...
srv := server.New(conf, nil)
srv.ShutterHandler(func(w http.ResponseWriter, req *http.Request) {
  _, _ = w.Write([]byte("Turnstile is under maintenance, please come back later"))
})

To remove top level shutter, call

srv.ShutterHandler(nil)
Proxy level shutter handler example:
proxyShutterHandler := func(w http.ResponseWriter, r *http.Request) {
  _, _ = w.Write([]byte("proxy destination is under maintenance, please come back later"))
}
conf := turnstile.Config{
	// ...
    &turnstile.HTTPProxy{
      ID: "proxy1",                   // this must be set if you want to dynamically set shutter page
      Domain: "public.domain.com",    // requests to this domain will be routed
      Target: "https://httpbin.org",  // .. to this server
      ShutterHandler: proxyShutterHandler,
    },
    // ....
},
srv := server.New(conf, nil)
// ...

To add or remove proxy level shutter dynamically, proxy must have ID set, do

// Add shutter
if proxy1 := srv.GetHTTPProxy("proxy1"); proxy1 != nil {
	proxy1.ShutterHandler = proxyShutterHandler
}
// Remove shutter
if proxy1 := srv.GetHTTPProxy("proxy1"); proxy1 != nil {
	proxy1.ShutterHandler = nil
}

Proxy Protocol

Proxy protocol is partially implemented thanks to go-proxyproto library.

  • Turnstile does consume proxy protocol from downstream load balancers.
  • Turnstile does not send proxy protocol information to upstream service consumers

CORS configuration

CORS handler is implemented thanks to rs/cors library.

Configuration can be provided in either top level turnstile config, or in the individual proxy configurations. If you want CORS to be delegated to underlying service, do not set CORS on neither of the turnstile config nor proxy config, otherwise you get CORSMultipleAllowOriginNotAllowed error in the browser.

Evaluation flow

The following stands for http and https proxies. GRPC and TCP proxies start their own listeners

  1. shutter
  2. authorisation
  3. grpc-web wrapper (if enabled for grpc services)
  4. custom routes
  5. proxies

TODO

  • provide parsers for environment variables (to be able to define path proxies in plain text - useful for docker, kubernetes)
  • implement set of common authorisers, API key, user / password from file, user / password from env. variables, user / password from db, etc.
  • implement session handling and session sharing across multiple instances
  • implement sso, oauth, etc.

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type CertProvider 

type CertProvider interface {
	Initialize() error
	GetCertificate(*tls.ClientHelloInfo) (*tls.Certificate, error)
}

CertProvider is an interface provided to the TLS config respondible for returning certificates

type Config 

type Config struct {
	//Proxies       []HTTPProxy // List of domain proxy definitions
	//TCPProxies    []TCPProxy  // list of ports to be forwarded to targets
	Proxies          ProxyList
	ListenerHTTPS    string        // HTTPS listener in unix socket format. Example = :8443
	ListenerHTTP     string        // HTTP listener in unix socket format. Example = :8080
	PublicURLs       []string      // List of URLs excluded from authorisation mechanism.
	PublicDomains    []string      // List of domains excluded from authorisation mechanism, accepts file-like matching (*, ?)
	CompressResponse bool          // Should all response be gzipped
	CORSOptions      *cors.Options // CORS configuration for turnstile
	GRPCConfig       *GrpcConfig   // GRPC configuration
}

Config is used by the server to set various internals

type GrpcConfig 

type GrpcConfig struct {
	CertProvider        CertProvider
	Listener            string
	Targets             map[string]string
	EnableGrpcWeb       bool // if set to true, grpc is wrapped to grpc-web. DisableGrpcListener can be true at the same time
	DisableGrpcListener bool // if set to true, standalone grpc server is disabled
}

GrpcConfig is a grpc server config struct

type HTTPProxy 

type HTTPProxy struct {
	ID               string                // ID can be set if you want to use turnstile.GetHTTPProxy functionality
	Domain           string                // Handler domain which should be pointing to downstream location. Example = /http-bin
	Path             string                // Handler path which should be pointing to downstream location. Example = /http-bin
	StripPath        bool                  // If set to true, the original path is stripped before sending to the target
	Target           string                // Downstream location target. Example = https://http-bin.org/get
	Authoriser       authoriser.Authoriser // List of authorizers for this specific transparent proxy
	CORSConfig       *cors.Options
	Transport        http.RoundTripper
	Handler403       http.HandlerFunc                                            // Possible to override 401 handler
	Handler401       http.HandlerFunc                                            // possible to override 403 handler
	ShutterHandler   http.HandlerFunc                                            // set this handler and every request will be responded by it
	InterceptHandler func(w http.ResponseWriter, r *http.Request) (claimed bool) // InterceptHandler is called before proxy handler (after shutter handler). Use this for handling login/logout, etc. Return claimed=true if you want to stop turnstile from further processing
	RequestModifiers []RequestModifier                                           // RequestModifiers are applied to the outgoing requests to the remote target
}

HTTPProxy is a VO for path proxy

type HTTPProxyList 

type HTTPProxyList []*HTTPProxy

HTTPProxyList represents list of http proxies

func (HTTPProxyList) Find 

func (l HTTPProxyList) Find(id string) (result *HTTPProxy)

Find goes through the list of proxies and finds one with the same id. If multiple, returns first ID of proxy must not be empty to be returned by this method

type Proxy 

type Proxy interface {
	// contains filtered or unexported methods
}

Proxy represents a proxy (either HTTPProxy or TCPProxy)

type ProxyList 

type ProxyList []Proxy

ProxyList represents list of proxies

func (ProxyList) HTTPProxies 

func (l ProxyList) HTTPProxies() (result HTTPProxyList)

HTTPProxies return list of http proxies

func (ProxyList) TCPProxies 

func (l ProxyList) TCPProxies() (result TCPProxyList)

TCPProxies returns list of TCP proxies

type RequestModifier 

type RequestModifier interface {
	Apply(req *http.Request)
}

RequestModifier is a generic request modifier

type TCPProxy 

type TCPProxy struct {
	ID         string                    // ID can be set if you want to use turnstile.GetTCPProxy functionality
	Listener   string                    // where to listen
	Target     string                    // where to point
	Authoriser authoriser.ConnAuthoriser // optional authoriser of the connection
}

TCPProxy hold information about the TCP proxy

type TCPProxyList 

type TCPProxyList []*TCPProxy

TCPProxyList represents list of TCP proxies

func (TCPProxyList) Find 

func (l TCPProxyList) Find(id string) (result *TCPProxy)

Find finds tcp proxy in the list

Source Files

View all Source files

Directories

Path Synopsis
cmd
turnstile
certprovider

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.