keyless-tls-terminator

README

Keyless TLS Terminator (KTT)

This is a package for TLS termination without keeping the certificate's private keys in the same process. It is inspired by Cloudflare's Keyless SSL offering.

Table of Contents

[[TOC]]

How does it work?

Funamentally, TLS provides two major guarantees: confidentiality and authentication. You know that the server you're talking to really is who they're claiming to be, and you know that anything between you and that server cannot read your messages. When most people think of TLS, they think of the little padlock on their search bar, or the https part of a URL. However, the distinction between the two facets of TLS is important in understanding how this works.

Confidentiality

Generally, when you assume that information is "confidential," you are assuming that only you and the intended recipients know what data is bring exchanged. This is achieved through Diffie-Hellman key exchange protocols. To avoid a lengthy explanation of the technique involved, it is a method by which two parties can mutually agree on a shared secret over an insecure channel.

This ensures that two systems, who have never communicated with each other before, and have no preexisting knowledge of each other, can form a secure link without the possibility of introspection.

Authentication

The "authentication" facet of TLS is where the certificates are involved. They are a pair of asymmetric keys, where everybody in the world can know the "public" side, but the "private" side must be kept secret. This way, you can perform a cryptographic proof, stating that you do, in fact, control the private side of the key pair. However, these keys are worthless without some form of attestation. Anybody can claim to be Google, but in order for your connection to be "authenticated," they have to prove that they really are Google, and not just someone claiming to be.

This is accomplished through a chain of trust. Essentially, there are private keys that are stored very securely, that are hard-coded on each person's computer. Those private keys can create a signature of somebody else's public keys, and write in that signature what this other person has ownership and control over. Then, a third person can be signed by that second person, and so on.

Why are these public keys hard-coded on my machine?

This is an incredibly complex topic to delve into for any length of time. However, the big reason is that of trust: it is assumed that these keys are controlled by corporations or people who know just how bad it would be if those private keys were stolen. In fact, it's a very similar reason to why the US government receives low interest rates - they know that if they default on their loans, the implications would be far worse than just the amount of money involved.

The separation

These two halves of TLS are very much not dependent on each other. It is possible to create a confidential connection without any authentication, and it is possible to authenticate without confidentiality. That separation is where this technology comes into play.

Proofs

The cryptographic proof mentioned above is essentially the same thing as is used in TLS certificates. To perform a TLS proof, you compute a signature on all the communication up until the current moment. This way, you can demonstrate that the ephemeral values were generated by someone who has control of the private key associated with the certificate.

Therefore, if you can extricate the calculation of the proof from the calculation of the ephemeral keys, you can host the private keys on a separate process, or even separate machine from the one that is actually terminating the TLS connection.

And that is how this technology works.

Security

Security is like a chain - only as strong as its weakest link. This application has been designed to be secure, but is not impenetrable. Here is a brief explanation of the security implications it was designed to defend against, and the ones it is not designed to defend against.

Design features

Reduced attack surface for long-lived keys

This application is designed to reduce the number of servers which are required to store secret values. Instead of having tens of thousands of machines, each of which stores a private key for the certificate you are providing, it can be reduced to just a few machines in each datacenter.

Same-process memory disclosure

As Heartbleed demonstrated, when an application or library has a security vunlerability, it is easy to accidentally divulge memory contents in the same process. However, it is significantly more difficult to disclose memory contents of a separate, unrelated process, without root privileges - this is all thanks to virtual addressing. Processes, generally, are not capable of accessing each other's memory - simply because it's not technically possible. If both processes have memory addresses ranging from 0x00000000 to 0xffffffff, there is no way for them to accidentally - or intentionally - access the other processes memory - there is no bug to exploit, there simply is no overlap.

Therefore, this application was designed such that in the event that a critical memory disclosure bug is discovered in either the application, the Golang crypto libraries, or Golang itself, the private keys could not be leaked. It is perfectly acceptable to run the key server on the same machine, as they do not share any memory, and communicate in the same fashion as if the two processes were run on servers on opposite sides of the globe.

Man-in-the-Middle

This application uses mTLS to authenticate in both directions. It means that a short-lived certificate can be stored in the client process that connects to the key server, which has no implications if it is leaked accidentally after the certificate expires.

Arbitrary proof requests

While a compromise of a server hosting this application is not advisable, in the event that an attacker is able to generate malicious proof requests and send them to the keyserver, these proof requests will not compromise the private keys of the long-lived certificates, and are fully tracked and logged. In addition, malicious proof requests can not be used to retroactively decrypt communications - this is a feature of forward secrecy.

Out of Scope

Physical access

This application does not actively encrypt its own memory, and does not defend against memory introspection, cold boot bugs, or other similar security issues - at least, not in any way outside of what the operating system's kernel provides. If someone has access to the machine, there are bound to be problems.

Loss of short-lived private keys to connect to the keyserver

In the event that a bug similar to Heartbleed is discovered, this application is not able to defend against the loss of its own short-lived private keys. If an outside attacker can gain these, and then initiate a connection to the keyserver, it is equivalent to the attacker being able to extract the private keys from a regular web host.

It is highly recommended to deploy additional mitigations, such as IP ACLs, or even better, not assigning any sort of public interface to the keyservers. Ensuring they are on their own network, with no internet access, is an even better option.

Compromise of client-keyserver CA

Because this application uses mTLS, any application that is capable of minting valid certificates can utilize those certificates to perform arbitrary proof operations. While arbitrary proofs will not leak the private key, and are logged and tracked, an attacker could actively hijack traffic, if they are able to direct users to their servers.

Deploying

There are two separate halves to this repository, which are broken down into different packages.

NoKeyServer

The NoKeyServer application is available in the prebuilt binary. It is capable of being configured using a TOML file, where it will serve on a configurable port (default of TCP 7753). To execute it, simply run ktt keyserver /path/to/config.toml. It will automatically open the required ports and be ready to serve requests very quickly.

While it is available in the binary, it is also completely possible to implement your own. The internals of the entire repository are designed to be extremely modular - making use of native Golang cryptographic libraries and functions, and not applying any patches to the underlying runtime. In fact, for the development of this package, not a single line of cryptographic code was written - everything uses native functionality available in Go.

More information is available in the reference, and in the internal documentation, and a set of information surrounding a sample deployment is available here.

Proxy

The fundamental idea of this package is that it is used by a TLS-terminating reverse proxy. To that extent, a sample proxy is included in the binary, or in the proxy package, with the SimpleProxy struct. This is provided to allow you to get up and running as quickly as possible. However, it is by no means required to be used - the proxy package is fundamentally an API that can be used by any client application. The only items it requires are the address of the NoKeyServer, a certificate and key pair to use to communicate with the NoKeyServer, the paths to CAs that can be used on the mTLS connection, and a valid Listener.

The proxy implements the net.Listener interface, where new encrypted connections can be obtained by calling the Accept function. It will prepare the connection, terminate TLS, perform the remote proof, and perform the necessary logging and setup to use the connection. Once the Accept function returns, the resulting net.Conn can be used in any way that one could in standard Go.

In fact, it is completely possible to have the Proxy class terminate the connection, and then connect to a TLS-based upstream, such as Nginx. The possibilities are endless.

SimpleProxy

The SimpleProxy struct is an implementation of a proxy that forwards traffic to an upstream. When it receives a connection, the Proxy struct it contains terminates the TLS connection, then the SimpleProxy implementation pipes the data to a new connection to an upstream.

Note: The piping process is not implemented at the kernel level - it is imply an io.Copy call between the two net.Conn structs. Therefore, there may be some latency penalty, however, attempts have been made to minimize it as much as possible.

HTTP/TLS Redirection

In addition to supporting TLS termination, there is a simple package made available that accepts unencrypted HTTP connections, and returns a 301 Permanent Redirect to the HTTPS version of the URL they requested. This has been made available to ease adoption for the purpose of implementing a full HTTP reverse proxy ontop of the Proxy struct already available.

TLSRedirector

In the httpredir package, there is an HTTPRedirector struct, and a TLSRedirector struct as well. The HTTPRedirector is made available for custom redirects, while the TLSRedirector simply returns a redirect where the http portion of the URL requested is replaced with https. Internally, the TLSRedirector extends the functionality of the HTTPRedirector, both using muxie for the purpose of path handling. However, the use of the httpredir package is not necessary - and can be handled by any other application, including Nginx.

Sample Applications

There is a sample proxy package available in a separate repo. It contains documented code implementing the proxy, explaining its functionality, and some simplistic features. It is not intended to be an exhaustive example. The code is available here.

Logging

KTT uses zerolog for logging. The primary logger used throughout the package, for both the proxy side and the NoKeyServer side, along with all internals, is available in the kttlog package, with the exported variable Logger. This can be modified, replaced, and configured to the heart's desire, as long as it implements the same functions as zerolog.

Enabling Syslog/File logging

Both networked- and localized-syslog are supported, along with logging to a file, configurable in the kttlog package. Helper functions have been created:

func EnableFileLog(path string) error
func EnableLocalSyslog() error
func EnableNetworkSyslog(network, addr string) error
func EnableConsoleLog()

Logging Quirks & Features

Note that console logging is enabled by default, with pretty-printing. In the event that this is not desirable (it is significantly slower than JSON-based logging), simply call kttlog.ResetLogger(). It will create a brand new zerolog.Logger struct, which will discard all logs by default. To add a custom logger to the primary Logger object, simply call:

func AddWriter(w io.Writer)

This function will send all log events to the io.Writer interface provided. Each log event will have a single call, and will be formatted according to the primary Logger formatter.

In addition, the entire logger can be replaced with the following function:

func SetKTTLogger(l zerolog.Logger)

The above is simply a helper method. It is perfectly safe to access the kttlog.Logger object directly - though no guarantees are made about thread safety between the helper functions and modifying the Logger object directly.

The RebuildLogger function

The RebuildLogger function is slightly dangerous. In the event that you are modifying the Logger struct directly, and then one of the helper functions present in kttlog are called (they call RebuildLogger at the end of each function), it will overwrite any changes you have made to the Logger object. This is because it creates a new zerolog.Logger struct, using all of the writers that have been provided through the AddWriter function, and then attaching hooks provided through the AddHook function. In the event that you are accessing the Logger struct directly, it is recommend to not use this function, or any of the helper functions.

The ResetLogger function

The ResetLogger function will wipe all local hooks, all local writers, and creates a new zerolog.Logger from scratch with no output by default (console logging is also removed when this function is called). In the event that you are using the helper functions, it is recommended to only call this when it is necessary to wipe out the existing configuration. If you are accessing the Logger struct directly, it is recommended to never call this function, unless you intend to return the Logger to a clean state, and wipe all locally stored configuration (writers, hooks).

Setting Log Level

It is recommended to leave the settings at their defaults. However, two methods to specify log levels (for debugging, or for reducing message count) are provided.

Environment Variables

If you are using the ktt binary, it supports the DEBUG environment variable. It will set the log level to the value specified, or print an error if the value is unrecognized.

Log Levels (from most verbose to least)

1. trace
2. debug
3. info
4. warn
5. error
6. fatal
7. panic

The values are case-insensitive, as they are forced to lowercase internally. However, they are written in full lowercase above to keep in line with how they are interpreted.

Using the kttlog package

The kttlog package can have its logger's log level adjusted by calling the following function:

func SetLevel(level zerolog.Level)

Using a custom log hook

If it is desired to send logs to places other than what is made available by default or through configuration, a custom hook can be provided. You can add them using the AddHook function:

func AddHook(hook zerolog.Hook)

This will add the specified hook to the Logger struct stored in the kttlog package.

Reference

Terminology

If I'm having trouble understanding what the different terms mean, and I wrote this, then you probably will too.

Server

The server is key holder server, where the cryptographic operations take place.

Client

The client is the connection from a reverse proxy to a key holder server.

NoKeyServr

This is the key holder server, whose source code is in this repository. It will open TCP port 7753 (by default) to listen for connections from NoKeyClients, to answer signing requests.

NoKeyClient

This is the struct to connect to key holder servers. This would be used by a reverse proxy to request signing operations from a remote system.

License

This repository is available under the MIT license, available here.

Seurity Issues

In the event that a security issue is discovered that should not be publicly disclosed, please contact the owner of this repository directly, and do not create a publicly visible issue.