endless

README

endless

Zero downtime restarts for golang HTTP and HTTPS servers.

Inspiration & Credits

Well... it's what you want right - no need to hook in and out on a loadbalancer or something - just compile, SIGHUP, start new one, finish old requests etc.

There is https://github.com/rcrowley/goagain and i looked at https://fitstar.github.io/falcore/hot_restart.html which looked easier to do, but still some assembly required. I wanted something that's ideally as simple as

err := endless.ListenAndServe("localhost:4242", mux)

I found the excellent post Graceful Restart in Golang by Grisha Trubetskoy and took his code as a start. So a lot of credit to Grisha!

Features

• Drop-in replacement for http.ListenAndServe and http.ListenAndServeTLS
• Signal hooks to execute your own code before or after the listened to signals (SIGHUP, SIGUSR1, SIGUSR2, SIGINT, SIGTERM, SIGTSTP)
• You can start multiple servers from one binary and endless will take care of the different sockets/ports assignments when restarting

Default Timeouts & MaxHeaderBytes

There are three variables exported by the package that control the values set for DefaultReadTimeOut, DefaultWriteTimeOut, and MaxHeaderBytes on the inner http.Server:

DefaultReadTimeOut    time.Duration
DefaultWriteTimeOut   time.Duration
DefaultMaxHeaderBytes int

The endless default behaviour is to use the same defaults defined in net/http.

These have impact on endless by potentially not letting the parent process die until all connections are handled/finished.

Hammer Time

To deal with hanging requests on the parent after restarting endless will hammer the parent 60 seconds after receiving the shutdown signal from the forked child process. When hammered still running requests get terminated. This behaviour can be controlled by another exported variable:

DefaultHammerTime time.Duration

The default is 60 seconds. When set to -1 hammerTime() is not invoked automatically. You can then hammer the parent manually by sending SIGUSR2. This will only hammer the parent if it is already in shutdown mode. So unless the process had received a SIGTERM, SIGSTOP, or SIGINT (manually or by forking) before SIGUSR2 will be ignored.

If you had hanging requests and the server got hammered you will see a log message like this:

2015/04/04 13:04:10 [STOP - Hammer Time] Forcefully shutting down parent

Examples & Documentation

import "github.com/fvbock/endless"

and then replacing http.ListenAndServe with endless.ListenAndServe or http.ListenAndServeTLS with endless.ListenAndServeTLS

err := endless.ListenAndServe("localhost:4242", handler)

After starting your server you can make some changes, build, and send SIGHUP to the running process and it will finish handling any outstanding requests and serve all new incoming ones with the new binary.

More examples are in here

There is also GoDoc Documentation

Signals

The endless server will listen for the following signals: syscall.SIGHUP, syscall.SIGUSR1, syscall.SIGUSR2, syscall.SIGINT, syscall.SIGTERM, and syscall.SIGTSTP:

SIGHUP will trigger a fork/restart

syscall.SIGINT and syscall.SIGTERM will trigger a shutdown of the server (it will finish running requests)

SIGUSR2 will trigger hammerTime

SIGUSR1 and SIGTSTP are listened for but do not trigger anything in the endless server itself. (probably useless - might get rid of those two)

You can hook your own functions to be called pre or post signal handling - eg. pre fork or pre shutdown. More about that in the hook example.

Limitation: No changing of ports

Currently you cannot restart a server on a different port than the previous version was running on.

PID file

If you want to save actual pid file, you can change the BeforeBegin hook like this:

server := endless.NewServer("localhost:4242", handler)
server.BeforeBegin = func(add string) {
	log.Printf("Actual pid is %d", syscall.Getpid())
	// save it somehow
}
err := server.ListenAndServe()

TODOs

• tests
• documentation
• less ugly wrapping of the tls.listener

Documentation

Overview

endless provides a drop in replacement for the `net/http` stl functions `ListenAndServe` and `ListenAndServeTLS` to achieve zero downtime restarts and code upgrades.

The code is based on the excellent post [Graceful Restart in Golang](http://grisha.org/blog/2014/06/03/graceful-restart-in-golang/) by [Grisha Trubetskoy](https://github.com/grisha). I took his code as a start. So a lot of credit to Grisha!

The original endless has been modified to include ListenAndServeTLSWithSNI for name-resoved TLS. Also a number of defects have been fixed.

Index

• Constants
• Variables
• func ListenAndServe(addr string, handler http.Handler) error
• func ListenAndServeTLS(addr string, certFile string, keyFile string, handler http.Handler) error
• func ListenAndServeTLSWithSNI(addr string, handler http.Handler, tlsConfigs []TLSConfig) (err error)
• func NewServer(addr string, handler http.Handler) (srv *endlessServer)
• type TLSConfig

Constants

const (
	PRE_SIGNAL = iota
	POST_SIGNAL

	STATE_INIT
	STATE_RUNNING
	STATE_SHUTTING_DOWN
	STATE_TERMINATE
)

Variables

var (
	DefaultReadTimeOut    time.Duration
	DefaultWriteTimeOut   time.Duration
	DefaultMaxHeaderBytes int
	DefaultHammerTime     time.Duration
)

Functions

func ListenAndServe

func ListenAndServe(addr string, handler http.Handler) error

ListenAndServe listens on the TCP network address addr and then calls Serve with handler to handle requests on incoming connections. Handler is typically nil, in which case the DefaultServeMux is used.

func ListenAndServeTLS

func ListenAndServeTLS(addr string, certFile string, keyFile string, handler http.Handler) error

ListenAndServeTLS acts identically to ListenAndServe, except that it expects HTTPS connections. Additionally, files containing a certificate and matching private key for the server must be provided. If the certificate is signed by a certificate authority, the certFile should be the concatenation of the server's certificate followed by the CA's certificate.

func ListenAndServeTLSWithSNI

func ListenAndServeTLSWithSNI(addr string, handler http.Handler, tlsConfigs []TLSConfig) (err error)

PJS func (srv *endlessServer) ListenAndServeTLSWithSNI(tlsConfigs []TLSConfig) (err error) { func ListenAndServeTLSWithSNI(srv *http.Server, tlsConfigs []TLSConfig) (err error) {

func NewServer

func NewServer(addr string, handler http.Handler) (srv *endlessServer)

NewServer returns an intialized endlessServer Object. Calling Serve on it will actually "start" the server.

Types

type TLSConfig

type TLSConfig struct {
	Certificate              string   // file name of Cert Key
	Key                      string   // file name of Cert Key
	ProtocolMinVersion       uint16   // ?? - defaults? I think to 1.2 the highest level - so why set
	ProtocolMaxVersion       uint16   // ?? - defaults?
	Ciphers                  []uint16 // Flags of Ciphers - need special processing to read in
	PreferServerCipherSuites bool     // Flags of Ciphers - need special processing to read in
}