tableflip

package module
Version: v1.2.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 30, 2022 License: BSD-3-Clause Imports: 16 Imported by: 53

README

Graceful process restarts in Go

It is sometimes useful to update the running code and / or configuration of a network service, without disrupting existing connections. Usually, this is achieved by starting a new process, somehow transferring clients to it and then exiting the old process.

There are many ways to implement graceful upgrades. They vary wildly in the trade-offs they make, and how much control they afford the user. This library has the following goals:

  • No old code keeps running after a successful upgrade
  • The new process has a grace period for performing initialisation
  • Crashing during initialisation is OK
  • Only a single upgrade is ever run in parallel

tableflip works on Linux and macOS.

Using the library

upg, _ := tableflip.New(tableflip.Options{})
defer upg.Stop()

go func() {
	sig := make(chan os.Signal, 1)
	signal.Notify(sig, syscall.SIGHUP)
	for range sig {
		upg.Upgrade()
	}
}()

// Listen must be called before Ready
ln, _ := upg.Listen("tcp", "localhost:8080")
defer ln.Close()

go http.Serve(ln, nil)

if err := upg.Ready(); err != nil {
	panic(err)
}

<-upg.Exit()

Please see the more elaborate graceful shutdown with net/http example.

Integration with systemd

[Unit]
Description=Service using tableflip

[Service]
ExecStart=/path/to/binary -some-flag /path/to/pid-file
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/path/to/pid-file

See the documentation as well.

The logs of a process using tableflip may go missing due to a bug in journald, which has been fixed by systemd v244 release. If you are running an older version of systemd, you can work around this by logging directly to journald, for example by using go-systemd/journal and looking for the $JOURNAL_STREAM environment variable.

Documentation

Overview

Package tableflip implements zero downtime upgrades.

An upgrade spawns a new copy of argv[0] and passes file descriptors of used listening sockets to the new process. The old process exits once the new process signals readiness. Thus new code can use sockets allocated in the old process. This is similar to the approach used by nginx, but as a library.

At any point in time there are one or two processes, with at most one of them in non-ready state. A successful upgrade fully replaces all old configuration and code.

To use this library with systemd you need to use the PIDFile option in the service file.

[Unit]
Description=Service using tableflip

[Service]
ExecStart=/path/to/binary -some-flag /path/to/pid-file
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/path/to/pid-file

Then pass /path/to/pid-file to New. You can use systemd-run to test your implementation:

systemd-run --user -p PIDFile=/path/to/pid-file /path/to/binary

systemd-run will print a unit name, which you can use with systemctl to inspect the service.

NOTES:

Requires at least Go 1.9, since there is a race condition on the pipes used for communication between parent and child.

If you're seeing "can't start process: no such file or directory", you're probably using "go run main.go", for graceful reloads to work, you'll need use "go build main.go".

Tableflip does not work on Windows, because Windows does not have the mechanisms required to support this method of graceful restarting. It is still possible to include this package in code that runs on Windows, which may be necessary in certain development circumstances, but it will not provide zero downtime upgrades when running on Windows. See the `testing` package for an example of how to use it.

Example (HttpShutdown)

This shows how to use the upgrader with the graceful shutdown facilities of net/http.

package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/cloudflare/tableflip"
)

func main() {
	var (
		listenAddr = flag.String("listen", "localhost:8080", "`Address` to listen on")
		pidFile    = flag.String("pid-file", "", "`Path` to pid file")
	)

	flag.Parse()
	log.SetPrefix(fmt.Sprintf("%d ", os.Getpid()))

	upg, err := tableflip.New(tableflip.Options{
		PIDFile: *pidFile,
	})
	if err != nil {
		panic(err)
	}
	defer upg.Stop()

	// Do an upgrade on SIGHUP
	go func() {
		sig := make(chan os.Signal, 1)
		signal.Notify(sig, syscall.SIGHUP)
		for range sig {
			err := upg.Upgrade()
			if err != nil {
				log.Println("Upgrade failed:", err)
			}
		}
	}()

	// Listen must be called before Ready
	ln, err := upg.Listen("tcp", *listenAddr)
	if err != nil {
		log.Fatalln("Can't listen:", err)
	}

	server := http.Server{
		// Set timeouts, etc.
	}

	go func() {
		err := server.Serve(ln)
		if err != http.ErrServerClosed {
			log.Println("HTTP server:", err)
		}
	}()

	log.Printf("ready")
	if err := upg.Ready(); err != nil {
		panic(err)
	}
	<-upg.Exit()

	// Make sure to set a deadline on exiting the process
	// after upg.Exit() is closed. No new upgrades can be
	// performed if the parent doesn't exit.
	time.AfterFunc(30*time.Second, func() {
		log.Println("Graceful shutdown timed out")
		os.Exit(1)
	})

	// Wait for connections to drain.
	server.Shutdown(context.Background())
}
Output:

Example (TcpServer)

This shows how to use the Upgrader with a listener based service.

package main

import (
	"flag"
	"fmt"
	"log"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/cloudflare/tableflip"
)

func main() {
	var (
		listenAddr = flag.String("listen", "localhost:8080", "`Address` to listen on")
		pidFile    = flag.String("pid-file", "", "`Path` to pid file")
	)

	flag.Parse()
	log.SetPrefix(fmt.Sprintf("%d ", os.Getpid()))

	upg, err := tableflip.New(tableflip.Options{
		PIDFile: *pidFile,
	})
	if err != nil {
		panic(err)
	}
	defer upg.Stop()

	// Do an upgrade on SIGHUP
	go func() {
		sig := make(chan os.Signal, 1)
		signal.Notify(sig, syscall.SIGHUP)
		for range sig {
			err := upg.Upgrade()
			if err != nil {
				log.Println("upgrade failed:", err)
			}
		}
	}()

	ln, err := upg.Fds.Listen("tcp", *listenAddr)
	if err != nil {
		log.Fatalln("Can't listen:", err)
	}

	go func() {
		defer ln.Close()

		log.Printf("listening on %s", ln.Addr())

		for {
			c, err := ln.Accept()
			if err != nil {
				return
			}

			go func() {
				c.SetDeadline(time.Now().Add(time.Second))
				c.Write([]byte("It is a mistake to think you can solve any major problems just with potatoes.\n"))
				c.Close()
			}()
		}
	}()

	log.Printf("ready")
	if err := upg.Ready(); err != nil {
		panic(err)
	}
	<-upg.Exit()
}
Output:

Index

Examples

Constants

View Source
const DefaultUpgradeTimeout time.Duration = time.Minute

DefaultUpgradeTimeout is the duration before the Upgrader kills the new process if no readiness notification was received.

Variables

View Source
var ErrNotSupported = errors.New("tableflip: platform does not support graceful restart")

Functions

This section is empty.

Types

type Conn

type Conn interface {
	net.Conn
	syscall.Conn
}

Conn can be shared between processes.

type Fds

type Fds struct {
	// contains filtered or unexported fields
}

Fds holds all file descriptors inherited from the parent process.

func (*Fds) AddConn

func (f *Fds) AddConn(network, addr string, conn Conn) error

AddConn adds a connection.

It is safe to close conn after calling this method.

func (*Fds) AddFile

func (f *Fds) AddFile(name string, file *os.File) error

AddFile adds a file.

func (*Fds) AddListener

func (f *Fds) AddListener(network, addr string, ln Listener) error

AddListener adds a listener.

It is safe to close ln after calling the method. Any existing listener with the same address is overwitten.

func (*Fds) AddPacketConn added in v1.2.0

func (f *Fds) AddPacketConn(network, addr string, conn PacketConn) error

AddPacketConn adds a PacketConn.

It is safe to close conn after calling the method. Any existing packet connection with the same address is overwitten.

func (*Fds) Conn

func (f *Fds) Conn(network, addr string) (net.Conn, error)

Conn returns an inherited connection or nil.

It is safe to close the returned Conn.

func (*Fds) File

func (f *Fds) File(name string) (*os.File, error)

File returns an inherited file or nil.

The descriptor may be in blocking mode.

func (*Fds) Files added in v1.2.3

func (f *Fds) Files() ([]*os.File, error)

Files returns all inherited files and mark them as used.

The descriptors may be in blocking mode.

func (*Fds) Listen

func (f *Fds) Listen(network, addr string) (net.Listener, error)

Listen returns a listener inherited from the parent process, or creates a new one.

func (*Fds) ListenPacket added in v1.2.0

func (f *Fds) ListenPacket(network, addr string) (net.PacketConn, error)

ListenPacket returns a packet conn inherited from the parent process, or creates a new one.

func (*Fds) ListenPacketWithCallback added in v1.2.1

func (f *Fds) ListenPacketWithCallback(network, addr string, callback func(network, addr string) (net.PacketConn, error)) (net.PacketConn, error)

ListenPacketWithCallback returns a packet conn inherited from the parent process, or calls the supplied callback to create a new one.

This should be used in case some customization has to be applied to create the connection. Note that the callback must not use the underlying `Fds` object as it will be locked during the call.

func (*Fds) ListenWithCallback added in v1.2.1

func (f *Fds) ListenWithCallback(network, addr string, callback func(network, addr string) (net.Listener, error)) (net.Listener, error)

ListenWithCallback returns a listener inherited from the parent process, or calls the supplied callback to create a new one.

This should be used in case some customization has to be applied to create the connection. Note that the callback must not use the underlying `Fds` object as it will be locked during the call.

func (*Fds) Listener

func (f *Fds) Listener(network, addr string) (net.Listener, error)

Listener returns an inherited listener or nil.

It is safe to close the returned listener.

func (*Fds) PacketConn added in v1.2.0

func (f *Fds) PacketConn(network, addr string) (net.PacketConn, error)

PacketConn returns an inherited packet connection or nil.

It is safe to close the returned packet connection.

type Listener

type Listener interface {
	net.Listener
	syscall.Conn
}

Listener can be shared between processes.

type Options

type Options struct {
	// Time after which an upgrade is considered failed. Defaults to
	// DefaultUpgradeTimeout.
	UpgradeTimeout time.Duration
	// The PID of a ready process is written to this file.
	PIDFile string
	// ListenConfig is a custom ListenConfig. Defaults to an empty ListenConfig
	ListenConfig *net.ListenConfig
}

Options control the behaviour of the Upgrader.

type PacketConn added in v1.2.0

type PacketConn interface {
	net.PacketConn
	syscall.Conn
}

PacketConn can be shared between processes.

type Upgrader

type Upgrader struct {
	*Fds
	// contains filtered or unexported fields
}

Upgrader handles zero downtime upgrades and passing files between processes.

func New

func New(opts Options) (upg *Upgrader, err error)

New creates a new Upgrader. Files are passed from the parent and may be empty.

Only the first call to this function will succeed. May return ErrNotSupported.

func (*Upgrader) Exit

func (u *Upgrader) Exit() <-chan struct{}

Exit returns a channel which is closed when the process should exit.

func (*Upgrader) HasParent

func (u *Upgrader) HasParent() bool

HasParent checks if the current process is an upgrade or the first invocation.

func (*Upgrader) Ready

func (u *Upgrader) Ready() error

Ready signals that the current process is ready to accept connections. It must be called to finish the upgrade.

All fds which were inherited but not used are closed after the call to Ready.

func (*Upgrader) Stop

func (u *Upgrader) Stop()

Stop prevents any more upgrades from happening, and closes the exit channel.

If this function is called before a call to Upgrade() has succeeded, it is assumed that the process is being shut down completely. All Unix sockets known to Upgrader.Fds are then unlinked from the filesystem.

func (*Upgrader) Upgrade

func (u *Upgrader) Upgrade() error

Upgrade triggers an upgrade.

func (*Upgrader) WaitForParent

func (u *Upgrader) WaitForParent(ctx context.Context) error

WaitForParent blocks until the parent has exited.

Returns an error if the parent misbehaved during shutdown.

Directories

Path Synopsis
Package testing provides a stub implementation that can be used for simplified testing of applications that normally use tableflip.
Package testing provides a stub implementation that can be used for simplified testing of applications that normally use tableflip.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL