grace

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 2, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

README

grace

Go Reference

A Go library for starting and stopping applications gracefully.

Grace facilitates gracefully starting and stopping a Go web application. It helps with waiting for dependencies - such as sidecar upstreams - to be available and handling operating system signals to shut down.

Requires Go >= 1.21.

Usage

In your project directory:

go get github.com/morningconsult/grace

Features

  • Graceful handling of upstream dependencies that might not be available when your application starts
  • Graceful shutdown of multiple HTTP servers when operating system signals are received, allowing in-flight requests to finish.
  • Automatic startup and control of a dedicated health check HTTP server.
  • Passing of signal context to other non-HTTP components with a generic function signature.
Gracefully shutting down an application

Many HTTP applications need to handle graceful shutdowns so that in-flight requests are not terminated, leaving an unsatisfactory experience for the requester. Grace helps with this by catching operating system signals and allowing your HTTP servers to finish processing requests before being forcefully stopped.

To use this, add something similar to the following example to the end of your application's entrypoint. grace.Run should be returned in your entrypoint/main function.

An absolute minimal configuration to get a graceful server would be the following:

ctx := context.Background()
httpHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
  w.Write([]byte("hello there"))
})

// This is the absolute minimum configuration necessary to have a gracefully
// shutdown server.
g := grace.New(ctx, grace.WithServer("localhost:9090", httpHandler))
err := g.Run(ctx)

Additionally, it will also handle setting up a health check server with any check functions necessary. The health server will be shut down as soon as a signal is caught. This helps to ensure that the orchestration system running your application marks it as unhealthy and stops sending it any new requests, while the in-flight requests to your actual application are still allowed to finish gracefully.

An minimal example with a health check server and your application server would be similar to the following:

httpHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
  w.Write([]byte("hello there"))
})

dbPinger := grace.HealthCheckerFunc(func(ctx context.Context) error {
  // ping database
  return nil
})

g := grace.New(
  ctx,
  grace.WithHealthCheckServer("localhost:9092", grace.WithCheckers(dbPinger)),
  grace.WithServer("localhost:9090", httpHandler, grace.WithServerName("api")),
)

A full example with multiple servers, background jobs, and health checks:

// Set up database pools, other application things, server handlers,
// etc.
httpHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
  w.Write([]byte("hello there"))
})

metricsHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
  w.Write([]byte("here are the metrics"))
})

dbPinger := grace.HealthCheckerFunc(func(ctx context.Context) error {
  // ping database
  return nil
})

redisPinger := grace.HealthCheckerFunc(func(ctx context.Context) error {
  // ping redis.
  return nil
})

bgWorker := func(ctx context.Context) error {
  // Start some background work
  return nil
}

// Create the new grace instance with your addresses/handlers.
// Here, we create:
//
//  1. A health check server listening on 0.0.0.0:9092 that will
//     respond to requests at /-/live and /-/ready, running the dbPinger
//     and redisPinger functions for each request to /-/ready.
//     This overrides the default endpoints of /livez and /readyz.
//  2. Our application server on localhost:9090 with the httpHandler.
//     It specifies the default read and write timeouts, and a graceful
//     stop timeout of 10 seconds.
//  3. Our metrics server on localhost:9091, with a shorter stop timeout
//     of 5 seconds.
//  4. A function to start a background worker process that will be called
//     with the context to be notified from OS signals, allowing for background
//     processes to also get stopped when a signal is received.
//  5. A custom list of operating system signals to intercept that override the
//     defaults.
g := grace.New(
  ctx,
  grace.WithHealthCheckServer(
    "0.0.0.0:9092",
    grace.WithCheckers(dbPinger, redisPinger),
    grace.WithLivenessEndpoint("/-/live"),
    grace.WithReadinessEndpoint("/-/ready"),
  ),
  grace.WithServer(
    "localhost:9090",
    httpHandler,
    grace.WithServerName("api"),
    grace.WithServerReadTimeout(grace.DefaultReadTimeout),
    grace.WithServerStopTimeout(10*time.Second),
    grace.WithServerWriteTimeout(grace.DefaultWriteTimeout),
  ),
  grace.WithServer(
    "localhost:9091",
    metricsHandler,
    grace.WithServerName("metrics"),
    grace.WithServerStopTimeout(5*time.Second),
  ),
  grace.WithBackgroundJobs(bgWorker),
  grace.WithStopSignals(
    os.Interrupt,
    syscall.SIGHUP,
    syscall.SIGTERM,
  ),
)

if err = g.Run(ctx); err != nil {
  log.Fatal(err)
}
Waiting for dependencies

If your application has upstream dependencies, such as a sidecar that exposes a remote database, you can use grace to wait for them to be available before attempting a connection.

At the top of your application's entrypoint (before setting up database connections!) use the Wait method to wait for specific addresses to respond to TCP/HTTP pings before continuing with your application setup:

err := grace.Wait(
  ctx,
  10*time.Second,
  grace.WithWaitForTCP("localhost:6379"), // redis
  grace.WithWaitForTCP("localhost:5432"), // postgres
  grace.WithWaitForUnix("/tmp/something.sock"), // something on a unix socket
  grace.WithWaitForHTTP("http://localhost:9200"), // elasticsearch
  grace.WithWaitForHTTP("http://localhost:19000/ready"), // envoy sidecar
  grace.WithWaitForUnixHTTP("/tmp/envoy.sock", "/ready"), // HTTP over unix socket
)
if err != nil {
	log.Fatal(err)
}

Local Development

Testing
Linting

The project uses golangci-lint for linting. Run with

golangci-lint run

Configuration is found in:

  • ./.golangci.yaml - Linter configuration.
Unit Tests

Run unit tests with

go test ./...

Documentation

Overview

Example (Full) 
package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"syscall"
	"time"

	"github.com/morningconsult/grace"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
	defer cancel()

	// Get addresses of dependencies, such as redis, postgres, etc
	// from CLI flags or other configuration. Wait for them to be available
	// before proceeding with setting up database connections and such.
	err := grace.Wait(ctx, 10*time.Second, grace.WithWaitForTCP("example.com:80"))
	if err != nil {
		log.Fatal(err)
	}

	// Set up database pools, other application things, server handlers,
	// etc.
	// ....

	httpHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Write([]byte("hello there"))
	})

	metricsHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Write([]byte("here are the metrics"))
	})

	dbPinger := grace.HealthCheckerFunc(func(context.Context) error {
		// ping database
		return nil
	})

	redisPinger := grace.HealthCheckerFunc(func(context.Context) error {
		// ping redis.
		return nil
	})

	bgWorker := func(context.Context) error {
		// Start some background work
		return nil
	}

	otherBackgroundWorker := func(context.Context) error {
		// Start some more background work
		return nil
	}

	// Create the new grace instance with your addresses/handlers.
	g := grace.New(
		ctx,
		grace.WithHealthCheckServer(
			"localhost:9092",
			grace.WithCheckers(dbPinger, redisPinger),
			grace.WithLivenessEndpoint("/-/live"),
			grace.WithReadinessEndpoint("/-/ready"),
		),
		grace.WithServer(
			"localhost:9090",
			httpHandler,
			grace.WithServerName("api"),
			grace.WithServerReadTimeout(grace.DefaultReadTimeout),
			grace.WithServerStopTimeout(10*time.Second),
			grace.WithServerWriteTimeout(grace.DefaultWriteTimeout),
		),
		grace.WithServer(
			"localhost:9091",
			metricsHandler,
			grace.WithServerName("metrics"),
			grace.WithServerStopTimeout(5*time.Second),
		),
		grace.WithBackgroundJobs(
			bgWorker,
			otherBackgroundWorker,
		),
		grace.WithStopSignals(
			os.Interrupt,
			syscall.SIGHUP,
			syscall.SIGTERM,
		),
	)

	if err = g.Run(ctx); err != nil {
		log.Fatal(err)
	}

}

Output:
Example (Minimal) 
package main

import (
	"context"
	"log"
	"net/http"
	"time"

	"github.com/morningconsult/grace"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
	defer cancel()

	// Set up database pools, other application things, server handlers,
	// etc.
	// ....

	httpHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Write([]byte("hello there"))
	})

	// This is the absolute minimum configuration necessary to have a gracefully
	// shutdown server.
	g := grace.New(ctx, grace.WithServer("localhost:9090", httpHandler))
	if err := g.Run(ctx); err != nil {
		log.Fatal(err)
	}

}

Output:
Example (Minimal_with_healthcheck) 
package main

import (
	"context"
	"log"
	"net/http"
	"time"

	"github.com/morningconsult/grace"
)

func main() {
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
	defer cancel()

	// Set up database pools, other application things, server handlers,
	// etc.
	// ....

	httpHandler := http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
		w.Write([]byte("hello there"))
	})

	dbPinger := grace.HealthCheckerFunc(func(context.Context) error {
		// ping a database, etc.
		return nil
	})

	// This is the minimum configuration for a gracefully shutdown server
	// along with a health check server. This is most likely what you would
	// want to implement.
	g := grace.New(
		ctx,
		grace.WithHealthCheckServer(
			"localhost:9092",
			grace.WithCheckers(dbPinger),
		),
		grace.WithServer(
			"localhost:9090",
			httpHandler,
			grace.WithServerName("api"),
		),
	)

	if err := g.Run(ctx); err != nil {
		log.Fatal(err)
	}

}

Output:

Index

Examples

Constants

View Source 
const (
	// DefaultReadTimeout is the default timeout for reading http requests for
	// a server.
	DefaultReadTimeout = 1 * time.Minute

	// DefaultWriteTimeout is the default timeout for writing http responses for
	// a server.
	DefaultWriteTimeout = 5 * time.Minute

	// DefaultStopTimeout is the default timeout for stopping a server after a
	// signal is encountered.
	DefaultStopTimeout = 10 * time.Second

	// DefaultLivenessEndpoint is the default liveness endpoint for the health server.
	DefaultLivenessEndpoint = "/livez"

	// DefaultReadinessEndpoint is the default readiness endpoint for the health server.
	DefaultReadinessEndpoint = "/readyz"
)

Variables

This section is empty.

Functions

func Wait 

func Wait(ctx context.Context, timeout time.Duration, opts ...WaitOption) error

Wait waits for all the provided checker pings to be successful until the specified timeout is exceeded. It will block until all of the pings are successful and return nil, or return an error if any checker is failing by the time the timeout elapses.

Wait can be used to wait for dependent services like sidecar upstreams to be available before proceeding with other parts of an application startup.

Example 
package main

import (
	"context"
	"errors"
	"log"
	"net"
	"net/http"
	"time"

	"github.com/morningconsult/grace"
)

func main() {
	ctx := context.Background()

	es := &http.Server{
		Addr: "localhost:9200",
		Handler: http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
			w.WriteHeader(http.StatusOK)
		}),
	}
	defer es.Shutdown(ctx) //nolint:errcheck

	go func() {
		if err := es.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatal(err)
		}
	}()

	pg, err := net.Listen("tcp", "localhost:6379")
	if err != nil {
		log.Fatal(err)
	}
	defer pg.Close() //nolint:errcheck

	redis, err := net.Listen("tcp", "localhost:5432")
	if err != nil {
		log.Fatal(err)
	}
	defer redis.Close() //nolint:errcheck

	// Get addresses of dependencies, such as redis, postgres, etc
	// from CLI flags or other configuration. Wait for them to be available
	// before proceeding with setting up database connections and such.
	err = grace.Wait(
		ctx,
		50*time.Millisecond,
		grace.WithWaitForTCP("localhost:6379"),
		grace.WithWaitForTCP("localhost:5432"),
		grace.WithWaitForHTTP("http://localhost:9200"),
	)
	if err != nil {
		log.Fatal(err)
	}

}

Output:

Types

type BackgroundJobFunc 

type BackgroundJobFunc func(ctx context.Context) error

BackgroundJobFunc is a function to invoke with the context returned from signal.NotifyContext. This can be used to ensure that non-http servers in the application, such as background workers, can also be tied into the signal context.

The function will be called within a golang.org/x/sync/errgroup.Group and must be blocking.

type Grace 

type Grace struct {
	// contains filtered or unexported fields
}

Grace handles graceful shutdown of http servers.

Each of the servers specified will be started in order, with graceful handling of OS signals to allow any in-flight requests to complete before stopping them entirely.

Additionally, a health check server will be started to receive health requests from external orchestration systems to confirm the aliveness of the application if added using WithHealthCheckServer.

func New 

func New(ctx context.Context, options ...Option) Grace

New creates a new grace. Specify one or more Option to configure the new grace client.

The provided context will be used as the base context for all created servers.

New does not start listening for OS signals, it only creates the new grace that can be started by calling Grace.Run.

func (Grace) Run 

func (g Grace) Run(ctx context.Context) error

Run starts all of the registered servers and creates a new health check server, if configured with WithHealthCheckServer.

The created health check server will not be gracefully shutdown and will instead be stopped as soon as any stop signals are encountered or the context is finished. This is to ensure that any health checks to the application begin to fail immediately.

They will all be stopped gracefully when the configured stop signals are encountered or the provided context is finished.

type HealthChecker 

type HealthChecker interface {
	CheckHealth(ctx context.Context) error
}

HealthChecker is something that needs its health checked to be "ready".

type HealthCheckerFunc 

type HealthCheckerFunc func(context.Context) error

HealthCheckerFunc is a function that can be used as a HealthChecker.

func (HealthCheckerFunc) CheckHealth 

func (hcf HealthCheckerFunc) CheckHealth(ctx context.Context) error

CheckHealth checks the health of a resource using the HealthCheckerFunc.

type HealthOption 

type HealthOption func(cfg healthConfig) healthConfig

A HealthOption is is used to modify the grace health check server config.

func WithCheckers 

func WithCheckers(checkers ...HealthChecker) HealthOption

WithCheckers sets the HealthChecker functions to the health server will run.

func WithLivenessEndpoint 

func WithLivenessEndpoint(endpoint string) HealthOption

WithLivenessEndpoint sets the liveness endpoint for the health check server. If not used, it will default to DefaultLivenessEndpoint.

func WithReadinessEndpoint 

func WithReadinessEndpoint(endpoint string) HealthOption

WithReadinessEndpoint sets the liveness endpoint for the health check server. If not used, it will default to DefaultReadinessEndpoint.

type Option 

type Option func(cfg config) config

An Option is used to modify the grace config.

func WithBackgroundJobs 

func WithBackgroundJobs(jobs ...BackgroundJobFunc) Option

WithBackgroundJobs sets the BackgroundJobFunc functions that will be invoked when [Run] is called.

func WithHealthCheckServer 

func WithHealthCheckServer(addr string, opts ...HealthOption) Option

WithHealthCheckServer adds a health check server to be run on the provided address in the form "ip:port" or "host:port". The checkers are the health checking functions to run for each request to the health check server.

func WithLogger 

func WithLogger(logger *slog.Logger) Option

WithLogger configures the logger to use.

func WithServer 

func WithServer(addr string, handler http.Handler, options ...ServerOption) Option

WithServer adds a new server to be handled by grace with the provided address and http.Handler. The address of the server to listen on should be in the form 'ip:port' or 'host:port'.

The server's http.Server.BaseContext will be set to the context used when New is invoked.

func WithStopSignals 

func WithStopSignals(signals ...os.Signal) Option

WithStopSignals sets the stop signals to listen for.

StopSignals are the signals to listen for to gracefully stop servers when encountered. If not specified, it defaults to os.Interrupt, syscall.SIGHUP, and syscall.SIGTERM.

type ServerOption 

type ServerOption func(cfg serverConfig) serverConfig

A ServerOption is used to modify a server config.

func WithServerName 

func WithServerName(name string) ServerOption

WithServerName sets the name of the server, which is a helpful name for the server for logging purposes.

func WithServerReadTimeout 

func WithServerReadTimeout(timeout time.Duration) ServerOption

WithServerReadTimeout sets the read timeout for the server.

ReadTimeout is the http.Server.ReadTimeout for the server. If not used, the ReadTimeout defaults to DefaultReadTimeout.

func WithServerStopTimeout 

func WithServerStopTimeout(timeout time.Duration) ServerOption

WithServerStopTimeout sets the stop timeout for the server.

The StopTimeout is the amount of time to wait for the server to exit before forcing a shutdown. This determines the period that the "graceful" shutdown will last.

If not used, the StopTimeout defaults to DefaultStopTimeout. A timeout of 0 will result in the server being shut down immediately.

func WithServerWriteTimeout 

func WithServerWriteTimeout(timeout time.Duration) ServerOption

WithServerWriteTimeout sets the read timeout for the server.

WriteTimeout is the http.Server.WriteTimeout for the server. If not used, the WriteTimeout defaults to DefaultWriteTimeout.

type WaitOption 

type WaitOption func(cfg waitConfig) waitConfig

WaitOption is a configurable option for Wait.

func WithWaitForHTTP 

func WithWaitForHTTP(url string) WaitOption

WithWaitForHTTP makes a new HTTP waiter that will make GET requests to a URL until it returns a non-500 error code. All statuses below 500 mean the dependency is accepting requests, even if the check is unauthorized or invalid.

func WithWaitForTCP 

func WithWaitForTCP(addr string) WaitOption

WithWaitForTCP makes a new TCP waiter that will ping an address and return once it is reachable.

The addr can be a valid URL (as accepted by net/url) or a network address in "host:port" form. URLs that do not follow HTTP or HTTPS schemes must specify the port explicitly.

func WithWaitForUnix added in v0.2.0 

func WithWaitForUnix(socketPath string) WaitOption

WithWaitForUnix makes a new unix waiter that will ping a socket and return once it is reachable. The socketPath must be a valid filepath to the unix socket to connect with.

func WithWaitForUnixHTTP added in v0.2.0 

func WithWaitForUnixHTTP(socketPath, urlPath string) WaitOption

WithWaitForUnixHTTP makes a new HTTP waiter that will make GET requests to a unix domain socket and URL path until it returns a non-500 error code. All statuses below 500 mean the dependency is accepting requests, even if the check is unauthorized or invalid.

func WithWaitLogger 

func WithWaitLogger(logger *slog.Logger) WaitOption

WithWaitLogger configures the logger to use when calling Wait.

func WithWaiter 

func WithWaiter(w Waiter) WaitOption

WithWaiter adds a waiter for use with Wait.

func WithWaiterFunc 

func WithWaiterFunc(w WaiterFunc) WaitOption

WithWaiterFunc adds a waiter for use with Wait.

type Waiter 

type Waiter interface {
	Wait(ctx context.Context) error
}

Waiter is something that waits for a thing to be "ready".

type WaiterFunc 

type WaiterFunc func(context.Context) error

WaiterFunc is a function that can be used as a Waiter.

func (WaiterFunc) Wait 

func (w WaiterFunc) Wait(ctx context.Context) error

Wait waits for a resource using the WaiterFunc.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.