health

README

Health

A simple and flexible health check library for Go.

Documentation · Report Bug · Request Feature

Features

This library allows you to build health checks that do not simply return HTTP status code 200 but actually check if all necessary components are healthy.

This library provides the following features:

• Request based ("synchronous") and fixed-schedule ("asynchronous") health checks.
• Timeout management.
• Health status change listeners.
• Flexible lifecycle hooks
• Caching
• Failure tolerance based on fail count and/or time thresholds.
• Provides an http.Handler and http.HandlerFunc that are fully compatible with net/http.

This example showcases all features of this library.

Getting Started

package main

import (
	"context"
	"database/sql"
	"fmt"
	"github.com/alexliesenfeld/health"
	_ "github.com/mattn/go-sqlite3"
	"net/http"
	"time"
)

func main() {
	db, _ := sql.Open("sqlite3", "simple.sqlite")
	defer db.Close()

	// Create a new Checker
	checker := health.NewChecker(

		// Configure a global timeout that will be applied to all checks.
		health.WithTimeout(10*time.Second),

		// A simple check to see if database connection is up.
		health.WithCheck(health.Check{
			Name:    "database",
			Timeout: 2 * time.Second, // A check specific timeout.
			Check:   db.PingContext,
		}),

		// The following check will be executed periodically every 30 seconds 
		// started without an initial delay.
		health.WithPeriodicCheck(30*time.Second, 0, health.Check{
			Name: "search",
			Check: func(ctx context.Context) error {
				return fmt.Errorf("this makes the check fail")
			},
		}),
	)

	// We Create a new http.Handler that provides health check information
	// serialized as a JSON string via HTTP.
	http.Handle("/health", health.NewHandler(checker))
	http.ListenAndServe(":3000", nil)
}

Because our search component is down, the request curl -u username:password http://localhost:3000/health would yield a response with HTTP status code 503 (Service Unavailable), and the following JSON response body:

{
  "status": "down",
  "details": {
    "database": {
      "status": "up",
      "timestamp": "2021-07-01T08:05:14.603364Z"
    },
    "search": {
      "status": "down",
      "timestamp": "2021-07-01T08:05:08.522685Z",
      "error": "this makes the check fail"
    }
  }
}

Periodic Checks

With "synchronous" health checks we mean that every HTTP request initiates a health check and waits until all check functions complete before returning an aggregated health status. This approach is usually OK for smaller applications with a low number of quickly checkable dependencies. However, it will not scale well enough for more involved applications that either have many dependencies and/or some relatively slow check functions.

Rather than executing health check functions on every HTTP request, periodic (or "asynchronous") health checks execute the check function on a fixed schedule. With this approach, the health status is always read from a local cache that is regularly updated in the background. This allows responding to HTTP requests instantly without waiting for the check function to complete.

Periodic checks can be configured using the WithPeriodicCheck configuration option (see example above).

You can mix synchronous and asynchronous checks in your application.

Caching

Health check responses are cached to avoid sending too many request to the services that your program checks and to mitigate "denial of service" attacks. The TTL is set to 1 second by default. If you do not want to use caching altogether, you can disable it using the health.WithDisabledCache() configuration option.

Failure Tolerance

This library lets you configure failure tolerant checks that allow some degree of failure. The check is only considered failed, when predefined tolerance thresholds are crossed.

Example

Let's assume that your app provides a REST API but also consumes messages from a Kafka topic. If the connection to Kafka is down, your app can still serve API requests, but it will not process any messages during this time. If the Kafka health check is configured without any failure tolerance, your whole application will become unhealthy. This is most likely not what you want. However, if Kafka is down for too long, there may indeed be a problem that requires attention. In this case, you still may want to flag your app unhealthy by returning a failing health check, so that it can be automatically restarted by your infrastructure.

Failure tolerant health checks let you configure this kind of behaviour.

health.WithCheck(health.Check{
    Name:    "unreliable-service",
    // Check is allowed to fail up to 4 times until considered unavailable
    MaxContiguousFails: 4,
    // Check is allowed to be in an erroneous state for up to 1 minute until considered unavailable.
    MaxTimeInError:      1 * time.Minute,
    Check: myCheckFunc,
}),

Hooks

Listening to Status Changes

It can be useful to react to health status changes. For example, you might want to log status changes, so you can easier correlate logs during root cause analysis or perform actions to mitigate the impact of an unhealthy component.

This library allows you to configure listener functions that will be called when either the overall/aggregated health status changes, or that of a specific component.

Example

health.WithPeriodicCheck(5*time.Second, health.Check{
    Name:   "search",
    Check:  myCheckFunc,
    StatusListener: func(ctx context.Context, name string, state CheckState) ) {
        log.Printf("status of component '%s' changed to %s", name, state.Status)
    },
}),

health.WithStatusListener(func(ctx context.Context, state CheckerState)) {
    log.Printf("overall system health status changed to %s", state.Status)
}),

Middleware and Interceptors

It can be useful to hook into the checking lifecycle to do some processing before and after a health check. For example, you might want to add some tracing information to the Context before the check function executes, do some logging or modify the check result before sending the HTTP response (e.g., removing details on failed authentication).

This library provides two mechanisms that allow you to hook into processing:

• Interceptors make it possible to intercept all calls to a check function. This is useful if you have cross-functional code that needs to be reusable and should have access to check state information.
• Middleware gives you the possibility to intercept all calls of Checker.Check, which corresponds to every incoming HTTP request. In contrary to the usually used middleware pattern, this middleware allows you to access check related information and post-process a check result before sending it in an HTTP response.

License

health is free software: you can redistribute it and/or modify it under the terms of the MIT Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the MIT Public License for more details.

Documentation

Index

• func NewHandler(checker Checker, options ...HandlerOption) http.HandlerFunc
• type AvailabilityStatus
• type Check
• type CheckResult
• type CheckState
• type Checker
  • func NewChecker(options ...CheckerOption) Checker
• type CheckerOption
  • func WithCacheDuration(duration time.Duration) CheckerOption
  • func WithCheck(check Check) CheckerOption
  • func WithDisabledAutostart() CheckerOption
  • func WithDisabledCache() CheckerOption
  • func WithDisabledDetails() CheckerOption
  • func WithMaxErrorMessageLength(length uint) CheckerOption
  • func WithPeriodicCheck(refreshPeriod time.Duration, initialDelay time.Duration, check Check) CheckerOption
  • func WithStatusListener(listener func(ctx context.Context, state CheckerState)) CheckerOption
  • func WithTimeout(timeout time.Duration) CheckerOption
• type CheckerResult
• type CheckerState
• type HandlerOption
  • func WithMiddleware(middleware ...Middleware) HandlerOption
  • func WithResultWriter(writer ResultWriter) HandlerOption
  • func WithStatusCodeDown(httpStatus int) HandlerOption
  • func WithStatusCodeUp(httpStatus int) HandlerOption
• type Interceptor
• type InterceptorFunc
• type JSONResultWriter
  • func NewJSONResultWriter() *JSONResultWriter
  • func (r *JSONResultWriter) Write(result *CheckerResult, w http.ResponseWriter, req *http.Request) error
• type Middleware
• type MiddlewareFunc
• type ResultWriter

Functions

func NewHandler

func NewHandler(checker Checker, options ...HandlerOption) http.HandlerFunc

NewHandler creates a new health check http.Handler. If the Checker was not yet started (see Checker.IsStarted), it will be started automatically (see Checker.Start). You can disable this autostart by adding the WithDisabledAutostart configuration option.

Types

type AvailabilityStatus

type AvailabilityStatus string

AvailabilityStatus expresses the availability of either a component or the whole system.

const (
	// StatusUnknown holds the information that the availability
	// status is not known, because not all checks were executed yet.
	StatusUnknown AvailabilityStatus = "unknown"
	// StatusUp holds the information that the system or a component
	// is up and running.
	StatusUp AvailabilityStatus = "up"
	// StatusDown holds the information that the system or a component
	// down and not available.
	StatusDown AvailabilityStatus = "down"
)

type Check

type Check struct {
	// The Name must be unique among all checks. Name is a required attribute.
	Name string // Required

	// Check is the check function that will be executed to check availability.
	// This function must return an error if the checked service is considered
	// not available. Check is a required attribute.
	Check func(ctx context.Context) error // Required

	// Timeout will override the global timeout value, if it is smaller than
	// the global timeout (see WithTimeout).
	Timeout time.Duration // Optional

	// MaxTimeInError will set a duration for how long a service must be
	// in an error state until it is considered down/unavailable.
	MaxTimeInError time.Duration // Optional

	// MaxContiguousFails will set a maximum number of contiguous
	// check fails until the service is considered down/unavailable.
	MaxContiguousFails uint // Optional

	// StatusListener allows to set a listener that will be called
	// whenever the AvailabilityStatus (e.g. from "up" to "down").
	StatusListener func(ctx context.Context, name string, state CheckState) // Optional

	// Interceptors holds a list of Interceptor instances that will be executed one after another in the
	// order as they appear in the list.
	Interceptors []Interceptor
	// contains filtered or unexported fields
}

Check allows to configure health checks.

type CheckResult

type CheckResult struct {
	// Status is the availability status of a component.
	Status AvailabilityStatus `json:"status"`
	// Timestamp holds the time when the check was executed.
	Timestamp *time.Time `json:"timestamp,omitempty"`
	// Error contains the check error message, if the check failed.
	Error *string `json:"error,omitempty"`
}

CheckResult holds a components health information.

type CheckState

type CheckState struct {
	// LastCheckedAt holds the time of when the check was last executed.
	LastCheckedAt *time.Time
	// LastCheckedAt holds the last time of when the check did not return an error.
	LastSuccessAt *time.Time
	// LastFailureAt holds the last time of when the check did return an error.
	LastFailureAt *time.Time
	// FirstCheckStartedAt holds the time of when the first check was started.
	FirstCheckStartedAt time.Time
	// ContiguousFails holds the number of how often the check failed in a row.
	ContiguousFails uint
	// Result holds the error of the last check (nil if successful).
	Result error
	// The current availability status of the check.
	Status AvailabilityStatus
}

CheckState represents the current state of a component check.

type Checker

type Checker interface {
	// Start will start all necessary background workers and prepare
	// the checker for further usage.
	Start()
	// Stop stops will stop the checker.
	Stop()
	// Check runs all synchronous (i.e., non-periodic) check functions.
	// It returns the aggregated health status (combined from the results
	// of this executions synchronous checks and the previously reported
	// results of asynchronous/periodic checks. This function expects a
	// context, that may contain deadlines to which will be adhered to.
	// The context will be passed to all downstream calls
	// (such as listeners, component check functions, and interceptors).
	Check(ctx context.Context) CheckerResult
	// GetRunningPeriodicCheckCount returns the number of currently
	// running periodic checks.
	GetRunningPeriodicCheckCount() int
	// IsStarted returns true, if the Checker was started (see Checker.Start)
	// and is currently still running. Returns false otherwise.
	IsStarted() bool
}

Checker is the main checker interface. It provides all health checking logic.

func NewChecker

func NewChecker(options ...CheckerOption) Checker

NewChecker creates a new Checker. The provided options will be used to modify its configuration.

type CheckerOption

type CheckerOption func(config *checkerConfig)

CheckerOption is a configuration option for a Checker.

func WithCacheDuration

func WithCacheDuration(duration time.Duration) CheckerOption

WithCacheDuration sets the duration for how long the aggregated health check result will be cached. By default, the cache TTL (i.e, the duration for how long responses will be cached) is set to 1 second. Caching will prevent that each incoming HTTP request triggers a new health check. A duration of 0 will effectively disable the cache and has the same effect as WithDisabledCache.

func WithCheck

func WithCheck(check Check) CheckerOption

WithCheck adds a new health check that contributes to the overall service availability status. This check will be triggered each time Checker.Check is called (i.e., for each HTTP request). If health checks are expensive or you expect a bigger amount of requests on your the health endpoint, consider using WithPeriodicCheck instead.

func WithDisabledAutostart

func WithDisabledAutostart() CheckerOption

func WithDisabledCache

func WithDisabledCache() CheckerOption

WithDisabledCache disabled the check cache. This is not recommended in most cases. This will effectively lead to a health endpoint that initiates a new health check for each incoming HTTP request. This may have an impact on the systems that are being checked (especially if health checks are expensive). Caching also mitigates "denial of service" attacks. Caching is enabled by default.

func WithDisabledDetails

func WithDisabledDetails() CheckerOption

WithDisabledDetails disables all data in the JSON response body. The AvailabilityStatus will be the only content. Example: { "status":"down" }. Enabled by default.

func WithMaxErrorMessageLength

func WithMaxErrorMessageLength(length uint) CheckerOption

WithMaxErrorMessageLength limits maximum number of characters in error messages. Default is 500.

func WithPeriodicCheck

func WithPeriodicCheck(refreshPeriod time.Duration, initialDelay time.Duration, check Check) CheckerOption

WithPeriodicCheck adds a new health check that contributes to the overall service availability status. The health check will be performed on a fixed schedule and will not be executed for each HTTP request (as in contrast to WithCheck). This allows to process a much higher number of HTTP requests without actually calling the checked services too often or to execute long running checks. This way Checker.Check (and the health endpoint) always returns the last result of the periodic check.

func WithStatusListener

func WithStatusListener(listener func(ctx context.Context, state CheckerState)) CheckerOption

WithStatusListener registers a listener function that will be called whenever the overall/aggregated system health status changes (e.g. from "up" to "down"). Attention: There is no restriction on check type. This listener will be called for all check types (periodic and non-periodic checks)! This is apposed to the other two global listeners as defined in WithBeforeCheckListener and WithAfterCheckListener.

func WithTimeout

func WithTimeout(timeout time.Duration) CheckerOption

WithTimeout defines a timeout duration for all checks. You can override this timeout by using the timeout value in the Check configuration. Default value is 30 seconds.

type CheckerResult

type CheckerResult struct {
	// Status is the aggregated system availability status.
	Status AvailabilityStatus `json:"status"`
	// Details contains health information for all checked components.
	Details *map[string]CheckResult `json:"details,omitempty"`
}

CheckerResult holds the aggregated system availability status and detailed information about the individual checks.

type CheckerState

type CheckerState struct {
	// Status is the aggregated system health status.
	Status AvailabilityStatus
	// CheckState contains the state of all checks.
	CheckState map[string]CheckState
}

CheckerState represents the current state of the Checker.

type HandlerOption

type HandlerOption func(*handlerConfig)

HandlerOption is a configuration option for a Handler (see NewHandler).

func WithMiddleware

func WithMiddleware(middleware ...Middleware) HandlerOption

func WithResultWriter

func WithResultWriter(writer ResultWriter) HandlerOption

func WithStatusCodeDown

func WithStatusCodeDown(httpStatus int) HandlerOption

func WithStatusCodeUp

func WithStatusCodeUp(httpStatus int) HandlerOption

type Interceptor

type Interceptor func(next InterceptorFunc) InterceptorFunc

Interceptor is factory function that allows creating new instances of a InterceptorFunc. The concept behind Interceptor is similar to the middleware pattern. A InterceptorFunc that is created by calling a Interceptor is expected to forward the function call to the next InterceptorFunc (passed to the Interceptor in parameter 'next'). This way, a chain of interceptors is constructed that will eventually invoke of the components health check function. Each interceptor must therefore invoke the 'next' interceptor. If the 'next' InterceptorFunc is not called, the components check health function will never be executed.

type InterceptorFunc

type InterceptorFunc func(ctx context.Context, name string, state CheckState) CheckState

InterceptorFunc is an interceptor function that intercepts any call to a components health check function.

type JSONResultWriter

type JSONResultWriter struct{}

JSONResultWriter writes a CheckerResult in JSON format into an http.ResponseWriter. This ResultWriter is set by default.

func NewJSONResultWriter

func NewJSONResultWriter() *JSONResultWriter

NewJSONResultWriter creates a new instance of a JSONResultWriter.

func (*JSONResultWriter) Write

func (r *JSONResultWriter) Write(result *CheckerResult, w http.ResponseWriter, req *http.Request) error

Write implements ResultWriter.Write.

type Middleware

type Middleware func(next MiddlewareFunc) MiddlewareFunc

Middleware is factory function that allows creating new instances of a MiddlewareFunc. A MiddlewareFunc is expected to forward the function call to the next MiddlewareFunc (passed in parameter 'next'). This way, a chain of interceptors is constructed that will eventually invoke of the Checker.Check function. Each interceptor must therefore invoke the 'next' interceptor. If the 'next' MiddlewareFunc is not called, Checker.Check will never be executed.

type MiddlewareFunc

type MiddlewareFunc func(ctx context.Context) CheckerResult

MiddlewareFunc is a middleware for a health Handler (see NewHandler). Is is invoked each time the Handler calls Checker.Check (which corresponds to every incoming HTTP request).

type ResultWriter

type ResultWriter interface {
	// Write writes a CheckerResult into a http.ResponseWriter in a format
	// that the ResultWriter supports (such as XML, JSON, etc.).
	Write(result *CheckerResult, w http.ResponseWriter, r *http.Request) error
}

ResultWriter enabled a Handler (see NewHandler) to write the CheckerResult to an http.ResponseWriter in a specific format. For example, the JSONResultWriter writes the result in JSON format into the response body).