health

README

Health

A simple and flexible health check library for Go.

Documentation · Report Bug · Request Feature

Table of Contents

1. Features
2. Getting Started
3. Caching
4. Periodic Checks
5. Failure Tolerant Checks
6. Listening for Status Changes
7. License

Features

This library allows you to build health checks that do not simply return HTTP status code 200 but actually check if all necessary components are healthy.

This library provides the following features:

• Request based and fixed-schedule health checks.
• Global and check-based timeout management.
• Caching (to avoid sending too many requests to downstream services).
• Custom HTTP request middleware for pre- and postprocessing HTTP requests and/or responses.
• Failure tolerance based on fail count and/or time thresholds.
• Provides an http.Handler that can be easily used with a mux.

Getting Started

package main

import (
	"context"
	"database/sql"
	"fmt"
	"github.com/alexliesenfeld/health"
	_ "github.com/mattn/go-sqlite3"
	"net/http"
	"time"
)

func main() {
	db, _ := sql.Open("sqlite3", "simple.sqlite")
	defer db.Close()

	router := http.NewServeMux()
	
	// Create a new http.Handler that provides health check information.
	router.Handle("/health", health.NewHandler(
		
		// Configure a global timeout that will be applied to all checks.
		health.WithTimeout(10*time.Second),

		// A simple check to see if database connection is up.
		health.WithCheck(health.Check{                          
			Name:  "database",
			Timeout: 2*time.Second, // A check specific timeout.
			Check: db.PingContext,
		}),
		
		// The following check will be executed periodically every 30 seconds.
		health.WithPeriodicCheck(30*time.Second, health.Check{  
			Name: "search",
			Check: func(ctx context.Context) error {
				return fmt.Errorf("this makes the check fail")
			},
		}),
	))

	http.ListenAndServe(":3000", router)
}

If our database is down, the request curl -u username:password http://localhost:3000/health would yield a response with HTTP status code 503 (Service Unavailable), and the following JSON response body:

{
   "status": "down",
   "details":{
      "database": {
         "status": "up",
         "timestamp": "2021-07-01T08:05:14.603364Z"
      },
      "search": {
         "status": "down",
         "timestamp": "2021-07-01T08:05:08.522685Z",
          "error": "this makes the check fail"
      }
   }
}

Caching

Health check responses are cached to avoid sending too many request to the services that your program checks and to mitigate "denial of service" attacks. The TTL is set to 1 second by default. If you do not want to use caching altogether, you can disable it using the health.WithDisabledCache() configuration option.

Periodic Checks

When executing health check functions synchronously (i.e. for every HTTP request), the overall response delay will be at least as high as the one of your slowest check function. This is usually OK for smaller applications with a low number of quickly checkable dependencies and with enabled caching. This approach, however, will likely be problematic for more involved applications that either have many dependencies and/or some relatively slow check functions.

Rather than executing a health check function on every request that is received over the health endpoint, periodic checks execute the check function on a fixed schedule. With this approach, the health status is always read from a local cache. It allows responding to HTTP requests instantly without waiting for the check function to complete.

Periodic checks can be configured using the WithPeriodicCheck configuration option (see example above).

Failure Tolerant Checks

This library lets you configure failure tolerant checks that allow some degree of failure. The check is only considered failed, when predefined tolerance thresholds have been crossed.

To allow some failure, please have a look at the FailureTolerance and FailureToleranceThreshold attributes in your check configuration.

Example

Let's assume that your app provides a REST API but also consumes messages from a Kafka topic. If the connection to Kafka is down, your app can still serve API requests, but it will not process any messages during this time. If the Kafka health check is configured without any failure tolerance, and the connection to Kafka is temporarily down, your whole application will become unhealthy. This is most likely not what you want. However, if Kafka is down for too long, there may indeed be a problem that requires attention. In this case, you still may want to flag your app unhealthy by returning a failing health check, so that it can be automatically restarted by your infrastructure.

Failure tolerant health checks let you configure this kind of behaviour.

Listening for Status Changes

It can be useful to react to health status changes. For example, you might want to log status changes, so you can easier correlate logs during root cause analysis or perform actions to mitigate the impact of an unhealthy component.

This library allows you to configure listener functions that will be called either when the overall system health status, or a component status changes.

Exmample

The exapmple below shows a configuration that adds

• a status listener to a check that will be called whenever the status of the check changes (e.g. from "up" to "down"), and
• an overall system status listener, that will be called whenever the overall system status changes.

health.WithPeriodicCheck(5*time.Second, health.Check{
        Name:   "search",
        Check:  myCheckFunc,
        StatusListener: func(name string, state health.CheckState) {
            log.Printf("status of component %s changed to %s", name, state.Status)
        },
    }),

    health.WithStatusListener(func(status health.Status, state map[string]health.CheckState) {
        log.Printf("overall system health status changed to %s", status)
    }),

License

health is free software: you can redistribute it and/or modify it under the terms of the MIT Public License.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the MIT Public License for more details.

Documentation

Index

• func WithCacheDuration(duration time.Duration) option
• func WithCheck(check Check) option
• func WithCustomStatusCodes(statusCodeUp int, statusCodeDown int) option
• func WithDisabledCache() option
• func WithDisabledDetails() option
• func WithManualStart() option
• func WithMaxErrorMessageLength(length uint) option
• func WithPeriodicCheck(refreshPeriod time.Duration, check Check) option
• func WithStatusListener(listener SystemStatusListener) option
• func WithTimeout(timeout time.Duration) option
• type Check
• type CheckResult
• type CheckState
• type CheckStatusListener
• type Handler
  • func NewHandler(options ...option) Handler
  • func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (h Handler) Start()
  • func (h Handler) Stop()
• type Status
  • func (s Status) Criticality() int
• type SystemStatusListener

Functions

func WithCacheDuration

func WithCacheDuration(duration time.Duration) option

WithCacheDuration sets the duration for how long the aggregated health check result will be cached. This is set to 1 second by default. Caching will prevent that each incoming HTTP request triggers a new health check. A duration of 0 will effectively disable the cache and has the same effect as WithDisabledCache.

func WithCheck

func WithCheck(check Check) option

WithCheck adds a new health check that contributes to the overall service availability Status. This check will be triggered each time the health check HTTP endpoint is called (and the cache has expired, see WithCacheDuration). If health checks are expensive or you expect a lot of calls to the health endpoint, consider using WithPeriodicCheck instead.

func WithCustomStatusCodes

func WithCustomStatusCodes(statusCodeUp int, statusCodeDown int) option

WithCustomStatusCodes allows to set custom HTTP Status code for the case when the system is evaluated to be up or down (based on check results). Default values are statusCodeUp = 200 (OK) and statusCodeDown = 503 (Service Unavailable).

func WithDisabledCache

func WithDisabledCache() option

WithDisabledCache disabled the check cache. This is not recommended in most cases. This will effectively lead to a health endpoint that initiates a new health check for each incoming HTTP request. This may have an impact on the systems that are being checked (especially if health checks are expensive). Caching also mitigates "denial of service" attacks.

func WithDisabledDetails

func WithDisabledDetails() option

WithDisabledDetails disables hides all data in the JSON response body but the the Status itself. Example: { "Status":"down" }

func WithManualStart

func WithManualStart() option

WithManualStart prevents an automatic start of periodic checks (see NewHandler). If this configuration option is used and you want to start periodic checks yourself, you need to start them by using Start.

func WithMaxErrorMessageLength

func WithMaxErrorMessageLength(length uint) option

WithMaxErrorMessageLength limits maximum number of characters in error messages.

func WithPeriodicCheck

func WithPeriodicCheck(refreshPeriod time.Duration, check Check) option

WithPeriodicCheck adds a new health check that contributes to the overall service availability Status. The health check will be performed on a fixed schedule and will not be executed for each HTTP request (as in contrast to WithCheck). This allows to process a much higher number of HTTP requests without actually calling the checked services too often or to execute long running checks. The health endpoint always returns the last result of the periodic check. When periodic checks are started (happens automatically if WithManualStart is not used) they are also executed for the first time. Until all periodic checks have not been executed at least once, the overall availability Status will be "unknown" with HTTP Status code 503 (Service Unavailable).

func WithStatusListener

func WithStatusListener(listener SystemStatusListener) option

WithStatusListener registers a handler function that will be called whenever the overall system health Status changes. Attention: Ideally, this method should be quick and not block for too long.

func WithTimeout

func WithTimeout(timeout time.Duration) option

WithTimeout globally defines a timeout duration for all checks. You can still override this timeout by using the timeout value in the Check configuration. Default value is 30 seconds.

Types

type Check

type Check struct {
	// The Name must be unique among all checks. Name is a required attribute.
	Name string
	// Check is the check function that will be executed to check availability.
	// This function must return an error if the checked service is considered
	// not available. Check is a required attribute.
	Check func(ctx context.Context) error
	// Timeout will override the global timeout value, if it is smaller than
	// the global timeout (see WithTimeout).
	Timeout time.Duration
	// FailureTolerance will set a duration for how long a service must be
	// "in error" until it is considered unavailable.
	FailureTolerance time.Duration
	// FailureToleranceThreshold will set a maximum number of consecutive
	// check fails until the service is considered unavailable.
	FailureToleranceThreshold uint
	// StatusListener allows to set a listener that will be called
	// whenever the Status of the check changes.
	StatusListener CheckStatusListener
	// contains filtered or unexported fields
}

Check allows to configure health checks.

type CheckResult

type CheckResult struct {
	Status    Status    `json:"status"`
	Timestamp time.Time `json:"timestamp,omitempty"`
	Error     *string   `json:"error,omitempty"`
}

type CheckState

type CheckState struct {
	LastCheckedAt    time.Time
	LastSuccessAt    time.Time
	LastResult       error
	ConsecutiveFails uint
	Status           Status
	// contains filtered or unexported fields
}

type CheckStatusListener

type CheckStatusListener func(name string, state CheckState)

type Handler

type Handler struct {
	http.Handler
	// contains filtered or unexported fields
}

func NewHandler

func NewHandler(options ...option) Handler

NewHandler creates a new health check http.Handler. If periodic checks have been configured (see WithPeriodicCheck), they will be started as well (if not explicitly turned off using WithManualStart).

func (Handler) ServeHTTP

func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request)

func (Handler) Start

func (h Handler) Start()

Start allows to start periodic checks manually if the health check was configured using WithManualStart or when checks have been stopped earlier using health.Stop. This function has no effect otherwise.

func (Handler) Stop

func (h Handler) Stop()

Stop stops all periodic checks. This function has only effect after automatic startup (i.e. when the handler was not configured using WithManualStart) or when peridic checks have been started before manually using Start. This function will have no effect otherwise. It is usually not necessary to call this function manually. Attention: This function does not block until all checks have been stopped!

type Status

type Status string

const (
	StatusUnknown Status = "unknown"
	StatusUp      Status = "up"
	StatusDown    Status = "down"
)

func (Status) Criticality

func (s Status) Criticality() int

type SystemStatusListener

type SystemStatusListener func(status Status, state map[string]CheckState)