health

README

go-health

A library that enables async dependency health checking for services running on an orchestrated container platform such as kubernetes or mesos.

Why is this important?

Container orchestration platforms require that the underlying service(s) expose a "health check" which is used by the platform to determine whether the container is in a good or bad state.

While this can be achieved by simply exposing a /status endpoint that performs synchronous checks against its dependencies (followed by returning a 200 or non-200 status code), it is not optimal for a number of reasons:

• It does not scale
  • The more dependencies you add, the longer your health check will take to complete (and potentially cause your service to be killed off by the orchestration platform).
  • Depending on the complexity of a given dependency, your check may be fairly involved where it is okay for it to take 30s+ to complete.
• It adds unnecessary load on yours deps or at worst, becomes a DoS target
  • Non-malicious scenario
    • Thundering herd problem -- in the event of a deployment (or restart, etc.), all of your service containers are likely to have their /status endpoints checked by the orchestration platform as soon as they come up. Depending on the complexity of the checks, running that many simultaneous checks against your dependencies could cause at worst the dependencies to experience problems and at minimum add unnecessary load.
    • Security scanners -- if your organization runs periodic security scans, they may hit your /status endpoint and trigger unnecessary dep checks.
  • Malicious scenario
    • Loading up any basic HTTP benchmarking tool and pointing it at your /status endpoint could choke your dependencies (and potentially your service).

With that said, not everyone needs asynchronous checks. If your service has one dependency (and that is unlikely to change), it is trivial to write a basic, synchronous check and it will probably suffice.

However, if you anticipate that your service will have several dependencies, with varying degrees of complexity for determining their health state - you should probably think about introducing asynchronous health checks.

How does this library help?

Writing an async health checking framework for your service is not a trivial task, especially if Go is not your primary language.

This library:

• Allows you to define how to check your dependencies.
• Allows you to define warning and fatal thresholds.
• Will run your dependency checks on a given interval, in the background. [1]
• Exposes a way for you to gather the check results in a fast and thread-safe manner to help determine the final status of your /status endpoint. [2]
• Comes bundled w/ pre-built checkers for well-known dependencies such as Redis, Mongo, HTTP and more.
• Makes it simple to implement and provide your own checkers (by adhering to the checker interface).
• Allows you to trigger listener functions when your health checks fail or recover using the IStatusListener interface.
• Allows you to run custom logic when a specific health check completes by using the OnComplete hook.

[1] Make sure to run your checks on a "sane" interval - ie. if you are checking your Redis dependency once every five minutes, your service is essentially running blind for about 4.59/5 minutes. Unless you have a really good reason, check your dependencies every X seconds, rather than X minutes.

[2] go-health continuously writes dependency health state data and allows you to query that data via .State(). Alternatively, you can use one of the pre-built HTTP handlers for your /healthcheck endpoint (and thus not have to manually inspect the state data).

Example

For full examples, look through the examples dir

1. Create an instance of health and configure a checker (or two)

import (
	health "github.com/InVisionApp/go-health"
	"github.com/InVisionApp/go-health/checkers"
	"github.com/InVisionApp/go-health/handlers"
)

// Create a new health instance
h := health.New()

// Create a checker
myURL, _ := url.Parse("https://google.com")
myCheck, _ := checkers.NewHTTP(&checkers.HTTPConfig{
    URL: myURL,
})

1. Register your check with your health instance

h.AddChecks([]*health.Config{
    {
        Name:     "my-check",
        Checker:  myCheck,
        Interval: time.Duration(2) * time.Second,
        Fatal:    true,
    },
})

1. Start the health check

h.Start()

From here on, you can either configure an endpoint such as /healthcheck to use a built-in handler such as handlers.NewJSONHandlerFunc() or get the current health state of all your deps by traversing the data returned by h.State().

Sample /healthcheck output

Assuming you have configured go-health with two HTTP checkers, your /healthcheck output would look something like this:

{
    "details": {
        "bad-check": {
            "name": "bad-check",
            "status": "failed",
            "error": "Ran into error while performing 'GET' request: Get google.com: unsupported protocol scheme \"\"",
            "check_time": "2017-12-30T16:20:13.732240871-08:00"
        },
        "good-check": {
            "name": "good-check",
            "status": "ok",
            "check_time": "2017-12-30T16:20:13.80109931-08:00"
        }
    },
    "status": "ok"
}

Additional Documentation

• Examples
  • Status Listeners
  • OnComplete Hook
• Checkers

OnComplete Hook VS IStatusListener

At first glance it may seem that these two features provide the same functionality. However, they are meant for two different use cases:

The IStatusListener is useful when you want to run a custom function in the event that the overall status of your health checks change. I.E. if go-health is currently checking the health for two different dependencies A and B, you may want to trip a circuit breaker for A and/or B. You could also put your service in a state where it will notify callers that it is not currently operating correctly. The opposite can be done when your service recovers.

The OnComplete hook is called whenever a health check for an individual dependency is complete. This means that the function you register with the hook gets called every single time go-health completes the check. It's completely possible to register different functions with each configured health check or not to hook into the completion of certain health checks entirely. For instance, this can be useful if you want to perform cleanup after a complex health check or if you want to send metrics to your APM software when a health check completes. It is important to keep in mind that this hook effectively gets called on roughly the same interval you define for the health check.

Contributing

All PR's are welcome, as long as they are well tested. Follow the typical fork->branch->pr flow.

Documentation

Overview

Package health is a library that enables *async* dependency health checking for services running on an orchestrated container platform such as kubernetes or mesos.

For additional overview, documentation and contribution guidelines, refer to the project's "README.md".

For example usage, refer to https://github.com/InVisionApp/go-health/tree/master/examples/simple-http-server.

Index

• Variables
• type Config
• type Health
  • func New() *Health
  • func (h *Health) AddCheck(cfg *Config) error
  • func (h *Health) AddChecks(cfgs []*Config) error
  • func (h *Health) DisableLogging()
  • func (h *Health) Failed() bool
  • func (h *Health) Start() error
  • func (h *Health) State() (map[string]State, bool, error)
  • func (h *Health) Stop() error
• type ICheckable
• type IHealth
• type IStatusListener
• type State

Variables

var (
	// ErrNoAddCfgWhenActive is returned when you attempt to add check(s) to an already active healthcheck instance
	ErrNoAddCfgWhenActive = errors.New("Unable to add new check configuration(s) while healthcheck is active")

	// ErrAlreadyRunning is returned when you attempt to "h.Start()" an already running healthcheck
	ErrAlreadyRunning = errors.New("Healthcheck is already running - nothing to start")

	// ErrAlreadyStopped is returned when you attempt to "h.Stop()" a non-running healthcheck instance
	ErrAlreadyStopped = errors.New("Healthcheck is not running - nothing to stop")

	// ErrEmptyConfigs is returned when you attempt to add an empty slice of configs via "h.AddChecks()"
	ErrEmptyConfigs = errors.New("Configs appears to be empty - nothing to add")
)

Types

type Config

type Config struct {
	// Name of the check
	Name string

	// Checker instance used to perform health check
	Checker ICheckable

	// Interval between health checks
	Interval time.Duration

	// Fatal marks a failing health check so that the
	// entire health check request fails with a 500 error
	Fatal bool

	// Hook that gets called when this health check is complete
	OnComplete func(state *State)
}

Config is a struct used for defining and configuring checks.

type Health

type Health struct {
	Logger log.Logger

	// StatusListener will report failures and recoveries
	StatusListener IStatusListener
	// contains filtered or unexported fields
}

Health contains internal go-health internal structures.

func New

func New() *Health

New returns a new instance of the Health struct.

func (*Health) AddCheck

func (h *Health) AddCheck(cfg *Config) error

AddCheck is used for adding a single check definition to the current health instance.

func (*Health) AddChecks

func (h *Health) AddChecks(cfgs []*Config) error

AddChecks is used for adding multiple check definitions at once (as opposed to adding them sequentially via "AddCheck()").

func (*Health) DisableLogging

func (h *Health) DisableLogging()

DisableLogging will disable all logging by inserting the noop logger.

func (*Health) Failed

func (h *Health) Failed() bool

Failed will return the basic state of overall health. This should be used when details about the failure are not needed

func (*Health) Start

func (h *Health) Start() error

Start will start all of the defined health checks. Each of the checks run in their own goroutines (as "time.Ticker").

func (*Health) State

func (h *Health) State() (map[string]State, bool, error)

State will return a map of all current healthcheck states (thread-safe), a bool indicating whether the healthcheck has fully failed and a potential error.

The returned structs can be used for figuring out additional analytics or used for building your own status handler (as opposed to using the built-in "hc.HandlerBasic" or "hc.HandlerJSON").

The map key is the name of the check.

func (*Health) Stop

func (h *Health) Stop() error

Stop will cause all of the running health checks to be stopped. Additionally, all existing check states will be reset.

type ICheckable

type ICheckable interface {
	// Status allows you to return additional data as an "interface{}" and "error"
	// to signify that the check has failed. If "interface{}" is non-nil, it will
	// be exposed under "State.Details" for that particular check.
	Status() (interface{}, error)
}

ICheckable is an interface implemented by a number of bundled checkers such as "MySQLChecker", "RedisChecker" and "HTTPChecker". By implementing the interface, you can feed your own custom checkers into the health library.

type IHealth

type IHealth interface {
	AddChecks(cfgs []*Config) error
	AddCheck(cfg *Config) error
	Start() error
	Stop() error
	State() (map[string]State, bool, error)
	Failed() bool
}

The IHealth interface can be useful if you plan on replacing the actual health checker with a mock during testing. Otherwise, you can set "hc.Disable = true" after instantiation.

type IStatusListener

type IStatusListener interface {
	// HealthCheckFailed is a function that handles the failure of a health
	// check event. This function is called when a health check state
	// transitions from passing to failing.
	// 	* entry - The recorded state of the health check that triggered the failure
	HealthCheckFailed(entry *State)

	// HealthCheckRecovered is a function that handles the recovery of a failed
	// health check.
	// 	* entry - The recorded state of the health check that triggered the recovery
	// 	* recordedFailures - the total failed health checks that lapsed
	// 	  between the failure and recovery
	//	* failureDurationSeconds - the lapsed time, in seconds, of the recovered failure
	HealthCheckRecovered(entry *State, recordedFailures int64, failureDurationSeconds float64)
}

IStatusListener is an interface that handles health check failures and recoveries, primarily for stats recording purposes

type State

type State struct {
	// Name of the health check
	Name string `json:"name"`

	// Status of the health check state ("ok" or "failed")
	Status string `json:"status"`

	// Err is the error returned from a failed health check
	Err string `json:"error,omitempty"`

	// Fatal shows if the check will affect global result
	Fatal bool `json:"fatal,omitempty"`

	// Details contains more contextual detail about a
	// failing health check.
	Details interface{} `json:"details,omitempty"` // contains JSON message (that can be marshaled)

	// CheckTime is the time of the last health check
	CheckTime time.Time `json:"check_time"`

	ContiguousFailures int64     `json:"num_failures"`     // the number of failures that occurred in a row
	TimeOfFirstFailure time.Time `json:"first_failure_at"` // the time of the initial transitional failure for any given health check
}

State is a struct that contains the results of the latest run of a particular check.