health

README

go-sundheit

A library built to provide support for defining service health for golang services. It allows you to register async health checks for your dependencies and the service itself, and provides a health endpoint that exposes their status.

What's go-sundheit?

The project is named after the German word Gesundheit which means ‘health’, and it is pronounced /ɡəˈzʊntˌhaɪ̯t/.

Installation

Using go modules:

go get github.com/AppsFlyer/go-sundheit@v0.1.1

Using dep:

dep ensure -add github.com/AppsFlyer/go-sundheit@v0.1.1

Usage

import (
  "net/http"
  "time"
  "log"

  "github.com/pkg/errors"
  "github.com/AppsFlyer/go-sundheit"

  healthhttp "github.com/AppsFlyer/go-sundheit/http"
  "github.com/AppsFlyer/go-sundheit/checks"
)

func main() {
  // create a new health instance
  h := health.New()
  
  // define an HTTP dependency check
  httpCheckConf := checks.HTTPCheckConfig{
    CheckName: "httpbin.url.check",
    Timeout:   1 * time.Second,
    // dependency you're checking - use your own URL here...
    // this URL will fail 50% of the times
    URL:       "http://httpbin.org/status/200,300",
  }
  // create the HTTP check for the dependency
  // fail fast when you misconfigured the URL. Don't ignore errors!!!
  httpCheck, err := checks.NewHTTPCheck(httpCheckConf)
  if err != nil {
    fmt.Println(err)
    return // your call...
  }

  // Alternatively panic when creating a check fails
  httpCheck = checks.Must(checks.NewHTTPCheck(httpCheckConf))

  err = h.RegisterCheck(&health.Config{
    Check:           httpCheck, 
    InitialDelay:    time.Second,      // the check will run once after 1 sec
    ExecutionPeriod: 10 * time.Second, // the check will be executed every 10 sec
  })
  
  if err != nil {
    fmt.Println("Failed to register check: ", err)
    return // or whatever
  }

  // define more checks...
  
  // register a health endpoint
  http.Handle("/admin/health.json", healthhttp.HandleHealthJSON(h))
  
  // serve HTTP
  log.Fatal(http.ListenAndServe(":8080", nil))
}

Built-in Checks

The library comes with a set of built-in checks. Currently implemented checks are as follows:

HTTP built-in check

The HTTP check allows you to trigger an HTTP request to one of your dependencies, and verify the response status, and optionally the content of the response body. Example was given above in the usage section

DNS built-in check(s)

The DNS checks allow you to perform lookup to a given hostname / domain name / CNAME / etc, and validate that it resolves to at least the minimum number of required results.

Creating a host lookup check is easy:

// Schedule a host resolution check for `example.com`, requiring at least one results, and running every 10 sec
h.RegisterCheck(&health.Config{
  Check:           checks.NewHostResolveCheck("example.com", 200*time.Millisecond, 1),
  ExecutionPeriod: 10 * time.Second,
})

You may also use the low level checks.NewResolveCheck specifying a custom LookupFunc if you want to to perform other kinds of lookups. For example you may register a reverse DNS lookup check like so:

func ReverseDNLookup(ctx context.Context, addr string) (resolvedCount int, err error) {
  names, err := net.DefaultResolver.LookupAddr(ctx, addr)
  resolvedCount = len(names)
  return
}

//...

h.RegisterCheck(&health.Config{
  Check:           checks.NewResolveCheck(ReverseDNLookup, "127.0.0.1", 200*time.Millisecond, 3),
  ExecutionPeriod: 10 * time.Second,
})

Ping built-in check(s)

The ping checks allow you to verifies that a resource is still alive and reachable. For example, you can use it as a DB ping check (sql.DB implements the Pinger interface):

	db, err := sql.Open(...)
	dbCheck, err := checks.NewPingCheck("db.check", db, time.Millisecond*100)
	_ = h.RegisterCheck(&health.Config{
		Check: dbCheck,
		// ...
	})

You can also use the ping check to test a generic connection like so:

	pinger := checks.NewDialPinger("tcp", "example.com")
	pingCheck, err := checks.NewPingCheck("example.com.reachable", pinger, time.Second)
	h.RegisterCheck(&health.Config{
		Check: pingCheck,
		// ...
	})

The NewDialPinger function supports all the network/address parameters supported by the net.Dial() function(s)

Custom Checks

The library provides 2 means of defining a custom check. The bottom line is that you need an implementation of the checks.Check interface:

// Check is the API for defining health checks.
// A valid check has a non empty Name() and a check (Execute()) function.
type Check interface {
	// Name is the name of the check.
	// Check names must be metric compatible.
	Name() string
	// Execute runs a single time check, and returns an error when the check fails, and an optional details object.
	Execute() (details interface{}, err error)
}

See examples in the following 2 sections below.

Use the CustomCheck struct

The checksCustomCheck struct implements the checks.Check interface, and is the simplest way to implement a check if all you need is to define a check function.

Let's define a check function that fails 50% of the times:

func lotteryCheck() (details interface{}, err error) {
	lottery := rand.Float32()
	details = fmt.Sprintf("lottery=%f", lottery)
	if lottery < 0.5 {
		err = errors.New("Sorry, I failed")
	}
	return
}

Now we register the check to start running right away, and execute once per 2 minutes:

h := health.New()
...

h.RegisterCheck(&health.Config{
  Check: &checks.CustomCheck{
    CheckName: "lottery.check",
    CheckFunc: lotteryCheck,
  },
  InitialDelay:    0,
  ExecutionPeriod: 2 * time.Minute,
})

Implement the Check interface

Sometimes you need to define a more elaborate custom check. For example when you need to manage state. For these cases it's best to implement the checks.Check interface yourself.

Let's define a flexible example of the lottery check, that allows you to define a fail probability:

type Lottery struct {
	myname string
	probability float32
}

func (l Lottery) Execute() (details interface{}, err error) {
	lottery := rand.Float32()
	details = fmt.Sprintf("lottery=%f", lottery)
	if lottery < l.probability {
		err = errors.New("Sorry, I failed")
	}
	return
}

func (l Lottery) Name() string {
	return l.myname
}

And register our custom check, scheduling it to run after 1 sec, and every 30 sec:

h := health.New()
...

h.RegisterCheck(&health.Config{
  Check: Lottery{myname: "custom.lottery.check", probability:0.3,},
  InitialDelay: 1*time.Second,
  ExecutionPeriod: 30*time.Second,
})

Custom Checks Notes

1. If a check take longer than the specified rate period, then next execution will be delayed, but will not be concurrently executed.
2. Checks must complete within a reasonable time. If a check doesn't complete or gets hung, the next check execution will be delayed. Use proper time outs.
3. A health-check name must be a metric name compatible string (i.e. no funky characters, and spaces allowed - just make it simple like clicks-db-check). See here: https://help.datadoghq.com/hc/en-us/articles/203764705-What-are-valid-metric-names-

Expose Health Endpoint

The library provides an HTTP handler function for serving health stats in JSON format. You can register it using your favorite HTTP implementation like so:

http.Handle("/admin/health.json", healthhttp.HandleHealthJSON(h))

The endpoint can be called like so:

~ $ curl -i http://localhost:8080/admin/health.json
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Date: Tue, 22 Jan 2019 09:31:46 GMT
Content-Length: 701

{
	"custom.lottery.check": {
		"message": "lottery=0.206583",
		"error": {
			"message": "Sorry, I failed"
		},
		"timestamp": "2019-01-22T11:31:44.632415432+02:00",
		"num_failures": 2,
		"first_failure_time": "2019-01-22T11:31:41.632400256+02:00"
	},
	"lottery.check": {
		"message": "lottery=0.865335",
		"timestamp": "2019-01-22T11:31:44.63244047+02:00",
		"num_failures": 0,
		"first_failure_time": null
	},
	"url.check": {
		"message": "http://httpbin.org/status/200,300",
		"error": {
			"message": "unexpected status code: '300' expected: '200'"
		},
		"timestamp": "2019-01-22T11:31:44.632442937+02:00",
		"num_failures": 4,
		"first_failure_time": "2019-01-22T11:31:38.632485339+02:00"
	}
}

Or for the shorter version:

~ $ curl -i http://localhost:8080/admin/health.json?type=short
HTTP/1.1 503 Service Unavailable
Content-Type: application/json
Date: Tue, 22 Jan 2019 09:40:19 GMT
Content-Length: 105

{
	"custom.lottery.check": "PASS",
	"lottery.check": "PASS",
	"my.check": "FAIL",
	"url.check": "PASS"
}

The short response type is suitable for the consul health checks / LB heath checks.

The response code is 200 when the tests pass, and 503 when they fail.

CheckListener

It is sometimes desired to keep track of checks execution and apply custom logic. For example, you may want to add logging, or external metrics to your checks, or add some trigger some recovery logic when a check fails after 3 consecutive times.

The health.CheckListener interface allows you to hook this custom logic.

For example, lets add a logging listener to our health repository:

type checkEventsLogger struct{}

func (l checkEventsLogger) OnCheckStarted(name string) {
	log.Printf("Check %q started...\n", name)
}

func (l checkEventsLogger) OnCheckCompleted(name string, res health.Result) {
	log.Printf("Check %q completed with result: %v\n", name, res)
}

err := h.RegisterCheck(&Config{
	Check:           check,
	InitialDelay:    time.Second,      // the check will run once after 1 sec
	ExecutionPeriod: 10 * time.Second, // the check will be executed every 10 sec
	Listener:        checkEventsLogger{}, //listener will registered for this check
})

Please note that your CheckListener implementation must not block!

Metrics

The library exposes the following OpenCensus view for your convenience:

• health/check_status_by_name - An aggregated health status gauge (0/1 for fail/pass) at the time of sampling. The aggregation uses the following tags:
  • check=allChecks - all checks aggregation
  • check=<check-name> - specific check aggregation
• health/check_count_by_name_and_status - Aggregated pass/fail counts for checks, with the following tags:
  • check=allChecks - all checks aggregation
  • check=<check-name> - specific check aggregation
  • check-passing=[true|false]
• health/executeTime - The time it took to execute a checks. Using the following tag:
  • check=<check-name> - specific check aggregation

The views can be registered like so:

import (
	"github.com/AppsFlyer/go-sundheit"
	"go.opencensus.io/stats/view"
)

h := health.New()
// ...
view.Register(health.DefaultHealthViews...)
// or register individual views. For example:
view.Register(health.ViewCheckExecutionTime, health.ViewCheckStatusByName, ...)

Documentation

Index

• Constants
• Variables
• type CheckListener
• type Config
• type Health
  • func New() Health
• type Result
  • func (r Result) IsHealthy() bool
  • func (r Result) String() string

Constants

const (

	// ValAllChecks is the value used for the check tags when tagging all tests
	ValAllChecks = "all_checks"
)

Variables

var (

	// ViewCheckExecutionTime is the checks execution time aggregation tagged by check name
	ViewCheckExecutionTime = &view.View{
		Measure:     mCheckDuration,
		TagKeys:     []tag.Key{keyCheck},
		Aggregation: view.Distribution(0, 1, 2, 3, 4, 6, 8, 10, 13, 16, 20, 25, 30, 40, 50, 65, 80, 100, 120, 160, 200, 250, 300, 500),
	}

	// ViewCheckCountByNameAndStatus is the checks execution count aggregation grouped by check name, and check status
	ViewCheckCountByNameAndStatus = &view.View{
		Name:        "health/check_count_by_name_and_status",
		Measure:     mCheckStatus,
		TagKeys:     []tag.Key{keyCheck, keyCheckPassing},
		Aggregation: view.Count(),
	}

	// ViewCheckStatusByName is the checks status aggregation tagged by check name
	ViewCheckStatusByName = &view.View{
		Name:        "health/check_status_by_name",
		Measure:     mCheckStatus,
		TagKeys:     []tag.Key{keyCheck},
		Aggregation: view.LastValue(),
	}

	// DefaultHealthViews are the default health check views provided by this package.
	DefaultHealthViews = []*view.View{
		ViewCheckCountByNameAndStatus,
		ViewCheckStatusByName,
		ViewCheckExecutionTime,
	}
)

Types

type CheckListener

type CheckListener interface {
	// OnCheckStarted is called when a check with the specified name has started
	OnCheckStarted(name string)

	// OnCheckCompleted is called when the check with the specified name has completed it's execution.
	// The results are passed as an argument
	OnCheckCompleted(name string, result Result)
}

CheckListener can be used to gain check stats or log check transitions. Implementations of this interface **must not block!** If an implementation blocks, it may result in delayed execution of other health checks down the line. It's OK to log in the implementation and it's OK to add metrics, but it's not OK to run anything that takes long time to complete such as network IO etc.

type Config

type Config struct {
	// Check is the health Check to be scheduled for execution.
	Check checks.Check
	// CheckListener is the listener to result of check and will call after check execution.
	Listener CheckListener
	// ExecutionPeriod is the period between successive executions.
	ExecutionPeriod time.Duration
	// InitialDelay is the time to delay first execution; defaults to zero.
	InitialDelay time.Duration
	// InitiallyPassing indicates when true, the check will be treated as passing before the first run; defaults to false
	InitiallyPassing bool
}

Config defines a health Check and it's scheduling timing requirements.

type Health

type Health interface {
	// RegisterCheck registers a health check according to the given configuration.
	// Once RegisterCheck() is called, the check is scheduled to run in it's own goroutine.
	// Callers must make sure the checks complete at a reasonable time frame, or the next execution will delay.
	RegisterCheck(cfg *Config) error
	// Deregister removes a health check from this instance, and stops it's next executions.
	// If the check is running while Deregister() is called, the check may complete it's current execution.
	// Once a check is removed, it's results are no longer returned.
	Deregister(name string)
	// Results returns a snapshot of the health checks execution results at the time of calling, and the current health.
	// A system is considered healthy iff all checks are passing
	Results() (results map[string]Result, healthy bool)
	// IsHealthy returns the current health of the system.
	// A system is considered healthy iff all checks are passing.
	IsHealthy() bool
	// DeregisterAll Deregister removes all health checks from this instance, and stops their next executions.
	// It is equivalent of calling Deregister() for each currently registered check.
	DeregisterAll()
}

Health is the API for registering / deregistering health checks, and for fetching the health checks results.

func New

func New() Health

New returns a new Health instance.

type Result

type Result struct {
	// the details of task Result - may be nil
	Details interface{} `json:"message,omitempty"`
	// the error returned from a failed health check - nil when successful
	Error error `json:"error,omitempty"`
	// the time of the last health check
	Timestamp time.Time `json:"timestamp"`
	// the execution duration of the last check
	Duration time.Duration `json:"duration,omitempty"`
	// the number of failures that occurred in a row
	ContiguousFailures int64 `json:"contiguousFailures"`
	// the time of the initial transitional failure
	TimeOfFirstFailure *time.Time `json:"timeOfFirstFailure"`
}

Result represents the output of a health check execution.

func (Result) IsHealthy

func (r Result) IsHealthy() bool

func (Result) String

func (r Result) String() string