checkup

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 3, 2016 License: BSD-3-Clause Imports: 17 Imported by: 0

README

Checkup

GoDoc

Checkup is distributed, lock-free, self-hosted health checks and status pages, written in Go.

It features an elegant, minimalistic CLI and an idiomatic Go library. They are completely interoperable and their configuration is beautifully symmetric.

This tool is a WIP. Please use liberally with discretion and report any bugzies!

Intro

Checkup can be customized to check up on any of your sites or services at any time, from any infrastructure, using any storage provider of your choice. The status page can be customized to your liking since you can do your checks however you want.

Out of the box, Checkup currently supports:

  • Checking HTTP endpoints
  • Storing results on S3
  • Viewing results on a status page that is mobile-responsive and 100% static

How it Works

There are 3 components:

  1. Storage You set up storage space for the results of the checks.

  2. Checks You run checks on whatever endpoints you have as often as you want.

  3. Status Page You host the status page. Caddy makes this super easy. The status page downloads recent check files from storage and renders the results client-side.

Quick Start

$ checkup --help

Follow these instructions to get started quickly with Checkup.

Create your Checkup config

You can configure Checkup entirely with a simple JSON document. We recommend you configure storage and at least one checker:

{
	"checkers": [{
		"type": "http",
		"endpoint_name": "Example",
		"endpoint_url": "http://www.example.com",
		"attempts": 5
	}],
	"storage": {
		"provider": "s3",
		"access_key_id": "<yours>",
		"secret_access_key": "<yours>",
		"bucket": "<yours>"
	}
}

For the complete structure definition, please see the godoc. There are many elements of checkers and storage you may wish to customize!

Save this file as config.json in your working directory.

Setting up storage on S3

The easiest way to do this is to give an IAM user these two privileges (keep the credentials secret):

  • arn:aws:iam::aws:policy/IAMFullAccess
  • arn:aws:iam::aws:policy/AmazonS3FullAccess
Implicit Provisioning

If you give these permissions to the same user as with the credentials in your JSON config above, then you can simply run:

$ checkup provision

and checkup will read the config file and provision S3 for you. If the user is different, you may want to use explicit provisioning instead.

This command creates a new IAM user with read-only permission to S3 and also creates a new bucket just for your check files. The credentials of the new user are printed to your screen. Make note of the Public Access Key ID and Public Access Key! You won't be able to see them again.

Explicit Provisioning

If you do not prefer implicit provisioning using your checkup.json file, do this instead. Export the information to environment variables and run the provisioning command:

$ export AWS_ACCESS_KEY_ID=...
$ export AWS_SECRET_ACCESS_KEY=...
$ export AWS_BUCKET_NAME=...
$ checkup provision s3

Setting up the status page

In statuspage/js, use the contents of config_template.js to fill out config.js, which is used by the status page. This is where you put the read-only S3 credentials you just generated.

Then, the status page can be served over HTTPS by running caddy -host status.mysite.com on the command line. (You can use getcaddy.com to install Caddy.)

As you perform checks, the status page will update every so often with the latest results. Only checks that are stored will appear on the status page.

Performing checks

You can run checks many different ways: cron, AWS Lambda, or a time.Ticker in your own Go program, to name a few. Checks should be run on a regular basis. How often you run checks depends on your requirements and how much time you render on the status page.

For example, if you run checks every 10 minutes, showing the last 24 hours on the status page will require 144 check files to be downloaded on each page load. You can distribute your checks to help avoid localized network problems, but this multiplies the number of files by the number of nodes you run checks on, so keep that in mind.

Performing checks with the checkup command is very easy.

Just cd to the folder with your config.json from earlier, and checkup will automatically use it:

$ checkup

The vanilla checkup command runs a single check and prints the results to your screen, but does not save them to storage for your status page.

To store the results instead, use --store:

$ checkup --store

If you want Checkup to loop forever and perform checks and store them on a regular interval, use this:

$ checkup every 10m

And replace the duration with your own preference. In addition to the regular time.ParseDuration() formats, you can use shortcuts like second, minute, hour, day, or week.

You can also get some help using the -h option for any command or subcommand.

Posting status messages

Site reliability engineers should post messages when there are incidents or other news relevant for a status page. This is also very easy:

$ checkup message --about=Example "Oops. We're trying to fix the problem. Stay tuned."

This stores a check file with your message attached to the result for "Example" which you configured in config.json earlier.

Doing all that, but with Go

Checkup is as easy to use in a Go program as it is on the command line.

Using Go to set up storage on S3

(If you'd rather do this manually, see the instructions on the wiki.

First, create an IAM user with credentials as described in the section above.

Then go get github.com/sourcegraph/checkup and import it.

Then replace ACCESS_KEY_ID and SECRET_ACCESS_KEY below with the actual values for that user. Keep those secret. You'll also replace BUCKET_NAME with the unique bucket name to store your check files:

storage := checkup.S3{
	AccessKeyID:     "ACCESS_KEY_ID",
	SecretAccessKey: "SECRET_ACCESS_KEY",
	Bucket:          "BUCKET_NAME",
}
info, err := storage.Provision()
if err != nil {
	log.Fatal(err)
}
fmt.Println(info) // don't lose this output!

This method creates a new IAM user with read-only permission to S3 and also creates a new bucket just for your check files. The credentials of the new user are printed to your screen. Make note of the PublicAccessKeyID and PublicAccessKey! You won't be able to see them again.

Using Go to perform checks

First, go get github.com/sourcegraph/checkup and import it. Then configure it:

c := checkup.Checkup{
	Checkers: []checkup.Checker{
		checkup.HTTPChecker{Name: "Example (HTTP)", URL: "http://www.example.com", Attempts: 5},
		checkup.HTTPChecker{Name: "Example (HTTPS)", URL: "https://www.example.com", Attempts: 5},
	},
	Storage: checkup.S3{
		AccessKeyID:     "<yours>",
		SecretAccessKey: "<yours>",
		Bucket:          "<yours>",
		Region:          "us-east-1",
		CheckExpiry:     24 * time.Hour * 7,
	},
}

This sample checks 2 endpoints (HTTP and HTTPS). Each check consists of 5 attempts so as to smooth out the final results a bit. We will store results on S3. Notice the CheckExpiry value. The checkup.S3 type is also checkup.Maintainer type, which means it can maintain itself and purge any status checks older than CheckExpiry. We chose 7 days.

Then, to run checks every 10 minutes:

c.CheckAndStoreEvery(10 * time.Minute)
select {}

CheckAndStoreEvery() returns a time.Ticker that you can stop, but in this case we just want it to run forever, so we block forever using an empty select.

Using Go to post status messages

Simply perform a check, add the message to the corresponding result, and then store it:

results, err := c.Check()
if err != nil {
	// handle err
}

results[0].Message = "We're investigating connectivity issues."

err = c.Storage.Store(results)
if err != nil {
	// handle err
}

Of course, real status messages should be as descriptive as possible. You can use HTML in them.

Other topics

Getting notified when there are problems

Uh oh, having some fires? 🔥 You can create a type that implements checkup.Notifier. Checkup will invoke Notify() after every check, where you can evaluate the results and decide if and how you want to send a notification or trigger some event.

Other kinds of checks or storage providers

Need to check more than HTTP? S3 too Amazony for you? You can implement your own Checker and Storage types. If it's general enough, feel free to submit a pull request so others can use it too!

Documentation

Overview

Package checkup provides means for checking and reporting the status and performance of various endpoints in a distributed, lock-free, self-hosted fashion.

Index

Constants

View Source 
const FilenameFormatString = "%d-check.json"

FilenameFormatString is the format string used by GenerateFilename to create a filename.

Variables

View Source 
var DefaultConcurrentChecks = 5

DefaultConcurrentChecks is how many checks, at most, to perform concurrently.

View Source 
var DefaultHTTPClient = &http.Client{
	Transport: &http.Transport{
		Proxy: http.ProxyFromEnvironment,
		Dial: (&net.Dialer{
			Timeout:   10 * time.Second,
			KeepAlive: 0,
		}).Dial,
		TLSHandshakeTimeout:   5 * time.Second,
		ExpectContinueTimeout: 1 * time.Second,
		MaxIdleConnsPerHost:   1,
		DisableCompression:    true,
		DisableKeepAlives:     true,
		ResponseHeaderTimeout: 5 * time.Second,
	},
	CheckRedirect: func(req *http.Request, via []*http.Request) error {
		return fmt.Errorf("no redirects allowed")
	},
	Timeout: 10 * time.Second,
}

DefaultHTTPClient is used when no other http.Client is specified on a HTTPChecker.

Functions

func GenerateFilename 

func GenerateFilename() *string

GenerateFilename returns a filename that is ideal for storing the results file on a storage provider that relies on the filename for retrieval that is sorted by date/timeframe. It returns a string pointer to be used by the AWS SDK...

func Timestamp 

func Timestamp() int64

Timestamp returns the UTC Unix timestamp in nanoseconds.

Types

type Attempt 

type Attempt struct {
	RTT   time.Duration `json:"rtt"`
	Error string        `json:"error,omitempty"`
}

Attempt is an attempt to communicate with the endpoint.

type Attempts 

type Attempts []Attempt

Attempts is a list of Attempt that can be sorted by RTT.

func (Attempts) Len 

func (a Attempts) Len() int

func (Attempts) Less 

func (a Attempts) Less(i, j int) bool

func (Attempts) Swap 

func (a Attempts) Swap(i, j int)

type Checker 

type Checker interface {
	Check() (Result, error)
}

Checker can create a Result.

type Checkup 

type Checkup struct {
	// Checkers is the list of Checkers to use with
	// which to perform checks.
	Checkers []Checker `json:"checkers,omitempty"`

	// ConcurrentChecks is how many checks, at most, to
	// perform concurrently. Default is
	// DefaultConcurrentChecks.
	ConcurrentChecks int `json:"concurrent_checks,omitempty"`

	// Timestamp is the timestamp to force for all checks.
	// Useful if wanting to perform distributed check
	// "at the same time" even if they might actually
	// be a few milliseconds or seconds apart.
	Timestamp time.Time `json:"timestamp,omitempty"`

	// Storage is the storage mechanism for saving the
	// results of checks. Required if calling Store().
	// If Storage is also a Maintainer, its Maintain()
	// method will be called by c.CheckAndStore().
	Storage Storage `json:"storage,omitempty"`

	// Notifier is a notifier that will be passed the
	// results after checks from all checkers have
	// completed. Notifier may evaluate and choose to
	// send a notification of potential problems.
	Notifier Notifier `json:"notifier,omitempty"`
}

Checkup performs a routine checkup on endpoints or services.

func (Checkup) Check 

func (c Checkup) Check() ([]Result, error)

Check performs the health checks. An error is only returned in the case of a misconfiguration or if any one of the Checkers returns an error.

func (Checkup) CheckAndStore 

func (c Checkup) CheckAndStore() error

CheckAndStore performs health checks and immediately stores the results to the configured storage if there were no errors. Checks are not performed if c.Storage is nil. If c.Storage is also a Maintainer, Maintain() will be called if Store() is successful.

func (Checkup) CheckAndStoreEvery 

func (c Checkup) CheckAndStoreEvery(interval time.Duration) *time.Ticker

CheckAndStoreEvery calls CheckAndStore every interval. It returns the ticker that it's using so you can stop it when you don't want it to run anymore. This function does NOT block (it runs the ticker in a goroutine). Any errors are written to the standard logger. It would not be wise to set an interval lower than the time it takes to perform the checks.

func (Checkup) MarshalJSON 

func (c Checkup) MarshalJSON() ([]byte, error)

MarshalJSON marshals c into JSON with type information included on the interface values.

func (*Checkup) UnmarshalJSON 

func (c *Checkup) UnmarshalJSON(b []byte) error

UnmarshalJSON unmarshales b into c. To succeed, it requires type information for the interface values.

type Errors 

type Errors []error

Errors is an error type that concatenates multiple errors.

func (Errors) Empty 

func (e Errors) Empty() bool

Empty returns whether e has any non-nil errors in it.

func (Errors) Error 

func (e Errors) Error() string

Error returns a string containing all the errors in e.

type HTTPChecker 

type HTTPChecker struct {
	// Name is the name of the endpoint.
	Name string `json:"endpoint_name"`

	// URL is the URL of the endpoint.
	URL string `json:"endpoint_url"`

	// UpStatus is the HTTP status code expected by
	// a healthy endpoint. Default is http.StatusOK.
	UpStatus int `json:"up_status,omitempty"`

	// ThresholdRTT is the maximum round trip time to
	// allow for a healthy endpoint. If non-zero and a
	// request takes longer than ThresholdRTT, the
	// endpoint will be considered unhealthy. Note that
	// this duration includes any in-between network
	// latency.
	ThresholdRTT time.Duration `json:"threshold_rtt,omitempty"`

	// MustContain is a string that the response body
	// must contain in order to be considered up.
	// NOTE: If set, the entire response body will
	// be consumed, which has the potential of using
	// lots of memory and slowing down checks if the
	// response body is large.
	MustContain string `json:"must_contain,omitempty"`

	// MustNotContain is a string that the response
	// body must NOT contain in order to be considered
	// up. If both MustContain and MustNotContain are
	// set, they are and-ed together. NOTE: If set,
	// the entire response body will be consumed, which
	// has the potential of using lots of memory and
	// slowing down checks if the response body is large.
	MustNotContain string `json:"must_not_contain,omitempty"`

	// Attempts is how many requests the client will
	// make to the endpoint in a single check.
	Attempts int `json:"attempts,omitempty"`

	// Client is the http.Client with which to make
	// requests. If not set, DefaultHTTPClient is
	// used.
	Client *http.Client `json:"-"`
}

HTTPChecker implements a Checker for HTTP endpoints.

func (HTTPChecker) Check 

func (c HTTPChecker) Check() (Result, error)

Check performs checks using c according to its configuration. An error is only returned if there is a configuration error.

type Maintainer 

type Maintainer interface {
	Maintain() error
}

Maintainer can maintain a store of results by deleting old check files that are no longer needed or performing other required tasks.

type Notifier 

type Notifier interface {
	Notify([]Result) error
}

Notifier can notify ops or sysadmins of potential problems. A Notifier should keep state to avoid sending repeated notices more often than the admin would like.

type ProvisionInfo 

type ProvisionInfo struct {
	// The ID of a user that was created for accessing checks.
	UserID string `json:"user_id"`

	// The username of a user that was created for accessing checks.
	Username string `json:"username"`

	// The ID or name of the ID/key used to access checks. Expect
	// this value to be made public. (It should have read-only
	// access to the checks.)
	PublicAccessKeyID string `json:"public_access_key_id"`

	// The "secret" associated with the PublicAccessKeyID, but
	// expect this value to be made public. (It should provide
	// read-only access to the checks.)
	PublicAccessKey string `json:"public_access_key"`
}

ProvisionInfo contains the results of provisioning a new storage facility for check files. Its values should be used by the status page in order to obtain read-only access to the check files.

func (ProvisionInfo) String 

func (i ProvisionInfo) String() string

String returns the information in i in a human-readable format along with an important notice.

type Provisioner 

type Provisioner interface {
	Provision() (ProvisionInfo, error)
}

Provisioner is a type of storage mechanism that can provision itself for use with checkup. Provisioning need only happen once and is merely a convenience so that the user can get up and running with their status page more quickly. Presumably, the info returned from Provision should be used on the status page side of things ot access the check files (like a key pair that is used for read-only access).

type Result 

type Result struct {
	// Title is the title (or name) of the thing that was checked.
	// It should be unique, as it acts like an identifier to users.
	Title string `json:"title,omitempty"`

	// Endpoint is the URL/address/path/identifier/locator/whatever
	// of what was checked.
	Endpoint string `json:"endpoint,omitempty"`

	// Timestamp is when the check occurred; UTC UnixNano format.
	Timestamp int64 `json:"timestamp,omitempty"`

	// Times is a list of each individual check attempt.
	Times Attempts `json:"times,omitempty"`

	// ThresholdRTT is the maximum RTT that was tolerated before
	// considering performance to be degraded. Leave 0 if irrelevant.
	ThresholdRTT time.Duration `json:"threshold,omitempty"`

	// Healthy, Degraded, and Down contain the ultimate conclusion
	// about the endpoint. Exactly one of these should be true;
	// any more or less is a bug.
	Healthy  bool `json:"healthy,omitempty"`
	Degraded bool `json:"degraded,omitempty"`
	Down     bool `json:"down,omitempty"`

	// Notice contains a description of some condition of this
	// check that might have affected the result in some way.
	// For example, that the median RTT is above the threshold.
	Notice string `json:"notice,omitempty"`

	// Message is an optional message to show on the status page.
	// For example, what you're doing to fix a problem.
	Message string `json:"message,omitempty"`
}

Result is the result of a health check.

func (Result) ComputeStats 

func (r Result) ComputeStats() Stats

ComputeStats computes basic statistics about r.

func (Result) Status 

func (r Result) Status() StatusText

Status returns a text representation of the overall status indicated in r.

func (Result) String 

func (r Result) String() string

String returns a human-readable rendering of r.

type S3 

type S3 struct {
	AccessKeyID     string `json:"access_key_id"`
	SecretAccessKey string `json:"secret_access_key"`
	Region          string `json:"region,omitempty"`
	Bucket          string `json:"bucket"`

	// Check files older than CheckExpiry will be
	// deleted on calls to Maintain(). If this is
	// the zero value, no old check files will be
	// deleted.
	CheckExpiry time.Duration `json:"check_expiry,omitempty"`
}

S3 is a way to store checkup results in an S3 bucket.

func (S3) Maintain 

func (s S3) Maintain() error

Maintain deletes check files that are older than s.CheckExpiry.

func (S3) Provision 

func (s S3) Provision() (ProvisionInfo, error)

Provision creates a new IAM user in the account specified by s, and configures a bucket according to the values in s. The credentials in s must have the IAMFullAccess and AmazonS3FullAccess permissions in order to succeed.

The name of the created IAM user is "checkup-monitor-s3-public". It will have read-only permission to S3.

Provision need only be called once per status page (bucket), not once per endpoint.

func (S3) Store 

func (s S3) Store(results []Result) error

Store stores results on S3 according to the configuration in s.

type Stats 

type Stats struct {
	Total  time.Duration `json:"total,omitempty"`
	Mean   time.Duration `json:"mean,omitempty"`
	Median time.Duration `json:"median,omitempty"`
	Min    time.Duration `json:"min,omitempty"`
	Max    time.Duration `json:"max,omitempty"`
}

Stats is a type that holds information about a Result, especially its various Attempts.

type StatusText 

type StatusText string

StatusText is the textual representation of the result of a status check. 

const (
	Healthy  StatusText = "healthy"
	Degraded StatusText = "degraded"
	Down     StatusText = "down"
	Unknown  StatusText = "unknown"
)

Text representations for the status of a check.

func (StatusText) PriorityOver 

func (s StatusText) PriorityOver(other StatusText) bool

PriorityOver returns whether s has priority over other. For example, a Down status has priority over Degraded.

type Storage 

type Storage interface {
	Store([]Result) error
}

Storage can store results.

Source Files

View all Source files

Directories

Path Synopsis
cmd
checkup

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.