README

Fault

go.dev reference

The fault package provides go http middleware that makes it easy to inject faults into your service. Use the fault package to reject incoming requests, respond with an HTTP error, inject latency into a percentage of your requests, or inject any of your own custom faults.

Features

The fault package works through standard go http middleware. You first create an Injector, which is a middleware with the code to be run on injection. Then you wrap that Injector in a Fault which handles logic about when to run your Injector.

There are currently three kinds of injectors: SlowInjector, ErrorInjector, and RejectInjector. Each of these injectors can be configured thorugh a Fault to run on a small percent of your requests. You can also configure the Fault to blocklist/allowlist certain paths.

See the usage section below for an example of how to get started and the godoc for further documentation.

Limitations

This package is useful for safely testing failure scenarios in go services that can make use of net/http handlers/middleware.

One common failure scenario that we cannot perfectly simulate is dropped requests. The RejectInjector will always return immiediately to the user, but in many cases requests can be dropped without ever sending a response. The best way to simulate this scenario using the fault packge is to chain a SlowInjector with a very long wait time in front of an eventual RejectInjector.

Status

This project is in a stable and supported state. There are no plans to introduce significant new features however we welcome and encourage any ideas and contributions from the community. Contributions should follow the guidelines in our CONTRIBUTING.md.

Usage

// main.go
package main

import (
        "net/http"

        "github.com/github/go-fault"
)

var mainHandler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        http.Error(w, testHandlerBody, testHandlerCode)
})

func main() {
        slowInjector, _ := fault.NewSlowInjector(time.Second * 2)
        slowFault, _ := fault.NewFault(slowInjector,
                fault.WithEnabled(true),
                fault.WithParticipation(0.25),
                fault.WithPathBlocklist([]string{"/ping", "/health"}),
        )

        // Add 2 seconds of latency to 25% of our requests
        handlerChain := slowFault.Handler(mainHandler)

        http.ListenAndServe("127.0.0.1:3000", handlerChain)
}

Development

This package uses standard go tooling for testing and development. The go language is all you need to contribute. Tests use the popular testify/assert which will be downloaded automatically the first time you run tests. GitHub Actions will also run a linter using golangci-lint after you push. You can also download the linter and use golangci-lint run to run locally.

Testing

The fault package has extensive tests that are run in GitHub Actions on every push.

You can also run tests locally:

$ go test -v -cover -race ./...
[...]
PASS
coverage: 100.0% of statements
ok      github.com/github/go-fault      0.575s

Benchmarks

The fault package is safe to leave implemented even when you are not running a fault injection. While the fault is disabled there is negligible performance degradation compared to removing the package from the request path. While enabled there may be minor performance differences, but this will only be the case while you are already injecting faults.

Benchmarks are provided to compare without faults, with faults disabled, and with faults enabled. Benchmarks are uploaded as artifacts in GitHub Actions and you can download them from any Validate Workflow.

You can also run benchmarks locally (example output):

$ go test -run=XXX -bench=.
goos: darwin
goarch: amd64
pkg: github.com/github/go-fault
BenchmarkNoFault-8                        684826              1734 ns/op
BenchmarkFaultDisabled-8                  675291              1771 ns/op
BenchmarkFaultErrorZeroPercent-8          667903              1823 ns/op
BenchmarkFaultError100Percent-8           663661              1833 ns/op
PASS
ok      github.com/github/go-fault      8.814s

Maintainers

@lingrino

License

This project is licensed under the MIT License.

Expand ▾ Collapse ▴

Documentation

Overview

Package fault provides standard http middleware for fault injection in go.

Basics

Use the fault package to inject faults into the http request path of your service. Faults work by modifying and/or delaying your service's http responses. Place the Fault middleware high enough in the chain that it can act quickly, but after any other middlewares that should complete before fault injection (auth, redirects, etc...).

The type and severity of injected faults is controlled by options passed to NewFault(Injector, Options). NewFault must be passed an Injector, which is an interface that holds the actual fault injection code in Injector.Handler. The Fault wraps Injector.Handler in another Fault.Handler that applies generic Fault logic (such as what % of requests to run the Injector on) to the Injector.

Make sure you use the NewFault() and NewTypeInjector() constructors to create valid Faults and Injectors.

Injectors

There are three main Injectors provided by the fault package:

fault.RejectInjector
fault.ErrorInjector
fault.SlowInjector

RejectInjector

Use fault.RejectInjector to immediately return an empty response. For example, a curl for a rejected response will produce:

$ curl https://github.com
curl: (52) Empty reply from server

ErrorInjector

Use fault.ErrorInjector to immediately return a valid http status code of your choosing along with the standard HTTP response body for that code. For example, you can return a 200, 301, 418, 500, or any other valid status code to test how your clients respond to different statuses. Pass the WithStatusText() option to customize the response text.

SlowInjector

Use fault.SlowInjector to wait a configured time.Duration before proceeding with the request. For example, you can use the SlowInjector to add a 10ms delay to your requests.

RandomInjector

Use fault.RandomInjector to randomly choose one of the above faults to inject. Pass a list of Injector to fault.NewRandomInjector and when RandomInjector is evaluated it will randomly run one of the injectors that you passed.

Combining Faults

It is easy to combine any of the Injectors into a chained action. There are two ways you might want to combine Injectors.

First, you can create separate Faults for each Injector that are sequential but independent of each other. For example, you can chain Faults such that 1% of requests will return a 500 error and another 1% of requests will be rejected.

Second, you might want to combine Faults such that 1% of requests will be slowed for 10ms and then rejected. You want these Faults to depend on each other. For this use the special ChainInjector, which consolidates any number of Injectors into a single Injector that runs each of the provided Injectors sequentially. When you add the ChainInjector to a Fault the entire chain will always execute together.

Allowing & Blocking Paths

The NewFault() constructor has WithPathBlocklist() and WithPathAllowlist() options. Any path you include in the PathBlocklist will never have faults run against it. With PathAllowlist, if you provide a non-empty list then faults will not be run against any paths except those specified in PathAllowlist. The PathBlocklist take priority over the PathAllowlist, a path in both lists will never have a fault run against it. The paths that you include must match exactly the path in req.URL.Path, including leading and trailing slashes.

Simmilarly, you may also use WithHeaderBlocklist() and WithHeaderAllowlist() to block or allow faults based on a map of header keys to values. These lists behave in the same way as the path allowlists and blocklists except that they operate on headers. Header equality is determined using http.Header.Get(key) which automatically canonicalizes your keys and does not support multi-value headers. Keep these limitations in mind when working with header allowlists and blocklists.

Specifying very large lists of paths or headers may cause memory or performance issues. If you're running into these problems you should instead consider using your http router to enable the middleware on only a subset of your routes.

Custom Injectors

The fault package provides an Injector interface and you can satisfy that interface to provide your own Injector. Use custom injectors to add additional logic to the package-provided injectors or to create your own completely new Injector that can still be managed by a Fault.

Reporter

The package provides a Reporter interface that can be added to Faults and Injectors using the WithReporter option. A Reporter will receive events when the state of the Injector changes. For example, Reporter.Report(InjectorName, StateStarted) is run at the beginning of all Injectors. The Reporter is meant to be provided by the consumer of the package and integrate with services like stats and logging. The default Reporter throws away all events.

Random Seeds

By default all randomness is seeded with defaultRandSeed(1), the same default as math/rand. This helps you reproduce any errors you see when running an Injector. If you prefer, you can also customize the seed passing WithRandSeed() to NewFault and NewRandomInjector.

Custom Injector Functions

Some Injectors support customizing the functions they use to run their injections. You can take advantage of these options to add your own logic to an existing Injector instead of creating your own. For example, modify the SlowInjector function to slow in a rancom distribution instead of for a fixed duration. Be careful when you use these options that your return values fall within the same range of values expected by the default functions to avoid panics or other undesirable begavior.

Customize the function a Fault uses to determine participation (default: rand.Float32) by passing WithRandFloat32Func() to NewFault().

Customize the function a RandomInjector uses to choose which injector to run (default: rand.Intn) by passing WithRandIntFunc() to NewRandomInjector().

Customize the function a SlowInjector uses to wait (default: time.Sleep) by passing WithSlowFunc() to NewSlowInjector().

Configuration

All configuration for the fault package is done through options passed to NewFault and NewInjector. There is no other way to manage configuration for the package. It is up to the user of the fault package to manage how the options are generated. Common options are feature flags, environment variables, or code changes in deploys.

Example

    Example is a package-level documentation example.

    Output:
    
    500
    Internal Server Error
    

    Index

    Examples

    Constants

    This section is empty.

    Variables

    View Source
    var (
    	// ErrNilInjector when a nil Injector is passed.
    	ErrNilInjector = errors.New("injector cannot be nil")
    	// ErrInvalidPercent when a percent is outside of [0.0,1.0).
    	ErrInvalidPercent = errors.New("percent must be 0.0 <= percent <= 1.0")
    )
    View Source
    var (
    	// ErrInvalidHTTPCode when an invalid status code is provided.
    	ErrInvalidHTTPCode = errors.New("not a valid http status code")
    )

    Functions

    This section is empty.

    Types

    type ChainInjector

    type ChainInjector struct {
    	// contains filtered or unexported fields
    }

      ChainInjector combines many Injectors into a single Injector that runs them in order.

      func NewChainInjector

      func NewChainInjector(is []Injector, opts ...ChainInjectorOption) (*ChainInjector, error)

        NewChainInjector combines many Injectors into a single Injector that runs them in order.

        Example

          ExampleNewChainInjector shows how to create a new ChainInjector.

          Output:
          
          <nil>
          

          func (*ChainInjector) Handler

          func (i *ChainInjector) Handler(next http.Handler) http.Handler

            Handler executes ChainInjector.middlewares in order and then returns.

            type ChainInjectorOption

            type ChainInjectorOption interface {
            	// contains filtered or unexported methods
            }

              ChainInjectorOption configures a ChainInjector.

              type ErrorInjector

              type ErrorInjector struct {
              	// contains filtered or unexported fields
              }

                ErrorInjector responds with an http status code and message.

                func NewErrorInjector

                func NewErrorInjector(code int, opts ...ErrorInjectorOption) (*ErrorInjector, error)

                  NewErrorInjector returns an ErrorInjector that reponds with a status code.

                  Example

                    ExampleNewErrorInjector shows how to create a new ErrorInjector.

                    Output:
                    
                    <nil>
                    

                    func (*ErrorInjector) Handler

                    func (i *ErrorInjector) Handler(next http.Handler) http.Handler

                      Handler responds with the configured status code and text.

                      type ErrorInjectorOption

                      type ErrorInjectorOption interface {
                      	// contains filtered or unexported methods
                      }

                        ErrorInjectorOption configures an ErrorInjector.

                        func WithStatusText

                        func WithStatusText(t string) ErrorInjectorOption

                          WithStatusText sets custom status text to write.

                          type Fault

                          type Fault struct {
                          	// contains filtered or unexported fields
                          }

                            Fault combines an Injector with options on when to use that Injector.

                            func NewFault

                            func NewFault(i Injector, opts ...Option) (*Fault, error)

                              NewFault sets/validates the Injector and Options and returns a usable Fault.

                              Example

                                ExampleNewFault shows how to create a new Fault.

                                Output:
                                
                                <nil>
                                
                                Example (Allowlist)

                                  ExampleNewFault_allowlist shows how to create a new Fault with a path/header allowlist.

                                  Output:
                                  
                                  <nil>
                                  
                                  Example (Blocklist)

                                    ExampleNewFault_blocklist shows how to create a new Fault with a path/header blocklist.

                                    Output:
                                    
                                    <nil>
                                    

                                    func (*Fault) Handler

                                    func (f *Fault) Handler(next http.Handler) http.Handler

                                      Handler determines if the Injector should execute and runs it if so.

                                      type Injector

                                      type Injector interface {
                                      	Handler(next http.Handler) http.Handler
                                      }

                                        Injector are added to Faults and run as middleware in a request.

                                        type InjectorState

                                        type InjectorState int

                                          InjectorState represents the states an injector can be in.

                                          const (
                                          	StateStarted InjectorState = iota + 1
                                          	StateFinished
                                          	StateSkipped
                                          )

                                          type NoopReporter

                                          type NoopReporter struct{}

                                            NoopReporter is a reporter that does nothing.

                                            func NewNoopReporter

                                            func NewNoopReporter() *NoopReporter

                                              NewNoopReporter returns a new NoopReporter.

                                              func (*NoopReporter) Report

                                              func (r *NoopReporter) Report(name string, state InjectorState)

                                                Report does nothing.

                                                type Option

                                                type Option interface {
                                                	// contains filtered or unexported methods
                                                }

                                                  Option configures a Fault.

                                                  func WithEnabled

                                                  func WithEnabled(e bool) Option

                                                    WithEnabled sets if the Fault should evaluate.

                                                    func WithHeaderAllowlist

                                                    func WithHeaderAllowlist(allowlist map[string]string) Option

                                                      WithHeaderAllowlist is, if set, a map of header keys to values of the only headers that the Injector will run against.

                                                      func WithHeaderBlocklist

                                                      func WithHeaderBlocklist(blocklist map[string]string) Option

                                                        WithHeaderBlocklist is a map of header keys to values that the Injector will not run against.

                                                        func WithParticipation

                                                        func WithParticipation(p float32) Option

                                                          WithParticipation sets the percent of requests that run the Injector. 0.0 <= p <= 1.0.

                                                          func WithPathAllowlist

                                                          func WithPathAllowlist(allowlist []string) Option

                                                            WithPathAllowlist is, if set, a list of the only paths that the Injector will run against.

                                                            func WithPathBlocklist

                                                            func WithPathBlocklist(blocklist []string) Option

                                                              WithPathBlocklist is a list of paths that the Injector will not run against.

                                                              func WithRandFloat32Func

                                                              func WithRandFloat32Func(f func() float32) Option

                                                                WithRandFloat32Func sets the function that will be used to randomly get our float value. Default rand.Float32. Always returns a float32 between [0.0,1.0) to avoid errors.

                                                                type RandSeedOption

                                                                type RandSeedOption interface {
                                                                	Option
                                                                	RandomInjectorOption
                                                                }

                                                                  RandSeedOption configures things that can set a random seed.

                                                                  func WithRandSeed

                                                                  func WithRandSeed(s int64) RandSeedOption

                                                                    WithRandSeed sets the rand.Rand seed for this struct.

                                                                    type RandomInjector

                                                                    type RandomInjector struct {
                                                                    	// contains filtered or unexported fields
                                                                    }

                                                                      RandomInjector combines many Injectors into a single Injector that runs one randomly.

                                                                      func NewRandomInjector

                                                                      func NewRandomInjector(is []Injector, opts ...RandomInjectorOption) (*RandomInjector, error)

                                                                        NewRandomInjector combines many Injectors into a single Injector that runs one randomly.

                                                                        Example

                                                                          ExampleNewChainInjector shows how to create a new RandomInjector.

                                                                          Output:
                                                                          
                                                                          <nil>
                                                                          

                                                                          func (*RandomInjector) Handler

                                                                          func (i *RandomInjector) Handler(next http.Handler) http.Handler

                                                                            Handler executes a random Injector from RandomInjector.middlewares.

                                                                            type RandomInjectorOption

                                                                            type RandomInjectorOption interface {
                                                                            	// contains filtered or unexported methods
                                                                            }

                                                                              RandomInjectorOption configures a RandomInjector.

                                                                              func WithRandIntFunc

                                                                              func WithRandIntFunc(f func(int) int) RandomInjectorOption

                                                                                WithRandIntFunc sets the function that will be used to randomly get an int. Default rand.Intn. Always returns an integer between [0,n) to avoid panics.

                                                                                type RejectInjector

                                                                                type RejectInjector struct {
                                                                                	// contains filtered or unexported fields
                                                                                }

                                                                                  RejectInjector sends back an empty response.

                                                                                  func NewRejectInjector

                                                                                  func NewRejectInjector(opts ...RejectInjectorOption) (*RejectInjector, error)

                                                                                    NewRejectInjector returns a RejectInjector.

                                                                                    Example

                                                                                      ExampleNewRejectInjector shows how to create a new RejectInjector.

                                                                                      Output:
                                                                                      
                                                                                      <nil>
                                                                                      

                                                                                      func (*RejectInjector) Handler

                                                                                      func (i *RejectInjector) Handler(next http.Handler) http.Handler

                                                                                        Handler rejects the request, returning an empty response.

                                                                                        type RejectInjectorOption

                                                                                        type RejectInjectorOption interface {
                                                                                        	// contains filtered or unexported methods
                                                                                        }

                                                                                          RejectInjectorOption configures a RejectInjector.

                                                                                          type Reporter

                                                                                          type Reporter interface {
                                                                                          	Report(name string, state InjectorState)
                                                                                          }

                                                                                            Reporter receives events from faults to use for logging, stats, and other custom reporting.

                                                                                            type ReporterOption

                                                                                            type ReporterOption interface {
                                                                                            	RejectInjectorOption
                                                                                            	ErrorInjectorOption
                                                                                            	SlowInjectorOption
                                                                                            }

                                                                                              ReporterOption configures structs that accept a Reporter.

                                                                                              func WithReporter

                                                                                              func WithReporter(r Reporter) ReporterOption

                                                                                                WithReporter sets the Reporter.

                                                                                                type SlowInjector

                                                                                                type SlowInjector struct {
                                                                                                	// contains filtered or unexported fields
                                                                                                }

                                                                                                  SlowInjector waits and then continues the request.

                                                                                                  func NewSlowInjector

                                                                                                  func NewSlowInjector(d time.Duration, opts ...SlowInjectorOption) (*SlowInjector, error)

                                                                                                    NewSlowInjector returns a SlowInjector.

                                                                                                    Example

                                                                                                      ExampleNewSlowInjector shows how to create a new SlowInjector.

                                                                                                      Output:
                                                                                                      
                                                                                                      <nil>
                                                                                                      

                                                                                                      func (*SlowInjector) Handler

                                                                                                      func (i *SlowInjector) Handler(next http.Handler) http.Handler

                                                                                                        Handler runs i.slowF to wait the set duration and then continues.

                                                                                                        type SlowInjectorOption

                                                                                                        type SlowInjectorOption interface {
                                                                                                        	// contains filtered or unexported methods
                                                                                                        }

                                                                                                          SlowInjectorOption configures a SlowInjector.

                                                                                                          func WithSlowFunc

                                                                                                          func WithSlowFunc(f func(t time.Duration)) SlowInjectorOption

                                                                                                            WithSlowFunc sets the function that will be used to wait the time.Duration.