hoverfly

README

Dependencies without the sting

Hoverfly is a lightweight, open source service virtualization tool. Using Hoverfly, you can virtualize your application dependencies to create a self-contained development or test environment.

Hoverfly is a proxy written in Go. It can capture HTTP(s) traffic between an application under test and external services, and then replace the external services. Another powerful feature: middleware modules, where users can introduce their own custom logic. Middleware modules can be written in any language.

More information about Hoverfly and how to use it:

• https://www.specto.io/speeding-up-your-slow-dependencies/
• https://www.specto.io/service-virtualization-modifying-traffiic/
• https://www.specto.io/api-mocking-for-dev-and-e2e-testing-from-zero-to-hero/

Installation

Pre-built binary

Pre-built Hoverfly binaries are available here.

OSX, Linux & *BSD

Download a binary, set the correct permissions:

chmod +x hoverfly*

And run it:

./hoverfly*

Windows

Simply download a .exe file and run it.

Build it yourself

To set up your Go environment - look here.

This project uses Git submodules to handle Go dependencies. You must have Go > 1.5 installed, and the 'Go 1.5 Vendor Experiment' flag enabled.

export GO15VENDOREXPERIMENT=1
mkdir -p "$GOPATH/src/github.com/SpectoLabs/"
git clone https://github.com/SpectoLabs/hoverfly.git "$GOPATH/src/github.com/SpectoLabs/hoverfly"
cd "$GOPATH/src/github.com/SpectoLabs/hoverfly"
git submodule update --init --recursive

Build Hoverfly (main application file is in /cmd/hoverfly/ directory):

cd cmd/hoverfly/
go build

Set the correct permissions:

chmod +x hoverfly

And run it:

./hoverfly

Admin UI

The Hoverfly admin UI is available at http://localhost:8888/. It uses the API (as described below) to change state. It also allows you to wipe the captured requests/responses and shows the number of captured records. For other functions, such as export/import, you can use the API directly.

Hoverfly is a proxy

Configuring your application to use Hoverfly is simple. All you have to do is set the HTTP_PROXY environment variable:

 export HTTP_PROXY=http://localhost:8500/

Destination configuration

You can specify which site to capture or virtualize with a regular expression (by default, Hoverfly processes everything):

./hoverfly --destination="."

Modes (Virtualize / Capture / Synthesize / Modify)

Hoverfly has different operating modes. Each mode changes the behavior of the proxy. Based on the selected mode, Hoverfly can either capture the requests and responses, look for them in the cache, or send them directly to the middleware and respond with a payload that is generated by the middleware (more on middleware below).

Virtualize

By default, the proxy starts in virtualize mode. You can apply middleware to each response.

Capture

When capture mode is active, Hoverfly acts as a "man-in-the-middle". It makes requests on behalf of a client and records the responses. The response is then sent back to the original client.

To switch to capture mode, you can add the "--capture" flag during startup:

./hoverfly --capture

Or you can select "Capture" using the admin UI at http://localhost:8888/

Or you can use API call to change proxy state while running (see API below).

Do a curl request with proxy details:

curl http://mirage.readthedocs.org --proxy http://localhost:8500/

Synthesize

Hoverfly can create responses to requests on the fly. Synthesize mode intercepts requests (it also respects the --destination flag) and applies the supplied middleware (the user is required to supply --middleware flag, you can read more about it below). The middleware is expected to populate response payload, so Hoverfly can then reconstruct it and return it to the client. An example of a synthetic service can be found in this_repo/examples/middleware/synthetic_service/synthetic.py. You can test it by running:

./hoverfly --synthesize --middleware "../../examples/middleware/synthetic_service/synthetic.py"

Modify

Modify mode applies the selected middleware (the user is required to supply the --middleware flag - you can read more about this below) to outgoing requests and incoming responses (before they are returned to the client). It allows users to use Hoverfly without storing any data about requests. It can be used when it's difficult to - or there is little point in - describing a service, and you only need to change certain parts of the requests or responses (eg: adding authentication headers through with middleware). The example below changes the destination host to "mirage.readthedocs.org" and sets the method to "GET":

./hoverfly --modify --middleware "../../examples/middleware/modify_request/modify_request.py

HTTPS capture

Add ca.pem to your trusted certificates or turn off verification. With curl you can make insecure requests with -k:

curl https://www.bbc.co.uk --proxy http://localhost:8500 -k

API

You can access the administrator API under the default hostname of 'localhost' and port '8888':

• Recorded requests: GET http://localhost:8888/records ( curl http://localhost:8888/records )
• Wipe cache: DELETE http://localhost:8888/records ( curl -X DELETE http://localhost:8888/records )
• Get current proxy state: GET http://localhost:8888/state ( curl http://localhost:8888/state )
• Set proxy state: POST http://localhost:8888/state, where
  • body to start virtualizing: {"mode":"virtualize"}
  • body to start capturing: {"mode":"capture"}
• Exporting recorded requests to a file: curl http://localhost:8888/records > requests.json
• Importing requests from file: curl --data "@/path/to/requests.json" http://localhost:8888/records

Middleware

Hoverfly supports external middleware modules. You can write them in any language you want !. These middleware modules are expected to take standard input (stdin) and should output a structured JSON string to stdout. Payload example:

{
	"response": {
		"status": 200,
		"body": "body here",
		"headers": {
			"Content-Type": ["text/html"],
			"Date": ["Tue, 01 Dec 2015 16:49:08 GMT"],
		}
	},
	"request": {
		"path": "/",
		"method": "GET",
		"destination": "1stalphaomega.readthedocs.org",
		"query": ""
	},
	"id": "5d4f6b1d9f7c2407f78e4d4e211ec769"
}

Middleware is executed only when request is matched so for fully dynamic responses where you are generating response on the fly you can use ""--synthesize" flag.

In order to use your middleware, just add path to executable:

./hoverfly --middleware "../../examples/middleware/modify_response/modify_response.py"

You should be treating Hoverfly middleware as commands, this enables a bit more complex execution:

.hoverfly --middleware "go run ../../examples/middleware/go_example/change_to_custom_404.go"

In this example Hoverfly is also compiling Go middleware on the fly. However this increases time taken to execute middleware (and therefore to respond) so it's better to precompile middleware binaries before using them.

Python middleware example

Basic example of a Python module to change response body, set different status code and add 2 second delay:

#!/usr/bin/env python
import sys
import logging
import json
from time import sleep


logging.basicConfig(filename='middleware.log', level=logging.DEBUG)
logging.debug('Middleware is called')


def main():
    data = sys.stdin.readlines()
    # this is a json string in one line so we are interested in that one line
    payload = data[0]
    logging.debug(payload)

    payload_dict = json.loads(payload)

    payload_dict['response']['status'] = 201
    payload_dict['response']['body'] = "body was replaced by middleware"

    # now let' sleep for 2 seconds
    sleep(2)

    # returning new payload
    print(json.dumps(payload_dict))

if __name__ == "__main__":
    main()

Save this file with python extension, chmod +x it and run hoverfly:

./hoverfly --middleware "./this_file.py"

JavaScript middleware example

You can also execute JavaScript middleware using Node. Make sure that you can execute Node, you can brew install it on OSX.

Below is an example how to take data in, parse JSON, modify it and then encode it back to string and return:

#!/usr/bin/env node

process.stdin.resume();  
process.stdin.setEncoding('utf8');  
process.stdin.on('data', function(data) {
  var parsed_json = JSON.parse(data);
  // changing response
  parsed_json.response.status = 201;
  parsed_json.response.body = "body was replaced by JavaScript middleware\n";

  // stringifying JSON response
  var newJsonString = JSON.stringify(parsed_json);

  process.stdout.write(newJsonString);
});

You see, it's really easy to use it to create a synthetic service to simulate backend when you are working on the frontend side :)

How middleware interacts with different modes

Each mode is affected by middleware in a different way. Since the JSON payload has request and response structures, some middleware will not change anything. Some information about middleware behaviour:

• Capture Mode: middleware affects only outgoing requests.
• Virtualize Mode: middleware affects only responses (cache contents remain untouched).
• Synthesize Mode: middleware creates responses.
• Modify Mode: middleware affects requests and responses.

Debugging

You can supply "-v" flag to enable verbose logging.

Contributing

Contributions are welcome!

To submit a pull request you should fork the Hoverfly repository, and make your change on a feature branch of your fork. Then generate a pull request from your branch against master of the Hoverfly repository. Include in your pull request details of your change (why and how, as well as the testing you have performed). To read more about forking model, check out this link: forking workflow. Hoverfly is a new project, we will soon provide detailed roadmap.

License

Apache License version 2.0 See LICENSE for details.

(c) SpectoLabs 2015.

Documentation

Index

• Constants
• Variables
• func GetDB(name string) *bolt.DB
• func GetRandomName(n int) []byte
• func Pipeline(cmds ...*exec.Cmd) (pipeLineOutput, collectedStandardError []byte, pipeLineError error)
• func SynthesizeResponse(req *http.Request, middleware string) (*http.Response, error)
• type ActionType
• type ActionTypeHooks
  • func (hooks ActionTypeHooks) Add(hook Hook)
  • func (hooks ActionTypeHooks) Fire(ac ActionType, entry *Entry) error
• type BoltCache
  • func NewBoltDBCache(db *bolt.DB, bucket []byte) *BoltCache
  • func (c *BoltCache) CloseDB()
  • func (c *BoltCache) DeleteBucket(name []byte) (err error)
  • func (c *BoltCache) DeleteData() error
  • func (c *BoltCache) Get(key []byte) (value []byte, err error)
  • func (c *BoltCache) GetAllKeys() (keys map[string]bool, err error)
  • func (c *BoltCache) GetAllRequests() (payloads []Payload, err error)
  • func (c *BoltCache) RecordsCount() (count int, err error)
  • func (c *BoltCache) Set(key, value []byte) error
• type Cache
• type Client
• type Configuration
  • func InitSettings() *Configuration
  • func (c *Configuration) GetMode() (mode string)
  • func (c *Configuration) SetMode(mode string)
• type Constructor
  • func NewConstructor(req *http.Request, payload Payload) *Constructor
  • func (c *Constructor) ApplyMiddleware(middleware string) error
  • func (c *Constructor) ReconstructRequest() (*http.Request, error)
  • func (c *Constructor) ReconstructResponse() *http.Response
• type CounterByMode
  • func NewModeCounter() *CounterByMode
  • func (c *CounterByMode) Count(mode string)
  • func (c *CounterByMode) Flush() (h Stats)
  • func (c *CounterByMode) Init()
• type DBClient
  • func GetNewHoverfly(cfg *Configuration, cache Cache) (*goproxy.ProxyHttpServer, DBClient)
  • func (d *DBClient) AddHook(hook Hook)
  • func (d *DBClient) AllRecordsHandler(w http.ResponseWriter, req *http.Request)
  • func (d *DBClient) CurrentStateHandler(w http.ResponseWriter, req *http.Request)
  • func (d *DBClient) DeleteAllRecordsHandler(w http.ResponseWriter, req *http.Request)
  • func (d *DBClient) ImportRecordsHandler(w http.ResponseWriter, req *http.Request)
  • func (d *DBClient) RecordsCount(w http.ResponseWriter, req *http.Request)
  • func (d *DBClient) StartAdminInterface()
  • func (d *DBClient) StateHandler(w http.ResponseWriter, r *http.Request)
  • func (d *DBClient) StatsHandler(w http.ResponseWriter, req *http.Request)
  • func (d *DBClient) StatsWSHandler(w http.ResponseWriter, r *http.Request)
• type Entry
• type Hook
• type Payload
  • func ExecuteMiddleware(command string, payload Payload) (Payload, error)
  • func (p *Payload) Encode() ([]byte, error)
• type RequestContainer
  • func (r *RequestContainer) Hash() string
• type RequestDetails
• type ResponseDetails
• type Stats

Constants

const ActionTypeConfigurationChanged = "configurationChanged"

ActionTypeConfigurationChanged - default action name for identifying configuration changes

const ActionTypeRequestCaptured = "requestCaptured"

ActionTypeRequestCaptured - default action type name for identifying

const ActionTypeWipeDB = "wipeDatabase"

ActionTypeWipeDB - default action type for wiping database

const CaptureMode = "capture"

CaptureMode - requests are captured and stored in cache

const DefaultAdminPort = "8888"

DefaultAdminPort - default admin interface port

const DefaultDatabaseName = "requests.db"

DefaultDatabaseName - default database name that will be created or used by Hoverfly

const DefaultPort = "8500"

DefaultPort - default proxy port

const ModifyMode = "modify"

ModifyMode - middleware is applied to outgoing and incoming traffic

const RequestsBucketName = "rqbucket"

RequestsBucketName - default name for BoltDB bucket

const SynthesizeMode = "synthesize"

SynthesizeMode - all requests are sent to middleware to create response

const VirtualizeMode = "virtualize"

VirtualizeMode - default mode when Hoverfly looks for captured requests to respond

Variables

var TestDB *bolt.DB

TestDB - holds connection to database during tests

Functions

func GetDB

func GetDB(name string) *bolt.DB

GetDB - returns open BoltDB database with read/write permissions or goes down in flames if something bad happends

func GetRandomName

func GetRandomName(n int) []byte

GetRandomName - provides random name for buckets. Each test case gets it's own bucket

func Pipeline

func Pipeline(cmds ...*exec.Cmd) (pipeLineOutput, collectedStandardError []byte, pipeLineError error)

Pipeline - to provide input to the pipeline, assign an io.Reader to the first's Stdin.

func SynthesizeResponse

func SynthesizeResponse(req *http.Request, middleware string) (*http.Response, error)

SynthesizeResponse calls middleware to populate response data, nothing gets pass proxy

Types

type ActionType

type ActionType string

ActionType - action type can be things such as "RequestCaptured", "GotResponse" - anything

type ActionTypeHooks

type ActionTypeHooks map[ActionType][]Hook

ActionTypeHooks type for storing the hooks

func (ActionTypeHooks) Add

func (hooks ActionTypeHooks) Add(hook Hook)

Add a hook

func (ActionTypeHooks) Fire

func (hooks ActionTypeHooks) Fire(ac ActionType, entry *Entry) error

Fire all the hooks for the passed ActionType

type BoltCache

type BoltCache struct {
	DS             *bolt.DB
	RequestsBucket []byte
}

Cache - provides access to BoltDB and holds current bucket name

func NewBoltDBCache

func NewBoltDBCache(db *bolt.DB, bucket []byte) *BoltCache

NewBoltDBCache - returns new BoltCache instance

func (*BoltCache) CloseDB

func (c *BoltCache) CloseDB()

CloseDB - closes database

func (*BoltCache) DeleteBucket

func (c *BoltCache) DeleteBucket(name []byte) (err error)

DeleteBucket - deletes bucket with all saved data

func (*BoltCache) DeleteData

func (c *BoltCache) DeleteData() error

DeleteData - deletes bucket with all saved data

func (*BoltCache) Get

func (c *BoltCache) Get(key []byte) (value []byte, err error)

Get - searches for given key in the cache and returns value if found

func (*BoltCache) GetAllKeys

func (c *BoltCache) GetAllKeys() (keys map[string]bool, err error)

GetAllKeys - gets all current keys

func (*BoltCache) GetAllRequests

func (c *BoltCache) GetAllRequests() (payloads []Payload, err error)

GetAllRequests - returns all captured requests/responses

func (*BoltCache) RecordsCount

func (c *BoltCache) RecordsCount() (count int, err error)

RecordsCount - returns records count

func (*BoltCache) Set

func (c *BoltCache) Set(key, value []byte) error

Set - saves given key and value pair to cache

type Cache

type Cache interface {
	Set(key, value []byte) error
	Get(key []byte) ([]byte, error)
	GetAllRequests() ([]Payload, error)
	RecordsCount() (int, error)
	DeleteData() error
	GetAllKeys() (map[string]bool, error)
	CloseDB()
}

Cache - cache interface used to store and retrieve request/response payloads

type Client

type Client struct {
	HTTPClient *http.Client
}

Client structure to be injected into functions to perform HTTP calls

type Configuration

type Configuration struct {
	AdminPort    string
	ProxyPort    string
	Mode         string
	Destination  string
	Middleware   string
	DatabaseName string
	Verbose      bool
	Development  bool
	// contains filtered or unexported fields
}

Configuration - initial structure of configuration

func InitSettings

func InitSettings() *Configuration

InitSettings gets and returns initial configuration from env variables or sets defaults

func (*Configuration) GetMode

func (c *Configuration) GetMode() (mode string)

GetMode - provides safe way to get current mode

func (*Configuration) SetMode

func (c *Configuration) SetMode(mode string)

SetMode - provides safe way to set new mode

type Constructor

type Constructor struct {
	// contains filtered or unexported fields
}

Constructor - holds information about original request (which is needed to create response and also holds payload

func NewConstructor

func NewConstructor(req *http.Request, payload Payload) *Constructor

NewConstructor - returns constructor instance

func (*Constructor) ApplyMiddleware

func (c *Constructor) ApplyMiddleware(middleware string) error

ApplyMiddleware - activates given middleware, middleware should be passed as string to executable, can be full path.

func (*Constructor) ReconstructRequest

func (c *Constructor) ReconstructRequest() (*http.Request, error)

ReconstructRequest replaces original request with details provided in Constructor Payload.Request

func (*Constructor) ReconstructResponse

func (c *Constructor) ReconstructResponse() *http.Response

ReconstructResponse changes original response with details provided in Constructor Payload.Response

type CounterByMode

type CounterByMode struct {
	// contains filtered or unexported fields
}

CounterByMode - container for mode counters, registry and flush interval

func NewModeCounter

func NewModeCounter() *CounterByMode

NewModeCounter - returns new counter instance

func (*CounterByMode) Count

func (c *CounterByMode) Count(mode string)

Count - counts requests based on mode

func (*CounterByMode) Flush

func (c *CounterByMode) Flush() (h Stats)

Flush gets current metrics from stats registry

func (*CounterByMode) Init

func (c *CounterByMode) Init()

Init initializes logging

type DBClient

type DBClient struct {
	Cache   Cache
	HTTP    *http.Client
	Cfg     *Configuration
	Counter *CounterByMode
	Hooks   ActionTypeHooks
}

DBClient provides access to cache, http client and configuration

func GetNewHoverfly

func GetNewHoverfly(cfg *Configuration, cache Cache) (*goproxy.ProxyHttpServer, DBClient)

GetNewHoverfly returns a configured ProxyHttpServer and DBClient

func (*DBClient) AddHook

func (d *DBClient) AddHook(hook Hook)

AddHook - adds a hook to DBClient

func (*DBClient) AllRecordsHandler

func (d *DBClient) AllRecordsHandler(w http.ResponseWriter, req *http.Request)

AllRecordsHandler returns JSON content type http response

func (*DBClient) CurrentStateHandler

func (d *DBClient) CurrentStateHandler(w http.ResponseWriter, req *http.Request)

CurrentStateHandler returns current state

func (*DBClient) DeleteAllRecordsHandler

func (d *DBClient) DeleteAllRecordsHandler(w http.ResponseWriter, req *http.Request)

DeleteAllRecordsHandler - deletes all captured requests

func (*DBClient) ImportRecordsHandler

func (d *DBClient) ImportRecordsHandler(w http.ResponseWriter, req *http.Request)

ImportRecordsHandler - accepts JSON payload and saves it to cache

func (*DBClient) RecordsCount

func (d *DBClient) RecordsCount(w http.ResponseWriter, req *http.Request)

RecordsCount returns number of captured requests as a JSON payload

func (*DBClient) StartAdminInterface

func (d *DBClient) StartAdminInterface()

StartAdminInterface - starts admin interface web server

func (*DBClient) StateHandler

func (d *DBClient) StateHandler(w http.ResponseWriter, r *http.Request)

StateHandler handles current proxy state

func (*DBClient) StatsHandler

func (d *DBClient) StatsHandler(w http.ResponseWriter, req *http.Request)

StatsHandler - returns current stats about Hoverfly (request counts, record count)

func (*DBClient) StatsWSHandler

func (d *DBClient) StatsWSHandler(w http.ResponseWriter, r *http.Request)

StatsWSHandler - returns current stats about Hoverfly (request counts, record count) through the websocket

type Entry

type Entry struct {
	// Contains encoded data
	Data []byte

	// Time at which the action entry was fired
	Time time.Time

	ActionType ActionType

	// Message, can carry additional information
	Message string
}

Entry - holds information about action, based on action type - other clients will be able to decode the data field.

type Hook

type Hook interface {
	ActionTypes() []ActionType
	Fire(*Entry) error
}

Hook - an interface to add dynamic hooks to extend functionality

type Payload

type Payload struct {
	Response ResponseDetails `json:"response"`
	Request  RequestDetails  `json:"request"`
	ID       string          `json:"id"`
}

Payload structure holds request and response structure

func ExecuteMiddleware

func ExecuteMiddleware(command string, payload Payload) (Payload, error)

ExecuteMiddleware - takes command (middleware string) and payload, which is passed to middleware

func (*Payload) Encode

func (p *Payload) Encode() ([]byte, error)

Encode method encodes all exported Payload fields to bytes

type RequestContainer

type RequestContainer struct {
	Details RequestDetails
}

RequestContainer holds structure for request

func (*RequestContainer) Hash

func (r *RequestContainer) Hash() string

Hash returns unique hash key for request

type RequestDetails

type RequestDetails struct {
	Path        string              `json:"path"`
	Method      string              `json:"method"`
	Destination string              `json:"destination"`
	Scheme      string              `json:"scheme"`
	Query       string              `json:"query"`
	Body        string              `json:"body"`
	RemoteAddr  string              `json:"remoteAddr"`
	Headers     map[string][]string `json:"headers"`
}

RequestDetails stores information about request, it's used for creating unique hash and also as a payload structure

type ResponseDetails

type ResponseDetails struct {
	Status  int                 `json:"status"`
	Body    string              `json:"body"`
	Headers map[string][]string `json:"headers"`
}

ResponseDetails structure hold response body from external service, body is not decoded and is supposed to be bytes, however headers should provide all required information for later decoding by the client.

type Stats

type Stats struct {
	Counters    map[string]int64   `json:"counters"`
	Gauges      map[string]int64   `json:"gauges,omitempty"`
	GaugesFloat map[string]float64 `json:"gaugesFloat,omitempty"`
}

Stats - holds information about various system metrics like requests counts