recorder

README

recorder

• Overview
  • Example usage
  • Modes
  • Filters
    • Remove header from request
    • Remove header from response
    • Custom
  • Prior art
  • License

Overview

The recorder is a small helper package, primarily intended to help with unit tests. It is capable of recording and replaying traffic, avoiding real network requests.

In the default mode, responses are read from disk, allowing the network roundtrip to be avoided. This can be useful for a couple reasons:

• Faster
• Works offline
• Reduces side-effects on called APIs (such as when testing cloud service provider endpoints)

In addition, with the Passthrough mode, requests can be recorded an asserted in unit tests.

Unless the mode is set to Passthrough, the request-response is recorded in a yml on disk, such as:

# request 0
# timestamp 2019-04-30 10:02:04 +0000 UTC
# roundtrip 398ms
request:
  method: POST
  url: https://jsonplaceholder.typicode.com/posts
  headers:
    Content-Type: application/json
response:
  status_code: 201
  headers:
    Cache-Control: no-cache
    Content-Length: '69'
    Content-Type: application/json; charset=utf-8
    Date: Tue, 30 Apr 2019 10:02:04 GMT
    Location: http://jsonplaceholder.typicode.com/posts/101
    Pragma: no-cache
  body: |-
    {
      "title": "hello",
      "body": "world",
      "userId": 1,
      "id": 101
    }

Example usage

// Create a new recorder.
// Data will be saved in testdata/example.yml
rec := recorder.New("testdata/example")

// Create HTTP client with recorder transport
cli := &http.Client{
    Transport: rec,
}

// Perform a request
resp, err := cli.Get("https://jsonplaceholder.typicode.com/posts/1")
if err != nil {
    log.Fatal(err)
}

// Response is only done if required
b, err := httputil.DumpResponse(resp, true)
if err != nil {
    log.Fatal(err)
}
fmt.Println(string(b))

Modes

Modes allow granular control of behavior.

Mode	Behavior
Auto	Perform network requests if no stored file exists
ReplayOnly	Do not allow network traffic, only return stored files
Record	Always perform request and overwrite existing files
Passthrough	No files are saved on disk but requests can be retrieved with Lookup()

If no mode is set, Auto is used.

The Passthrough mode disabled loading and saving files but can be useful for asserting if the expected requests were made in tests.

Filters

Filters allow removing sensitive data from the saved files.

The filters are executed after the request but before saving files to disk.

Remove header from request

This will remove the Authorization header from the request:

rec := recorder.New("testdata/private-api", recorder.RemoveRequestHeader("Authorization"))

cli := &http.Client{
    Transport: rec,
}

req, _ := http.NewRequest("https://example.com", "application/json", strings.NewReader("{}"))
req.Header.Add("Authorization", "abcdef")

_, err := cli.Do(req)
if err != nil {
    log.Fatal(err)
}

// Authorization header is not saved to disk

Remove header from response

This will remove the Set-Cookie header from the response:

rec := recorder.New("testdata/private-api", recorder.RemoveResponseHeader("Set-Cookie"))

cli := &http.Client{
    Transport: rec,
}

_, err := cli.Get("https://example.com")
if err != nil {
    log.Fatal(err)
}

// The saved file will not contain the Set-Cookie header that was set by the server.

Custom

In addition to the built in filters, custom filters can be implemented by passing functions with a signature func (entry *recorder.Entry) {}.

rec := recorder.New("testdata/request-header", func(e *recorder.Entry) {
    // Modify e.Request and e.Response
})

cli := &http.Client{
    Transport: rec,
}

_, err := cli.Get("https://example.com")
if err != nil {
    log.Fatal(err)
}

Prior art

This library is inspired by:

• https://github.com/dnaeon/go-vcr
• https://github.com/ad2games/vcr-go
• https://github.com/nock/nock#recording

The reason for writing this library is primarily flexiblity in setting the mode and different API (no VCR references).

License

MIT

Documentation

Overview

Package recorder provides an HTTP record/replay transport.

The primary use-case is for tests where HTTP requests are sent and can be replayed without needing to reach out to the network. The Recorder is configurable to always perform request, never perform requests, or auto, where requests are performed when no existing entry exists.

Example

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httputil"

	"github.com/akupila/recorder"
)

func main() {
	// Create a new recorder.
	// Data will be saved in testdata/example.yml
	rec := recorder.New("testdata/example")

	// Create HTTP client with recorder transport
	cli := &http.Client{
		Transport: rec,
	}

	// Perform a request
	resp, err := cli.Get("https://jsonplaceholder.typicode.com/posts/1")
	if err != nil {
		log.Fatal(err)
	}

	// Response is only done if required
	b, err := httputil.DumpResponse(resp, true)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(b))
}

Index

• type Entry
• type Filter
  • func RemoveRequestHeader(name string) Filter
  • func RemoveResponseHeader(name string) Filter
• type Mode
• type NoRequestError
  • func (e NoRequestError) Error() string
• type OncePerCall
  • func (s *OncePerCall) Select(entries []Entry, req *http.Request) (Entry, bool)
• type Recorder
  • func New(filename string, filters ...Filter) *Recorder
  • func (r *Recorder) Lookup(method, url string) (Entry, bool)
  • func (r *Recorder) RoundTrip(req *http.Request) (*http.Response, error)
• type Request
• type Response
• type Selector

Examples

• Package
• Filter (Custom)
• NoRequestError
• RemoveRequestHeader
• RemoveResponseHeader

Types

type Entry

type Entry struct {
	Request  *Request  `yaml:"request"`
	Response *Response `yaml:"response"`
}

An Entry is a single recorded request-response entry.

type Filter

type Filter func(entry *Entry)

A Filter modifies the entry before it is saved to disk.

Filters are applied after the actual request, with the primary purpose being to remove sensitive data from the saved file.

Example (Custom)

package main

import (
	"log"
	"net/http"

	"github.com/akupila/recorder"
)

func main() {
	rec := recorder.New("testdata/request-header", func(e *recorder.Entry) {
		// Modify e.Request and e.Response
	})

	cli := &http.Client{
		Transport: rec,
	}

	_, err := cli.Get("https://example.com")
	if err != nil {
		log.Fatal(err)
	}
}

func RemoveRequestHeader

func RemoveRequestHeader(name string) Filter

RemoveRequestHeader removes a header with the given name from the request. The name of the header is case-sensitive.

Example

package main

import (
	"log"
	"net/http"
	"strings"

	"github.com/akupila/recorder"
)

func main() {
	rec := recorder.New("testdata/request-header", recorder.RemoveRequestHeader("Authorization"))

	cli := &http.Client{
		Transport: rec,
	}

	req, _ := http.NewRequest("https://example.com", "application/json", strings.NewReader("{}"))
	req.Header.Add("Authorization", "abcdef")

	_, err := cli.Do(req)
	if err != nil {
		log.Fatal(err)
	}

	// Authorization header is not saved to disk
}

func RemoveResponseHeader

func RemoveResponseHeader(name string) Filter

RemoveResponseHeader removes a header with the given name from the response. The name of the header is case-sensitive.

Example

package main

import (
	"log"
	"net/http"

	"github.com/akupila/recorder"
)

func main() {
	rec := recorder.New("testdata/secret", recorder.RemoveResponseHeader("Set-Cookie"))

	cli := &http.Client{
		Transport: rec,
	}

	_, err := cli.Get("https://example.com")
	if err != nil {
		log.Fatal(err)
	}

	// The saved file will not contain the Set-Cookie header that was set by the server.
}

type Mode

type Mode int

Mode controls the mode of the recorder.

const (
	// Auto reads requests from disk if a recording exists. If one does not
	// exist, the request is performed and results saved to disk.
	Auto Mode = iota

	// ReplayOnly only allows replaying from disk without network traffic.
	// If a recorded session does not exist, NoRequestError is returned.
	ReplayOnly

	// Record records all traffic even if an existing entry exists.
	// The new requests & responses overwrite any existing ones.
	Record

	// Passthrough disables the recorder and passes through all traffic
	// directly to client. Responses are not recorded to disk but can be
	// retrieved from the with Lookup().
	Passthrough
)

Possible values:

type NoRequestError

type NoRequestError struct{ Request *http.Request }

NoRequestError is returned when the recorder mode is ReplayOnly and a corresponding entry is not found for the current request.

Because the error is returned from the transport, it may be wrapped.

Example

package main

import (
	"log"
	"net/http"
	"net/url"

	"github.com/akupila/recorder"
)

func main() {
	rec := recorder.New("notfound")

	// Disallow network traffic so this returns an error.
	rec.Mode = recorder.ReplayOnly

	cli := &http.Client{Transport: rec}
	if _, err := cli.Get("https://example.com"); err != nil {
		uerr, ok := err.(*url.Error)
		if !ok {
			log.Fatal("Error is not *url.Error")
		}
		_, ok = uerr.Err.(recorder.NoRequestError)
		if ok {
			// Recorded entry was not found.
		}
	}
}

func (NoRequestError) Error

func (e NoRequestError) Error() string

Error implements the error interface.

type OncePerCall

type OncePerCall struct {
	// contains filtered or unexported fields
}

OncePerCall is a Selector that selects entries based on the method and URL, but it will only select any given entry at most once.

func (*OncePerCall) Select

func (s *OncePerCall) Select(entries []Entry, req *http.Request) (Entry, bool)

Select implements Selector and chooses an entry.

type Recorder

type Recorder struct {
	// Filename to use for saved entries. A .yml extension is added if not set.
	// Any subdirectories are created if needed.
	//
	// Required if mode is not Passthrough.
	Filename string

	// Mode to use. Default mode is Auto.
	Mode Mode

	// Filters to apply before saving to disk.
	// Filters are executed in the order specified.
	Filters []Filter

	// Transport to use for real request.
	// If nil, http.DefaultTransport is used.
	Transport http.RoundTripper

	// An optional Select function may be specified to control which recorded
	// Entry is selected to respond to a given request. If nil, the default
	// selection is used that picks the first recorded response with a matching
	// method and url.
	Selector Selector
	// contains filtered or unexported fields
}

Recorder wraps a http.RoundTripper by recording requests that go through it.

When recording, any observed requests are written to disk after response. In case previous entries were recorded for the same endpoint, the file is overwritten on first request.

func New

func New(filename string, filters ...Filter) *Recorder

New is a convenience function for creating a new recorder.

func (*Recorder) Lookup

func (r *Recorder) Lookup(method, url string) (Entry, bool)

Lookup returns an existing entry matching the given method and url.

The method and url are case-insensitive.

Returns false if no such entry exists.

func (*Recorder) RoundTrip

func (r *Recorder) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip implements http.RoundTripper and does the actual request.

The behavior depends on the mode set:

Auto:          If an existing entry exists, the response from the entry
               is returned.
ReplayOnly:    Returns a previously recorded response. Returns
               NoRequestError if an entry is found for the request.
Record:        Always send real request and record the response. If an
               existing entry is found, it is overwritten.
Passthrough:   The request is passed through to the underlying
               transport.

Attempting to set another mode will cause a panic.

type Request

type Request struct {
	Method  string            `yaml:"method"`
	URL     string            `yaml:"url"`
	Headers map[string]string `yaml:"headers,omitempty"`
	Body    string            `yaml:"body,omitempty"`
}

A Request is a recorded outgoing request.

The headers are flattened to a simple key-value map. The underlying request may contain multiple value for each key but in practice this is not very common and working with a simple key-value map is much more convenient.

type Response

type Response struct {
	StatusCode int               `yaml:"status_code"`
	Headers    map[string]string `yaml:"headers,omitempty"`
	Body       string            `yaml:"body,omitempty"`
}

A Response is a recorded incoming response.

The headers are flattened to a simple key-value map. The underlying request may contain multiple value for each key but in practice this is not very common and working with a simple key-value map is much more convenient.

type Selector

type Selector interface {
	Select(entries []Entry, req *http.Request) (Entry, bool)
}

Selector chooses a recorded Entry to response to a given request.