apiproxy

README

apiproxy

apiproxy is a proxy for HTTP/REST APIs with configurable cache timeouts, etc.

Documentation: https://sourcegraph.com/github.com/sourcegraph/apiproxy/tree

Installation

go get github.com/sourcegraph/apiproxy

Usage

apiproxy supports 3 modes of usage: as a standalone server, as a Go client, and as a Go HTTP server handler.

As a standalone HTTP proxy server

Running apiproxy as a standalone HTTP proxy server lets you access it from any HTTP client on any host.

The included apiproxy program runs a proxy server with a specified target URL:

$ go install github.com/sourcegraph/apiproxy/cmd/apiproxy
$ apiproxy http://api.example.com
2013/09/13 21:19:57 Starting proxy on :8080 with target http://api.example.com

Once launched, HTTP requests to http://localhost:8080 will be proxied to http://api.example.com and the responses cached according to the HTTP standard.

See apiproxy -h for more information.

As a Go client http.RoundTripper

Clients can use apiproxy.RevalidationTransport to modify the caching behavior of HTTP requests by setting a custom apiproxy.Validator on the transport. The Validator is used to determine whether a cache entry for a URL is still valid at a certain age.

A Validator can be created by wrapping a func(url *url.URL, age time.Duration) bool function with apiproxy.ValidatorFunc(...) or by using the built-in GitHub API implementation, MaxAge.

The RevalidationTransport can be used in an http.Client that is passed to external libraries, to give control over HTTP requests when using libraries whose only configuration point is an http.Client.

The file service/github/client_test.go contains a full example using the go-github library, summarized here:

transport := &apiproxy.RevalidationTransport{
  Transport: httpcache.NewMemoryCacheTransport(),
  Check: (&githubproxy.MaxAge{
    User:         time.Hour * 24,
    Repository:   time.Hour * 24,
    Repositories: time.Hour * 24,
    Activity:     time.Hour * 12,
  }).Validator(),
}
httpClient := &http.Client{Transport: transport}

client := github.NewClient(httpClient)

Now HTTP requests initiated by go-github will be subject to the caching policy set by the custom RevalidationTransport.

You can also inject a Cache-Control: no-cache header to a specific request if you use apiproxy.RequestModifyingTransport as follows:

// Wrap our transport from above in a RequestModifyingTransport.
transport = &apiproxy.RequestModifyingTransport{Transport: transport}
transport.Override(regexp.MustCompile(`^/repos/sourcegraph/apiproxy$`), apiproxy.NoCache, true)

// Now this call to the GitHub API will carry a `Cache-Control: no-cache` header.
client.Repositories.Get("sourcegraph", "apiproxy")

As a Go server http.Handler

The function apiproxy.NewCachingSingleHostReverseProxy(target *url.URL, cache Cache) *httputil.ReverseProxy returns a simple caching reverse proxy that you can use as an http.Handler.

You can wrap the handler's Transport in an 'apiproxy.RevalidationTransport` to specify custom cache timeout behavior.

The file cmd/apiproxy/apiproxy.go contains a full example, summarized here:

proxy := apiproxy.NewCachingSingleHostReverseProxy("https://api.github.com", httpcache.NewMemoryCache())
cachingTransport := proxy.Transport.(*httpcache.Transport)
cachingTransport.Transport = &apiproxy.RevalidationTransport{
  Check: apiproxy.ValidatorFunc(func(url *url.URL, age time.Duration) bool {
    // only revalidate expired cache entries older than 30 minutes
    return age > 30 * time.Minute
  }),
}
http.Handle("/", handlers.CombinedLoggingHandler(os.Stdout, proxy))
http.ListenAndServe(":8080", nil)

Examples

The included cmd/chirper/chirper.go example program helps demonstrate apiproxy's features. It returns a constantly updating JSON array of "chirps" at the path /chirps.

1. Run go run example/chirper/chirper.go in one terminal window.
2. Install apiproxy: go install github.com/sourcegraph/apiproxy/cmd/apiproxy
3. Run apiproxy -http=:8080 -never-revalidate http://localhost:9090 in another terminal window.

Now, let's make a request to the chirper API via apiproxy. Since this is our first request, apiproxy will fetch the response from the chirper HTTP server.

$ curl http://localhost:8080/chirps

Notice that apiproxy hit the chirper HTTP server: chirper logged the message "Listed chirps".

But next time we make the same request, apiproxy won't need to hit chirper, because the response has been cached and we are using the -never-revalidate option to treat cache entries as though they never expire.

$ curl http://localhost:8080/chirps

Note that we didn't hit chirper (it didn't log "Listed chirps").

However, if we pass a Cache-Control: no-cache header in our request, apiproxy will ignore the cache and hit chirper:

$ curl -H 'Cache-Control: no-cache' q:8080/chirps

Note that chirper logs "Listed chirps" after this request.

Cache backends

Any cache backend that implements httpcache.Cache suffices, including:

• httpcache.MemoryCache, instantiated with NewMemoryCache() *MemoryCache
• diskcache.Cache, instantiated with diskcache.New(basePath string) *Cache
• s3cache.Cache, instantiated with s3cache.New(bucketURL string) *Cache (requires env vars S3_ACCESS_KEY and S3_SECRET_KEY)

Contributing

Patches and bug reports welcomed! Report issues and submit pull requests using GitHub.

Documentation

Overview

Package apiproxy proxies HTTP/REST APIs and supports configurable cache timeouts, etc.

Index

• Variables
• func NewCachingSingleHostReverseProxy(target *url.URL, cache httpcache.Cache) *httputil.ReverseProxy
• func NewSingleHostReverseProxy(url *url.URL) *httputil.ReverseProxy
• type PathMatchValidator
  • func (v PathMatchValidator) Valid(url *url.URL, age time.Duration) bool
• type RequestModifyingTransport
  • func (t *RequestModifyingTransport) Override(requestURI *regexp.Regexp, setHeaders http.Header, runOnlyOnce bool)
  • func (t *RequestModifyingTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)
• type RevalidationTransport
  • func (t *RevalidationTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)
• type Validator
• type ValidatorFunc
  • func (f ValidatorFunc) Valid(url *url.URL, age time.Duration) bool

Variables

var NeverRevalidate = ValidatorFunc(func(_ *url.URL, _ time.Duration) bool {
	return true
})

NeverRevalidate is a Validator for use with RevalidationTransport that causes HTTP requests to never revalidate cache entries. If a cache entry exists, it will always be used, even if it is expired.

var NoCache = http.Header{"Cache-Control": []string{"no-cache"}}

Functions

func NewCachingSingleHostReverseProxy

func NewCachingSingleHostReverseProxy(target *url.URL, cache httpcache.Cache) *httputil.ReverseProxy

NewCachingSingleHostReverseProxy constructs a caching reverse proxy handler for target. If cache is nil, a volatile, in-memory cache is used.

func NewSingleHostReverseProxy

func NewSingleHostReverseProxy(url *url.URL) *httputil.ReverseProxy

NewSingleHostReverseProxy wraps net/http/httputil.NewSingleHostReverseProxy and sets the Host header based on the target URL.

Types

type PathMatchValidator

type PathMatchValidator map[*regexp.Regexp]time.Duration

PathMatchValidator is a map of path regexps to the maximum age of resources matching one of those regexps.

func (PathMatchValidator) Valid

func (v PathMatchValidator) Valid(url *url.URL, age time.Duration) bool

Valid implements Validator.

type RequestModifyingTransport

type RequestModifyingTransport struct {

	// Transport is the underlying transport. If nil, net/http.DefaultTransport
	// is used.
	Transport http.RoundTripper
	// contains filtered or unexported fields
}

RequestModifyingTransport is an implementation of net/http.RoundTripper that allows headers to be overwritten on requests matching certain predicates.

It gives more control over HTTP requests (e.g., caching) when using libraries whose only HTTP configuration point is a http.Client or http.RoundTripper.

func (*RequestModifyingTransport) Override

func (t *RequestModifyingTransport) Override(requestURI *regexp.Regexp, setHeaders http.Header, runOnlyOnce bool)

Override instructs the transport to set the specified headers (overwriting existing headers of the same name) on a GET or HEAD request whose request URI matches the regexp. If runOnlyOnce is true, the override will be deleted after execution (and won't affect any future requests); otherwise, it will remain in effect for the lifetime of the transport.

func (*RequestModifyingTransport) RoundTrip

func (t *RequestModifyingTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip implements net/http.RoundTripper.

type RevalidationTransport

type RevalidationTransport struct {
	// Check.Valid is called on each request in RoundTrip. If it returns true,
	// RoundTrip synthesizes and returns an HTTP 304 Not Modified response.
	// Otherwise, the request is passed through to the underlying transport.
	Check Validator

	// Transport is the underlying transport. If nil, net/http.DefaultTransport is used.
	Transport http.RoundTripper
}

RevalidationTransport is an implementation of net/http.RoundTripper that permits custom behavior with respect to cache entry revalidation for resources on the target server.

If the request contains cache validators (an If-None-Match or If-Modified-Since header), then Check.Valid is called to determine whether the cache entry should be revalidated (by being passed to the underlying transport). In this way, the Check Validator can effectively extend or shorten cache age limits.

If the request does not contain cache validators, then it is passed to the underlying transport.

func (*RevalidationTransport) RoundTrip

func (t *RevalidationTransport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip takes a Request and returns a Response.

type Validator

type Validator interface {
	Valid(url *url.URL, age time.Duration) bool
}

Validators are used to determine whether a cache entry for a URL is still valid at a certain age. They are used to set custom max-ages for cache entries (e.g., to extend the max-age of a certain API resource that an application knows to be rarely updated).

type ValidatorFunc

type ValidatorFunc func(url *url.URL, age time.Duration) bool

ValidatorFunc is an adapter type to allow the use of ordinary functions as validators. If f is a function with the appropriate signature, ValidatorFunc(f) is a Validator object that calls f.

func (ValidatorFunc) Valid

func (f ValidatorFunc) Valid(url *url.URL, age time.Duration) bool

Valid implements Validator.