requests

package module
v0.0.0-...-af9bbdb Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 9, 2016 License: MIT Imports: 9 Imported by: 6

README

requests

GoDoc Build Status Coverage Status Donate

The simplest and functional HTTP requests in Go.

Introduction

Ever find yourself going back and forth between net/http and io docs and your code while making HTTP calls? Requests takes care of that by abstracting several types on both ends to just Request and Response--just the way it should be. Find out how or jump right in to examples.

Purpose

I need an HTTP request package that:

Usage

the following are the core differences from the standard net/http package.

Functional Options

requests employs functional options as optional parameters, this approach being idiomatic, clean, and makes a friendly, extensible API. This pattern is adopted after feedback from the Go community.


jsontype := func(r *requests.Request) {
        r.Header.Add("content-type", "application/json")
}
res, err := requests.Get("http://example.com", jsontype)
Embedded Standard Types

requests uses custom Request and Response types to embed standard http.Request, http.Response, and http.Client in order to insert helper methods, make configuring options atomic, and handle asynchronous errors.

The principle is, the caller should be able to set all the configurations on the "Request" instead of doing it on the client, transport, vice versa. For instance, Timeout can be set on the Request.


timeout := func(r *requests.Request) {

        // Set Timeout on *Request instead of *http.Client
        r.Timeout = time.Duration(5) * time.Second
}
res, err := requests.Get("http://example.com", timeout)
if err != nil {
        panic(err)
}

// Helper method
htmlStr := res.String()

Also, where http.Transport was normally needed to set control over proxies, TLS configuration, keep-alives, compression, and other settings, now everything is handled by Request.


tlsConfig := func(r *requests.Request) {
        r.TLSClientConfig = &tls.Config{RootCAs: x509.NewCertPool()}
        r.DisableCompression = true
}
res, _ := requests.Get("http://example.com", tlsConfig)

See Types and Methods for more information.

Asynchronous APIs

requests provides the following abstractions around sending HTTP requests in goroutines:

  • requests.GetAsync
  • requests.PostAsync
  • requests.Pool

All return a receive-only channel on which *requests.Response can be waited on.


rc, err := requests.GetAsync("http://httpbin.org/get")
if err != nil {
        panic(err)
}
res := <-rc

// Handle connection errors.
if res.Error != nil {
        panic(res.Error)
}

// Helper method
content := res.Bytes()

See Async and Handling Async Errors for more usage information.

Install


go get github.com/jochasinga/requests

Testing

requests uses Go standard testing package. Run this in the project's directory:


go test -v -cover

Examples

requests.Get

Sending a basic GET request is straightforward.


res, err := requests.Get("http://httpbin.org/get")
if err != nil {
        panic(err)
}

fmt.Println(res.StatusCode)  // 200

To send additional data, such as a query parameter, or set basic authorization header or content type, use functional options.


// Add a query parameter.
addFoo := func(r *requests.Request) {
        r.Params.Add("foo", "bar")
}

// Set basic username and password.
setAuth := func(r *requests.Request) {
        r.SetBasicAuth("user", "pass")
}

// Set the Content-Type.
setMime := func(r *requests.Request) {
        r.Header.Add("content-type", "application/json")
}

// Pass as parameters to the function.
res, err := requests.Get("http://httpbin.org/get", addFoo, setAuth, setMime)

Or configure everything in one function.


opts := func(r *requests.Request) {
        r.Params.Add("foo", "bar")
        r.SetBasicAuth("user", "pass")
        r.Header.Add("content-type", "application/json")
}
res, err := requests.Get("http://httpbin.org/get", opts)
requests.Post

Send POST requests with specified bodyType and body.


res, err := requests.Post("https://httpbin.org/post", "image/jpeg", &buf)

It also accepts variadic number of functional options:


notimeout := func(r *requests.Request) {
        r.Timeout = 0
}
res, err := requests.Post("https://httpbin.org/post", "application/json", &buf, notimeout)
requests.PostJSON

Encode your map or struct data as JSON and set bodyType to application/json implicitly.


first := map[string][]string{
        "foo": []string{"bar", "baz"},
}
second := struct {
        Foo []string `json:"foo"`
}{[]string{"bar", "baz"}}

payload := map[string][]interface{}{
        "twins": {first, second}
}

res, err := requests.PostJSON("https://httpbin.org/post", payload)
Other Verbs

HEAD, PUT, PATCH, DELETE, and OPTIONS are supported. See the doc for more info.

Async

requests.GetAsync

After parsing all the options, GetAsync spawns a goroutine to send a GET request and return <-chan *Response immediately on which *Response can be waited.


timeout := func(r *requests.Request) {
        r.Timeout = time.Duration(5) * time.Second
}

rc, err := requests.GetAsync("http://golang.org", timeout)
if err != nil {
        panic(err)
}

// Do other things...

// Block and wait
res := <-rc

// Handle a "reject" with Error field.
if res.Error != nil {
	panic(res.Error)
}

fmt.Println(res.StatusCode)  // 200

select can be used to poll many channels asynchronously like normal.


res1, _ := requests.GetAsync("http://google.com")
res2, _ := requests.GetAsync("http://facebook.com")
res3, _ := requests.GetAsync("http://docker.com")

for i := 0; i < 3; i++ {
        select {
    	case r1 := <-res1:
    		fmt.Println(r1.StatusCode)
    	case r2 := <-res2:
    		fmt.Println(r2.StatusCode)
    	case r3 := <-res3:
    		fmt.Println(r3.StatusCode)
    	}
}

requests.Pool is recommended for collecting concurrent responses from multiple requests.

requests.PostAsync

An asynchronous counterpart of requests.Post.


query := bytes.NewBufferString(`{
        "query" : {
            "term" : { "user" : "poco" }
        }
}`)

// Sending query to Elasticsearch server
rc, err := PostAsync("http://localhost:9200/users/_search", "application/json", query)
if err != nil {
        panic(err)
}
resp := <-rc
if resp.Error != nil {
        panic(resp.Error)
}
result := resp.JSON()
requests.Pool

Contains a Responses field of type chan *Response with variable-sized buffer specified in the constructor. Pool is used to collect in-bound responses sent from numbers of goroutines corresponding to the number of URLs provided in the slice.


urls := []string{
        "http://golang.org",
        "http://google.com",
        "http://docker.com",
        "http://medium.com",
        "http://example.com",
        "http://httpbin.org/get",
        "https://en.wikipedia.org",
}

// Create a pool with the maximum buffer size.
p := requests.NewPool(10)
opts := func(r *requests.Request) {
        r.Header.Set("user-agent", "GoBot(http://example.org/"))
        r.Timeout = time.Duration(10) * time.Second
}

results, err := p.Get(urls, opts)

// An error is returned when an attempt to construct a
// request fails, probably from a malformed URL.
if err != nil {
        panic(err)
}
for res := range results {
        if res.Error != nil {
                panic(res.Error)
        }
        fmt.Println(res.StatusCode)
}

You may want to ignore errors from malformed URLs instead of handling each of them, for instance, when crawling mass URLs.

To suppress the errors from being returned, either thrown away the error with _ or set the IgnoreBadURL field to true, which suppress all internal errors from crashing the pool:


results, err := p.Get(urls, func(r *requests.Request) {
        r.IgnoreBadURL = true
})

Pool.Responses channel is closed internally when all the responses are sent.

Types and Methods

requests.Request

It has embedded types *http.Request and *http.Client, making it an atomic type to pass into a functional option. It also contains field Params, which has the type url.Values. Use this field to add query parameters to your URL. Currently, parameters in Params will replace all the existing query string in the URL.


addParams := func(r *requests.Request) {
        r.Params = url.Values{
	        "name" : { "Ava", "Sanchez", "Poco" },
        }
}

// "q=cats" will be replaced by the new query string
res, err := requests.Get("https://httpbin.org/get?q=cats", addParams)
requests.Response

It has embedded type *http.Response and provides extra byte-like helper methods such as:

  • Len() int
  • String() string
  • Bytes() []byte
  • JSON() []byte

These methods will return an equivalent of nil for each return type if a certain condition isn't met. For instance:


res, _ := requests.Get("http://somecoolsite.io")
fmt.Println(res.JSON())

If the response from the server does not specify Content-Type as "application/json", res.JSON() will return an empty bytes slice. It does not panic if the content type is empty.

These methods close the response's body automatically.

Another helper method, ContentType, is used to get the media type in the response's header, and can be used with the helper methods to determine the type before reading the output.


mime, _, err := res.ContentType()
if err != nil {
        panic(err)
}

switch mime {
case "application/json":
        fmt.Println(res.JSON())
case "text/html", "text/plain":
        fmt.Println(res.String())
default:
        fmt.Println(res.Bytes())
}
Handling Async Errors

requests.Response also has an Error field which will contain any error caused in the goroutine within requests.GetAsync and carries it downstream for proper handling (Think reject in Promise but more straightforward in Go-style).


rc, err := requests.GetAsync("http://www.docker.io")

// This error is returned before the goroutine i.e. malformed URL.
if err != nil {
        panic(err)
}

res := <-rc

// This connection error is "attached" to the response.
if res.Error != nil {
	panic(res.Error)
}

fmt.Println(res.StatusCode)

Response.Error is default to nil when there is no error or when the response is being retrieved from a synchronous function.

HTTP Test Servers

Check out my other project relay, useful test proxies and round-robin switchers for end-to-end HTTP tests.

Contribution

Yes, please fork away.

Disclaimer

To support my ends in NYC and help me push commits, please consider Donate to fuel me with quality 🍵 or 🌟 this repo for spiritual octane.
Reach me at @jochasinga.

Documentation

Overview

Package requests is a minimal, atomic and expressive way of making HTTP requests. It is inspired partly by the HTTP request libraries in other dynamic languages like Python and Javascript. It is safe for all rodents, not just Gophers.

Requests is built as a convenient, expressive API around Go's standard http package. With special Request and Response types to help facilitate and streamline RESTful tasks.

To send a common GET request just like you'd do with `http.Get`. 

import (
	"github.com/jochasinga/requests"
)

func main() {
	resp, err := requests.Get("http://httpbin.org/get")
	fmt.Println(resp.StatusCode)  // 200
}

To send additional data, such as a query parameter, basic authorization header, or content type, just pass in functional options: 

addFoo := func(r *Request) {
        r.Params.Add("foo", "bar")
}
addAuth := func(r *Request) {
        r.SetBasicAuth("user", "pass")
}
addMime := func(r *Request) {
        r.Header.Add("content-type", "application/json")
}
resp, err := requests.Get("http://httpbin.org/get", addFoo, addAuth, addMime)

// Or everything goes into one functional option
opts := func(r *Request) {
        r.Params.Add("foo", "bar")
        r.SetBasicAuth("user", "pass")
        r.Header.Add("content-type", "application/json")
}
resp, err := requests.Get("http://httpbin.org/get", opts)

The data can be a map or struct (anything JSON-marshalable). 

        data1 := map[string][]string{"foo": []string{"bar", "baz"}}
        data2 := struct {
                Foo []string `json:"foo"`
        }{[]string{"bar", "baz"}}

        data := map[string][]interface{}{
		"combined": {data1, data2},
	}

        res, err := requests.Post("http://httpbin.org/post", "application/json", data)

You can asynchronously wait on a GET response with `GetAsync`. 

timeout := time.Duration(1) * time.Second
resChan, _ := requests.GetAsync("http://httpbin.org/get", nil, nil, timeout)

// Do some other things

res := <-resChan
fmt.Println(res.StatusCode)  // 200

The response returned has the type *requests.Response which embeds *http.Response type to provide more buffer-like methods such as Len(), String(), Bytes(), and JSON(). 

// Len() returns the body's length.
var len int = res.Len()

// String() returns the body as a string.
var text string = res.String()

// Bytes() returns the body as bytes.
var content []byte = res.Bytes()

// JSON(), like Bytes() but returns an empty `[]byte` unless `Content-Type`
// is set to `application/json` in the response's header.
var jsn []byte = res.JSON()

These special methods use bytes.Buffer under the hood, thus unread portions of data are returned. Make sure not to read from the response's body beforehand.

Requests is an ongoing project. Any contribution is whole-heartedly welcomed.

Package requests provide useful and declarative methods for RESTful HTTP requests.

Package requests provide useful and declarative methods for RESTful HTTP requests.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetAsync 

func GetAsync(urlStr string, options ...func(*Request)) (<-chan *Response, error)

GetAsync sends a HTTP GET request to the provided URL and returns a <-chan *http.Response immediately. 

timeout := func(r *request.Request) {
        r.Timeout = time.Duration(10) * time.Second
}
rc, err := requests.GetAsync("http://httpbin.org/get", timeout)
if err != nil {
        panic(err)
}
resp := <-rc
if resp.Error != nil {
        panic(resp.Error)
}
fmt.Println(resp.String())

func PostAsync 

func PostAsync(urlStr, bodyType string, body io.Reader, options ...func(*Request)) (<-chan *Response, error)

PostAsync sends a HTTP POST request to the provided URL, body type, and body, and returns a <-chan *http.Response immediately. 

redirect := func(r *requests.Request) {
          r.CheckRedirect = redirectPolicyFunc
}

resp, err := requests.PostAsync("https://httpbin.org/post", "image/png", &buf, redirect) 

if err != nil {
        panic(err)
}

resp := <-rc 

if resp.Error != nil {
    panic(resp.Error)
}

fmt.Println(resp.String())

Types

type HTTPRequest 

type HTTPRequest interface {
	AddCookie(*http.Cookie)
	BasicAuth() (string, string, bool)
	Cookie(string) (*http.Cookie, error)
	Cookies() []*http.Cookie
	FormFile(string) (multipart.File, *multipart.FileHeader, error)
	FormValue(string) string
	MultipartReader() (*multipart.Reader, error)
	ParseForm() error
	ParseMultipartForm(int64) error
	PostFormValue(string) string
	ProtoAtLeast(int, int) bool
	Referer() string
	SetBasicAuth(string, string)
	UserAgent() string
	Write(io.Writer) error
	WriteProxy(io.Writer) error
}

HTTPRequest is for future uses.

type HTTPResponse 

type HTTPResponse interface {
	Cookies() []*http.Cookie
	Location() (*url.URL, error)
	ProtoAtLeast(major, minor int) bool
	Write(w io.Writer) error
	Bytes() []byte
	String() string
	JSON() []byte
	Len() int
}

HTTPResponse is meant for future use for the time being.

type Pool 

type Pool struct {
	Responses    chan *Response
	IgnoreBadURL bool
}

Pool represents a variable-sized bufferred channel of type *Response which collects results from each request in a goroutine.

func NewPool 

func NewPool(max int) *Pool

NewPool creates a *Pool instance with the channel's buffer size of max.

func (*Pool) Get 

func (p *Pool) Get(urls []string, options ...func(*Request)) (<-chan *Response, error)

Get sends asychronous GET requests to the provided URLs and returns a receive-only channel of type *Response which get closed after all responses are sent to.

type Request 

type Request struct {

	// "Inherite" both http.Request and http.Client
	*http.Request
	*http.Client
	*http.Transport

	// Params is an alias used to add query parameters to
	// the request through functional options.
	// Values in Params have higher precedence over the
	// query string in the initial URL.
	Params url.Values
}

Request is a hybrid descendant of http.Request and http.Client. It is used as an argument to the functional options.

type Response 

type Response struct {
	*http.Response
	Error error
}

Response is a *http.Response and implements HTTPResponse.

func Delete 

func Delete(urlStr string, options ...func(*Request)) (*Response, error)

Delete sends a HTTP DELETE request to the provided URL.

func Get 

func Get(urlStr string, options ...func(*Request)) (*Response, error)

Get sends a HTTP GET request to the provided url with the functional options to add query paramaters, headers, timeout, etc. 

addMimeType := func(r *Request) {
        r.Header.Add("content-type", "application/json")
}

resp, err := requests.Get("http://httpbin.org/get", addMimeType)
if err != nil {
        panic(err)
}
fmt.Println(resp.StatusCode)

func Head(urlStr string, options ...func(*Request)) (*Response, error)

Head sends a HTTP HEAD request to the provided url with the functional options to add query paramaters, headers, timeout, etc.

func Options 

func Options(urlStr string) (*Response, error)

Options sends a rarely-used HTTP OPTIONS request to the provided URL. Options only allows one parameter--the destination URL string.

func Patch 

func Patch(urlStr, bodyType string, body io.Reader, options ...func(*Request)) (*Response, error)

Patch sends a HTTP PATCH request to the provided URL with optional body to modify data.

func Post 

func Post(urlStr, bodyType string, body io.Reader, options ...func(*Request)) (*Response, error)

Post sends a HTTP POST request to the provided URL, and encode the data according to the appropriate bodyType. 

redirect := func(r *requests.Request) {
          r.CheckRedirect = redirectPolicyFunc
}

resp, err := requests.Post("https://httpbin.org/post", "image/png", &buf, redirect) 

if err != nil {
        panic(err)
}

fmt.Println(resp.JSON())

func PostJSON 

func PostJSON(urlStr string, body interface{}, options ...func(*Request)) (*Response, error)

PostJSON aka UnsafePost! It marshals your data as JSON and set the bodyType to "application/json" automatically. 

redirect := func(r *requests.Request) {
        r.CheckRedirect = redirectPolicyFunc
}

first := map[string][]string{"foo": []string{"bar", "baz"}} second := struct {Foo []string `json:"foo"`}{[]string{"bar", "baz"}} payload := map[string][]interface{}{"twins": {first, second}}

resp, err := requests.PostJSON("https://httpbin.org/post", payload, redirect) 

if err != nil {
        panic(err)
}

fmt.Println(resp.StatusCode)

func Put 

func Put(urlStr, bodyType string, body io.Reader, options ...func(*Request)) (*Response, error)

Put sends HTTP PUT request to the provided URL.

func (*Response) Bytes 

func (r *Response) Bytes() []byte

Bytes returns the response's Body as []byte. Any errors reading from the body is ignored for convenience.

func (*Response) ContentType 

func (r *Response) ContentType() (string, map[string]string, error)

ContentType is an alias for Response.Header.Get("content-type"), but filtered through mime.ParseMediaType to rid of extra arguments such as encoding.

func (*Response) JSON 

func (r *Response) JSON() []byte

JSON returns the response's body as []byte if Content-Type is in the header contains "application/json".

func (*Response) Len 

func (r *Response) Len() int

Len returns the response's body's unread portion's length, which is the full length provided it has not been read.

func (*Response) String 

func (r *Response) String() string

String returns the response's body as string. Any errors reading from the body is ignored for convenience.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.