bulk

README

httpbulk-go

httpbulk-go is a small wrapper lib, intended to ease the parallel loading of multiple http resources.

This is particularly useful, if you are aggregating an object from multiple resources.

Download:

go get github.com/kernle32dll/httpbulk-go

Detailed documentation can be found on pkg.go.dev.

Simple usage

First, you have to instantiate a bulk.Executor. This is done via bulk.NewExecutor (which takes option style parameters).

bulk.NewExecutor(bulk.Client(http.DefaultClient), bulk.ConcurrencyLimit(10))

With the executor instantiated, you can issue asynchronous requests via the AddRequests method, which returns a chan Result slice. Read from these channels, to retrieve the results.

urls := []string{
    "https://www.google.com",
    "https://www.bing.com",
    "https://www.yahoo.com",
    "https://www.tarent.de",
}

results := executor.AddRequests(context.Background(), urls...)

The Result object has several methods to introspect the response. The most useful being Res() for getting the original *http.Response, and Err() for getting the error, if any ocured while fetching.

Implementation note: If the context used for the requests is canceled, or exceeds its deadline, the corresponding error is propagated in the Result object.

Advanced usage (request interception)

For more control, you can use the AddRequestsWithInterceptor method, which allows you to modify the request prior to sending. This might be useful for setting headers and/or changing the request type.

executor.AddRequestsWithInterceptor(context.Background(), func(r *http.Request) error {
    // Change the request method to HEAD
    r.Method = "HEAD"
    return nil
}, urls...)

Advanced usage (future objects)

If you want to safely provide the result of the request to multiple receivers (e.g. multiple go routines), bulk.Future is to your rescue. A Future provides a simple Get() method, which blocks on the first execution to fetch the result from the underlining channel, but returns the same result on subsequent calls. Of course, this method is concurrency safe.

To get a bulk.Future, use the following two (otherwise semantically identical to their channel counterparts) methods:

executor.AddFutureRequests(context.Background(), urls...)

executor.AddFutureRequestsWithInterceptor(context.Background(), func(r *http.Request) error {
    // Change the request method to HEAD
    r.Method = "HEAD"
    return nil
}, urls...)

Thanks

This lib has been derived from the following code gist. All kudos to Montana Flynn (montanaflynn)

https://gist.github.com/montanaflynn/ea4b92ed640f790c4b9cee36046a5383

Compatibility

httpbulk-go is automatically tested against Go 1.14.X, 1.15.X and 1.16.X.

Documentation

Index

• Variables
• func FetchLastModDatesForURLs(ctx context.Context, executor *Executor, ...) ([]time.Time, error)
• type Executor
  • func NewExecutor(setters ...Option) *Executor
  • func (e Executor) AddFutureRequests(ctx context.Context, urls ...string) []*Future
  • func (e Executor) AddFutureRequestsWithInterceptor(ctx context.Context, modifyRequest func(r *http.Request) error, urls ...string) []*Future
  • func (e Executor) AddRequests(ctx context.Context, urls ...string) []chan Result
  • func (e Executor) AddRequestsWithInterceptor(ctx context.Context, modifyRequest func(r *http.Request) error, urls ...string) []chan Result
  • func (e Executor) Close()
• type Future
  • func (future *Future) Done() bool
  • func (future *Future) Get() Result
  • func (future *Future) UnmarshalResponse(target interface{}) error
• type Option
  • func Client(client *http.Client) Option
  • func ConcurrencyLimit(limit int) Option
• type Options
• type Result
  • func (r Result) Duration() time.Duration
  • func (r Result) Err() error
  • func (r Result) Res() http.Response
  • func (r Result) URL() string
  • func (r Result) UnmarshalResponse(target interface{}) error

Variables

var (
	ErrRequestFailed = errors.New("http request failed")
)

Functions

func FetchLastModDatesForURLs

func FetchLastModDatesForURLs(
	ctx context.Context, executor *Executor, modifyRequest func(r *http.Request) error, urls ...string,
) ([]time.Time, error)

FetchLastModDatesForURLs fetches the last modification date for multiple urls at once.

Types

type Executor

type Executor struct {
	// contains filtered or unexported fields
}

Executor is the central bulk request maintainer.

func NewExecutor

func NewExecutor(setters ...Option) *Executor

NewExecutor instantiates a new Executor.

func (Executor) AddFutureRequests

func (e Executor) AddFutureRequests(
	ctx context.Context,
	urls ...string,
) []*Future

AddFutureRequests i one or more urls to be called and wrapped in a bulk.Future.

func (Executor) AddFutureRequestsWithInterceptor

func (e Executor) AddFutureRequestsWithInterceptor(
	ctx context.Context,
	modifyRequest func(r *http.Request) error,
	urls ...string,
) []*Future

AddFutureRequestsWithInterceptor issues one or more urls to be called and wrapped in a bulk.Future. For each call, optional hooks for modifying the request are executed (if not nil).

func (Executor) AddRequests

func (e Executor) AddRequests(
	ctx context.Context,
	urls ...string,
) []chan Result

AddRequests issues one or more urls to be called.

func (Executor) AddRequestsWithInterceptor

func (e Executor) AddRequestsWithInterceptor(
	ctx context.Context,
	modifyRequest func(r *http.Request) error,
	urls ...string,
) []chan Result

AddRequestsWithInterceptor issues one or more urls to be called. For each call, optional hooks for modifying the request are executed (if not nil).

func (Executor) Close

func (e Executor) Close()

Close closes the internal channels, and makes the Executor unavailable for further usage.

type Future

type Future struct {
	// contains filtered or unexported fields
}

Future is a holder for a bulk.Response channel, which allows to safely "read" the result multiple times

func (*Future) Done

func (future *Future) Done() bool

Done allows to introspect if the channel has already been read.

func (*Future) Get

func (future *Future) Get() Result

Get retrieves the underlying result, and caches it. Subsequent calls to get will not query the channel, but return the cached result.

func (*Future) UnmarshalResponse

func (future *Future) UnmarshalResponse(target interface{}) error

UnmarshalResponse is the concurrency and multi-read safe version of the Result UnmarshalResponse function.

First, the result is retrieved and cached as described in the Get() method. Afterwards, (if not done already) the body of the response stream is read (and afterwards closed!), and also cached. Finally, this cached body is then used for all unmarshalling attempts.

This effectively means, that the same response can be marshalled into different target types via this function.

If the Result did have an error, or something goes wrong while unmarshalling, the error is returned (result error of course taking precedence). If the error originated from reading the body, all subsequent calls return the same error, and no attempt to read the stream will be done again.

type Option

type Option func(*Options)

func Client

func Client(client *http.Client) Option

Client sets the http client, which is used for issuing requests. Per default, the default http client is used.

func ConcurrencyLimit

func ConcurrencyLimit(limit int) Option

ConcurrencyLimit regulates how many requests will be done at the same time Note, you need a http client, which is actually capable of doing multiple requests at the same time, for this to work properly. You can use -1 to indicate no limit - use with care!

type Options

type Options struct {
	ConcurrencyLimit int
	Client           *http.Client
}

Options is the option-wrapper for defining the workings of an Executor

type Result

type Result struct {
	// contains filtered or unexported fields
}

Result is a simple data holder for bulk request results.

func (Result) Duration

func (r Result) Duration() time.Duration

Duration returns the amount of time the request took.

func (Result) Err

func (r Result) Err() error

Err returns an error, if any occurred.

func (Result) Res

func (r Result) Res() http.Response

Res returns the http response, if no error occurred.

func (Result) URL

func (r Result) URL() string

URL returns the originally requested url. If you want to know the final URL, look at the HTTP response.

func (Result) UnmarshalResponse

func (r Result) UnmarshalResponse(target interface{}) error

UnmarshalResponse unmarshals the http response directly into the provided interface type (remember to provide a reference, not a value!), and closes the stream afterwards.

If the Result did have an error, or something goes wrong while unmarshalling, the error is returned (result error of course taking precedence).