Documentation
¶
Overview ¶
Package fetch defines interface to retrieve contents to package.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var DefaultFetchClient = &http.Client{CheckRedirect: NeverRedirect}
DefaultFetchClient is a drop-in FetchClient to fetch content via HTTP in a usual manner.
var ErrURLMismatch = errors.New("fetch: URL doesn't match the fetch targets")
ErrURLMismatch is returned by WithSelector clients when the request URL does not match their selector.
Functions ¶
func NeverRedirect ¶
NeverRedirect instructs http.Client to stop handling the redirect and just return the last response instead, when set to the CheckRedirect field.
Technically, NeverRedirect is a function just returning ErrUseLastResponse.
Types ¶
type FetchClient ¶
type FetchClient interface {
// Do handles an HTTP request and returns an HTTP response. It is like
// http.Client.Do but may not send an HTTP request for real.
//
// Do should *not* handle redirects. See the document above.
Do(req *http.Request) (*http.Response, error)
}
FetchClient retrieves contents from the server or other data source.
FetchClient should not handle redirects. webpackager.Runner handles redirects in its own manner, hence FetchClient should pass any 30x responses through.
An http.Client set up with NeverRedirect, such as DefaultFetchClient, meets the contracts and is the most natural choice. Other implementations may retrieve contents from other sources such as database or filesystem.
func WithSelector ¶
func WithSelector(client FetchClient, selector urlmatcher.Matcher) FetchClient
WithSelector wraps client to issue a fetch only if the request URL matches selector.
type RequestTweaker ¶
type RequestTweaker interface {
// Tweak mutates req. parent is the request which spawned req. If req is
// a request to fetch a subresource, parent is typically the one used to
// fetch the main resource. parent can be nil (e.g. when req is a request
// to fetch a main resource) and should not be mutated.
Tweak(req *http.Request, parent *http.Request) error
}
RequestTweaker mutates an http.Request to set the refer(r)er, add custom HTTP headers, and so on.
var DefaultRequestTweaker RequestTweaker = SetReferer()
DefaultRequestTweaker is a RequestTweaker used by default.
func CopyParentHeaders ¶
func CopyParentHeaders(keys []string) RequestTweaker
CopyParentHeaders copies the header fields of the provided keys from the parent request. When the tweaked request already has those header fields, their values will be overwritten by the values from the parent request. CopyParentHeaders, however, only mutates the header fields present in the parent request and keeps all other header fields untouched.
keys are case insensitive: they are canonicalized by http.CanonicalHeaderKey.
func SetCustomHeaders ¶
func SetCustomHeaders(header http.Header) RequestTweaker
SetCustomHeaders populates the provided HTTP header fields to the request. When the request already has the same header fields, their values will be overwritten with the values from the provided http.Header.
func SetReferer ¶
func SetReferer() RequestTweaker
SetReferer sets the Referer HTTP header with the parent request URL.
type RequestTweakerSequence ¶
type RequestTweakerSequence []RequestTweaker
RequestTweakerSequence consists of a series of RequestTweakers.
type Selector ¶
type Selector struct {
Allow []urlmatcher.Matcher
Deny []urlmatcher.Matcher
}
Selector is a urlmatcher.Matcher designed for use with WithSelector. It is almost equivalent to
AllOf(AnyOf(Allow...), Not(AnyOf(Deny...)))
but empty Allow is interpreted as "allow any." Empty Deny is interpreted as "deny none," like AnyOf(empty). Using Selector with WithSelector makes the selector easier to understand, especially for those familiar with the Allow/Deny style. Using Selector is not mandatory though: WithSelector can use any urlmatcher.Matcher as the selector.
Source Files
Directories