curl

package
v7.0.0+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 31, 2020 License: BSD-3-Clause Imports: 16 Imported by: 0

Documentation

Overview

Package curl implements routines to fetch files given a URL.

curl currently supports HTTP, TFTP, and local files.

Index

Constants

This section is empty.

Variables

View Source
var (
	// ErrWrongScheme means the wrong mocked scheme was used.
	ErrWrongScheme = errors.New("wrong scheme")
	// ErrNoSuchHost means there is no host record in the mock.
	ErrNoSuchHost = errors.New("no such host exists")
	// ErrNoSuchFile means there is no file record in the mock.
	ErrNoSuchFile = errors.New("no such file exists on this host")
)
View Source
var (
	// DefaultHTTPClient is the default HTTP FileScheme.
	//
	// It is not recommended to use this for HTTPS. We recommend creating an
	// http.Client that accepts only a private pool of certificates.
	DefaultHTTPClient = NewHTTPClient(http.DefaultClient)

	// DefaultTFTPClient is the default TFTP FileScheme.
	DefaultTFTPClient = NewTFTPClient(tftp.ClientMode(tftp.ModeOctet), tftp.ClientBlocksize(1450), tftp.ClientWindowsize(65535))

	// DefaultSchemes are the schemes supported by default.
	DefaultSchemes = Schemes{
		"tftp": DefaultTFTPClient,
		"http": DefaultHTTPClient,
		"file": &LocalFileClient{},
	}
)
View Source
var (
	// ErrNoSuchScheme is returned by Schemes.Fetch and
	// Schemes.LazyFetch if there is no registered FileScheme
	// implementation for the given URL scheme.
	ErrNoSuchScheme = errors.New("no such scheme")
)

Functions

func IsURLError

func IsURLError(err error) bool

IsURLError returns true iff err is a URLError.

func RegisterScheme

func RegisterScheme(scheme string, fs FileScheme)

RegisterScheme calls DefaultSchemes.Register.

func RetryConnectErrors

func RetryConnectErrors(u *url.URL, err error) bool

RetryConnectErrors retries only connect(2) errors.

func RetryHTTP

func RetryHTTP(u *url.URL, err error) bool

RetryHTTP implements DoRetry for HTTP error codes where it makes sense.

func RetryTFTP

func RetryTFTP(u *url.URL, err error) bool

RetryTFTP retries downloads if the error does not contain FILE_NOT_FOUND.

pack.ag/tftp does not export the necessary structs to get the code out of the error message cleanly, but it does embed FILE_NOT_FOUND in the error string.

func RetryTemporaryNetworkErrors

func RetryTemporaryNetworkErrors(u *url.URL, err error) bool

RetryTemporaryNetworkErrors only retries temporary network errors.

This relies on Go's net.Error.Temporary definition of temporary network errors, which does not include network configuration errors. The latter are relevant for users of DHCP, for example.

Types

type DoRetry

type DoRetry func(u *url.URL, err error) bool

DoRetry returns true if the Fetch request for the URL should be retried. err is the error that Fetch previously returned.

DoRetry lets a FileScheme filter for errors returned by Fetch which are worth retrying. If this interface is not implemented, the default for SchemeWithRetries is to always retry. DoRetry returns true to indicate a request should be retried.

func RetryOr

func RetryOr(fn ...DoRetry) DoRetry

RetryOr returns a DoRetry function that returns true if any one of fn return true.

type File

type File interface {
	io.ReaderAt

	// URL is the file's original URL.
	URL() *url.URL
}

File is a reference to a file fetched through this library.

func Fetch

func Fetch(ctx context.Context, u *url.URL) (File, error)

Fetch fetchs a file via DefaultSchemes.

func LazyFetch

func LazyFetch(u *url.URL) (File, error)

LazyFetch calls LazyFetch on DefaultSchemes.

type FileScheme

type FileScheme interface {
	// Fetch returns a reader that gives the contents of `u`.
	//
	// It may do so by fetching `u` and placing it in a buffer, or by
	// returning an io.ReaderAt that fetchs the file.
	Fetch(ctx context.Context, u *url.URL) (io.ReaderAt, error)
}

FileScheme represents the implementation of a URL scheme and gives access to fetching files of that scheme.

For example, an http FileScheme implementation would fetch files using the HTTP protocol.

func NewTFTPClient

func NewTFTPClient(opts ...tftp.ClientOpt) FileScheme

NewTFTPClient returns a new TFTP client based on the given tftp.ClientOpt.

type HTTPClient

type HTTPClient struct {
	// contains filtered or unexported fields
}

HTTPClient implements FileScheme for HTTP files.

func NewHTTPClient

func NewHTTPClient(c *http.Client) *HTTPClient

NewHTTPClient returns a new HTTP FileScheme based on the given http.Client.

func (HTTPClient) Fetch

func (h HTTPClient) Fetch(ctx context.Context, u *url.URL) (io.ReaderAt, error)

Fetch implements FileScheme.Fetch.

type HTTPClientCodeError

type HTTPClientCodeError struct {
	Err      error
	HTTPCode int
}

HTTPClientCodeError is returned by HTTPClient.Fetch when the server replies with a non-200 code.

func (*HTTPClientCodeError) Error

func (h *HTTPClientCodeError) Error() string

Error implements error for HTTPClientCodeError.

func (*HTTPClientCodeError) Unwrap

func (h *HTTPClientCodeError) Unwrap() error

Unwrap implements errors.Unwrap.

type LocalFileClient

type LocalFileClient struct{}

LocalFileClient implements FileScheme for files on disk.

func (LocalFileClient) Fetch

func (lfs LocalFileClient) Fetch(_ context.Context, u *url.URL) (io.ReaderAt, error)

Fetch implements FileScheme.Fetch.

type MockScheme

type MockScheme struct {
	// scheme is the scheme name.
	Scheme string
	// contains filtered or unexported fields
}

MockScheme is a Scheme mock for testing.

func NewMockScheme

func NewMockScheme(scheme string) *MockScheme

NewMockScheme creates a new MockScheme with the given scheme name.

func (*MockScheme) Add

func (m *MockScheme) Add(host string, p string, content string)

Add adds a file to the MockScheme

func (*MockScheme) Fetch

func (m *MockScheme) Fetch(ctx context.Context, u *url.URL) (io.ReaderAt, error)

Fetch implements FileScheme.Fetch.

func (*MockScheme) NumCalled

func (m *MockScheme) NumCalled(u *url.URL) uint

NumCalled returns how many times a url has been looked up.

func (*MockScheme) SetErr

func (m *MockScheme) SetErr(err error, count int)

SetErr sets the error which is returned on the next count calls to Fetch.

type SchemeWithRetries

type SchemeWithRetries struct {
	Scheme FileScheme

	// DoRetry should return true to indicate the Fetch shall be retried.
	// Even if DoRetry returns true, BackOff can still determine whether to
	// stop.
	//
	// If DoRetry is nil, it will be retried if the BackOff agrees.
	DoRetry DoRetry

	// BackOff determines how often to retry and how long to wait between
	// each retry.
	BackOff backoff.BackOff
}

SchemeWithRetries wraps a FileScheme and automatically retries (with backoff) when Fetch returns a non-nil err.

func (*SchemeWithRetries) Fetch

func (s *SchemeWithRetries) Fetch(ctx context.Context, u *url.URL) (io.ReaderAt, error)

Fetch implements FileScheme.Fetch.

type Schemes

type Schemes map[string]FileScheme

Schemes is a map of URL scheme identifier -> implementation that can fetch a file for that scheme.

func (Schemes) Fetch

func (s Schemes) Fetch(ctx context.Context, u *url.URL) (File, error)

Fetch fetchs the file with the given `u`. `u.Scheme` is used to select the FileScheme via `s`.

If `s` does not contain a FileScheme for `u.Scheme`, ErrNoSuchScheme is returned.

func (Schemes) LazyFetch

func (s Schemes) LazyFetch(u *url.URL) (File, error)

LazyFetch returns a reader that will Fetch the file given by `u` when Read is called, based on `u`s scheme. See Schemes.Fetch for more details.

func (Schemes) Register

func (s Schemes) Register(scheme string, fs FileScheme)

Register registers a scheme identified by `scheme` to be `fs`.

type TFTPClient

type TFTPClient struct {
	// contains filtered or unexported fields
}

TFTPClient implements FileScheme for TFTP files.

func (*TFTPClient) Fetch

func (t *TFTPClient) Fetch(_ context.Context, u *url.URL) (io.ReaderAt, error)

Fetch implements FileScheme.Fetch.

type URLError

type URLError struct {
	URL *url.URL
	Err error
}

URLError is an error involving URLs.

func (*URLError) Error

func (s *URLError) Error() string

Error implements error.Error.

func (*URLError) Unwrap

func (s *URLError) Unwrap() error

Unwrap unwraps the underlying error.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL