grab

package module
v0.0.0-...-20d740a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 1, 2022 License: Apache-2.0 Imports: 21 Imported by: 0

README

grab

Documentation

Overview

Package grab provides a HTTP download manager implementation.

Get is the most simple way to download a file: 

resp, err := grab.Get("/tmp", "http://example.com/example.zip")
// ...

Get will download the given URL and save it to the given destination directory. The destination filename will be determined automatically by grab using Content-Disposition headers returned by the remote server, or by inspecting the requested URL path.

An empty destination string or "." means the transfer will be stored in the current working directory.

If a destination file already exists, grab will assume it is a complete or partially complete download of the requested file. If the remote server supports resuming interrupted downloads, grab will resume downloading from the end of the partial file. If the server does not support resumed downloads, the file will be retransferred in its entirety. If the file is already complete, grab will return successfully.

For control over the HTTP client, destination path, auto-resume, checksum validation and other settings, create a Client: 

client := grab.NewClient()
client.HTTPClient.Transport.DisableCompression = true

req, err := grab.NewRequest("/tmp", "http://example.com/example.zip")
// ...
req.NoResume = true
req.HTTPRequest.Header.Set("Authorization", "Basic YWxhZGRpbjpvcGVuc2VzYW1l")

resp := client.Do(req)
// ...

You can monitor the progress of downloads while they are transferring: 

client := grab.NewClient()
req, err := grab.NewRequest("", "http://example.com/example.zip")
// ...
resp := client.Do(req)

t := time.NewTicker(time.Second)
defer t.Stop()

for {
	select {
	case <-t.C:
		fmt.Printf("%.02f%% complete\n", resp.Progress())

	case <-resp.Done:
		if err := resp.Err(); err != nil {
			// ...
		}

		// ...
		return
	}
}

Index

Constants

View Source 
const (
	StatusEmpty    int64 = 0
	StatusStart    int64 = 1
	StatusStopped  int64 = 2
	StatusStopping int64 = 3
)

Variables

View Source 
var (
	// ErrBadLength indicates that the server response or an existing file does
	// not match the expected content length.
	ErrBadLength = errors.New("bad content length")

	// ErrBadChecksum indicates that a downloaded file failed to pass checksum
	// validation.
	ErrBadChecksum = errors.New("checksum mismatch")

	// ErrNoFilename indicates that a reasonable filename could not be
	// automatically determined using the URL or response headers from a server.
	ErrNoFilename = errors.New("no filename could be determined")

	// ErrNoTimestamp indicates that a timestamp could not be automatically
	// determined using the response headers from the remote server.
	ErrNoTimestamp = errors.New("no timestamp could be determined for the remote file")

	// ErrFileExists indicates that the destination path already exists.
	ErrFileExists = errors.New("file exists")
)
View Source 
var DefaultClient = NewClient()

DefaultClient is the default client and is used by all Get convenience functions.

View Source 
var ErrForbidden = errors.New("server returned 403 Forbidden")

Functions

func DividePart 

func DividePart(fileSize int64, last int) (int64, int64)

DividePart 根据文件大小计算分片数量和大小 fileSize: 文件大小 last: 分片大小(MB)

func FormatRangeOptions 

func FormatRangeOptions(opt *RangeOptions) string

func IsStatusCodeError 

func IsStatusCodeError(err error) bool

IsStatusCodeError returns true if the given error is of type StatusCodeError.

func LimitReadCloser 

func LimitReadCloser(r io.Reader, n int64) io.ReadCloser

Types

type BatchReq 

type BatchReq struct {
	Dst string
	Url string
}

type BatchResponse 

type BatchResponse struct {
	ResCh  <-chan *Response
	Cancel context.CancelFunc
}

func GetBatch 

func GetBatch(reqParams []BatchReq, opts ...DownloadOptionFunc) (*BatchResponse, error)

type Chunk 

type Chunk struct {
	sync.RWMutex
	Number    int   `json:"number,omitempty"`
	OffSet    int64 `json:"offSet,omitempty"`
	Size      int64 `json:"size,omitempty"`
	Done      bool  `json:"done,omitempty"`
	Completed int64 `json:"completed,omitempty"`
	// contains filtered or unexported fields
}

func SplitSizeIntoChunks 

func SplitSizeIntoChunks(totalBytes int64, partSizes ...int64) ([]*Chunk, int, error)

SplitSizeIntoChunks 根据文件大小和分片大小计算分片

type Client 

type Client struct {
	// HTTPClient specifies the http.Client which will be used for communicating
	// with the remote server during the file transfer.
	HTTPClient HTTPClient

	// UserAgent specifies the User-Agent string which will be set in the
	// headers of all requests made by this client.
	//
	// The user agent string may be overridden in the headers of each request.
	UserAgent string

	// BufferSize specifies the size in bytes of the buffer that is used for
	// transferring all requested files. Larger buffers may result in faster
	// throughput but will use more memory and result in less frequent updates
	// to the transfer progress statistics. The BufferSize of each request can
	// be overridden on each Request object. Default: 32KB.
	BufferSize int
	// contains filtered or unexported fields
}

A Client is a file download client.

Clients are safe for concurrent use by multiple goroutines.

func NewClient 

func NewClient() *Client

NewClient returns a new file download Client, using default configuration.

func (*Client) Do 

func (c *Client) Do(req *Request) *Response

Do sends a file transfer request and returns a file transfer response, following policy (e.g. redirects, cookies, auth) as configured on the client's HTTPClient.

Like http.Get, Do blocks while the transfer is initiated, but returns as soon as the transfer has started transferring in a background goroutine, or if it failed early.

An error is returned via Response.Err if caused by client policy (such as CheckRedirect), or if there was an HTTP protocol or IO error. Response.Err will block the caller until the transfer is completed, successfully or otherwise.

func (*Client) DoBatch 

func (c *Client) DoBatch(ctx context.Context, opt *DownloadOptions, requests ...*Request) <-chan *Response

DoBatch executes all the given requests using the given number of concurrent workers. Control is passed back to the caller as soon as the workers are initiated.

If the requested number of workers is less than one, a worker will be created for every request. I.e. all requests will be executed concurrently.

If an error occurs during any of the file transfers it will be accessible via call to the associated Response.Err.

The returned Response channel is closed only after all of the given Requests have completed, successfully or otherwise.

func (*Client) DoChannel 

func (c *Client) DoChannel(reqch <-chan *Request, respch chan<- *Response, respNum *int64)

DoChannel executes all requests sent through the given Request channel, one at a time, until it is closed by another goroutine. The caller is blocked until the Request channel is closed and all transfers have completed. All responses are sent through the given Re sponse channel as soon as they are received from the remote servers and can be used to track the progress of each download.

Slow Response receivers will cause a worker to block and therefore delay the start of the transfer for an already initiated connection - potentially causing a server timeout. It is the caller's responsibility to ensure a sufficient buffer size is used for the Response channel to prevent this.

If an error occurs during any of the file transfers it will be accessible via the associated Response.Err function.

func (*Client) GetProgress 

func (c *Client) GetProgress(reqParams []BatchReq) (int64, int64, error)

func (*Client) WithDownloadOptions 

func (c *Client) WithDownloadOptions(optsFunc ...DownloadOptionFunc) *Client

type DownloadFile 

type DownloadFile struct {
	Url      string `json:"url" yaml:"url"`
	FileName string `json:"fileName" yaml:"fileName"`
}

type DownloadOptionFunc 

type DownloadOptionFunc func(opt *DownloadOptions)

func WithDisablePartDownload 

func WithDisablePartDownload(disablePartDownload bool) DownloadOptionFunc

func WithDownloadWorkers 

func WithDownloadWorkers(workers int) DownloadOptionFunc

func WithPartSize 

func WithPartSize(partSize int64) DownloadOptionFunc

func WithRetryTimes 

func WithRetryTimes(retryTimes int) DownloadOptionFunc

func WithWriteHook 

func WithWriteHook(hook func(n int64)) DownloadOptionFunc

type DownloadOptions 

type DownloadOptions struct {
	// contains filtered or unexported fields
}

type DownloadedBlock 

type DownloadedBlock struct {
	From      int64 `json:"f,omitempty"`
	To        int64 `json:"t,omitempty"`
	Completed int64 `json:"c,omitempty"`
}

type Downloader 

type Downloader struct {
	// contains filtered or unexported fields
}

func NewDownloader 

func NewDownloader(path string, files []DownloadFile, opts ...DownloadOptionFunc) *Downloader

func (*Downloader) BytesPerSecond 

func (d *Downloader) BytesPerSecond() float64

func (*Downloader) Done 

func (d *Downloader) Done() bool

func (*Downloader) Err 

func (d *Downloader) Err() error

func (*Downloader) IsRunning 

func (d *Downloader) IsRunning() bool

func (*Downloader) PauseDownload 

func (d *Downloader) PauseDownload() error

func (*Downloader) StartDownload 

func (d *Downloader) StartDownload() error

func (*Downloader) Wait 

func (d *Downloader) Wait()

func (*Downloader) WithBytesPerSecondHook 

func (d *Downloader) WithBytesPerSecondHook(hook func(bps float64) bool)

func (*Downloader) WithErrHook 

func (d *Downloader) WithErrHook(hook func(err error))

func (*Downloader) WithProgressHook 

func (d *Downloader) WithProgressHook(hook func(current, total int64, err error))

type FileWriter 

type FileWriter struct {
	io.WriteCloser
	// contains filtered or unexported fields
}

func NewFileWriter 

func NewFileWriter(w io.WriteCloser, res *Results) *FileWriter

func (*FileWriter) Write 

func (w *FileWriter) Write(p []byte) (int, error)

type HTTPClient 

type HTTPClient interface {
	Do(req *http.Request) (*http.Response, error)
}

HTTPClient provides an interface allowing us to perform HTTP requests.

type Hook 

type Hook func(*Response) error

A Hook is a user provided callback function that can be called by grab at various stages of a requests lifecycle. If a hook returns an error, the associated request is canceled and the same error is returned on the Response object.

Hook functions are called synchronously and should never block unnecessarily. Response methods that block until a download is complete, such as Response.Err, Response.Cancel or Response.Wait will deadlock. To cancel a download from a callback, simply return a non-nil error.

type Jobs 

type Jobs struct {
	Url      string
	FilePath string
	Chunk    *Chunk
	Range    string
}

type LimitedReadCloser 

type LimitedReadCloser struct {
	io.LimitedReader
}

func (*LimitedReadCloser) Close 

func (lc *LimitedReadCloser) Close() error

type MultiCPInfo 

type MultiCPInfo struct {
	Size             int64                    `json:"s,omitempty"`
	LastModified     string                   `json:"lm,omitempty"`
	DownloadedBlocks map[int]*DownloadedBlock `json:"dbs,omitempty"`
}

type RangeOptions 

type RangeOptions struct {
	Start int64 `json:"start,omitempty"`
	End   int64 `json:"end,omitempty"`
}

type RateLimiter 

type RateLimiter interface {
	WaitN(ctx context.Context, n int) (err error)
}

RateLimiter is an interface that must be satisfied by any third-party rate limiters that may be used to limit download transfer speeds.

A recommended token bucket implementation can be found at https://godoc.org/golang.org/x/time/rate#Limiter.

type Request 

type Request struct {
	// Label is an arbitrary string which may used to label a Request with a
	// user friendly name.
	Label string

	// Tag is an arbitrary interface which may be used to relate a Request to
	// other data.
	Tag interface{}

	// HTTPRequest specifies the http.Request to be sent to the remote server to
	// initiate a file transfer. It includes request configuration such as URL,
	// protocol version, HTTP method, request headers and authentication.
	HTTPRequest *http.Request

	// Filename specifies the path where the file transfer will be stored in
	// local storage. If Filename is empty or a directory, the true Filename will
	// be resolved using Content-Disposition headers or the request URL.
	//
	// An empty string means the transfer will be stored in the current working
	// directory.
	Filename string

	// SkipExisting specifies that ErrFileExists should be returned if the
	// destination path already exists. The existing file will not be checked for
	// completeness.
	SkipExisting bool

	// NoResume specifies that a partially completed download will be restarted
	// without attempting to resume any existing file. If the download is already
	// completed in full, it will not be restarted.
	NoResume bool

	// NoStore specifies that grab should not write to the local file system.
	// Instead, the download will be stored in memory and accessible only via
	// Response.Open or Response.Bytes.
	NoStore bool

	// NoCreateDirectories specifies that any missing directories in the given
	// Filename path should not be created automatically, if they do not already
	// exist.
	NoCreateDirectories bool

	// IgnoreBadStatusCodes specifies that grab should accept any status code in
	// the response from the remote server. Otherwise, grab expects the response
	// status code to be within the 2XX range (after following redirects).
	IgnoreBadStatusCodes bool

	// IgnoreRemoteTime specifies that grab should not attempt to set the
	// timestamp of the local file to match the remote file.
	IgnoreRemoteTime bool

	// Size specifies the expected size of the file transfer if known. If the
	// server response size does not match, the transfer is cancelled and
	// ErrBadLength returned.
	Size int64

	// BufferSize specifies the size in bytes of the buffer that is used for
	// transferring the requested file. Larger buffers may result in faster
	// throughput but will use more memory and result in less frequent updates
	// to the transfer progress statistics. If a RateLimiter is configured,
	// BufferSize should be much lower than the rate limit. Default: 32KB.
	BufferSize int

	// RateLimiter allows the transfer rate of a download to be limited. The given
	// Request.BufferSize determines how frequently the RateLimiter will be
	// polled.
	RateLimiter RateLimiter

	// BeforeCopy is a user provided callback that is called immediately before
	// a request starts downloading. If BeforeCopy returns an error, the request
	// is cancelled and the same error is returned on the Response object.
	BeforeCopy Hook

	// AfterCopy is a user provided callback that is called immediately after a
	// request has finished downloading, before checksum validation and closure.
	// This hook is only called if the transfer was successful. If AfterCopy
	// returns an error, the request is canceled and the same error is returned on
	// the Response object.
	AfterCopy Hook

	DownloadOptions *DownloadOptions

	PartInfo string
	// contains filtered or unexported fields
}

A Request represents an HTTP file transfer request to be sent by a Client.

func NewRequest 

func NewRequest(dst, urlStr string) (*Request, error)

NewRequest returns a new file transfer Request suitable for use with Client.Do.

func (*Request) Context 

func (r *Request) Context() context.Context

Context returns the request's context. To change the context, use WithContext.

The returned context is always non-nil; it defaults to the background context.

The context controls cancelation.

func (*Request) SetChecksum 

func (r *Request) SetChecksum(h hash.Hash, sum []byte, deleteOnError bool)

SetChecksum sets the desired hashing algorithm and checksum value to validate a downloaded file. Once the download is complete, the given hashing algorithm will be used to compute the actual checksum of the downloaded file. If the checksums do not match, an error will be returned by the associated Response.Err method.

If deleteOnError is true, the downloaded file will be deleted automatically if it fails checksum validation.

To prevent corruption of the computed checksum, the given hash must not be used by any other request or goroutines.

To disable checksum validation, call SetChecksum with a nil hash.

func (*Request) URL 

func (r *Request) URL() *url.URL

URL returns the URL to be downloaded.

func (*Request) WithContext 

func (r *Request) WithContext(ctx context.Context) *Request

WithContext returns a shallow copy of r with its context changed to ctx. The provided ctx must be non-nil.

type Response 

type Response struct {
	// The Request that was submitted to obtain this Response.
	Request *Request

	// HTTPResponse represents the HTTP response received from an HTTP request.
	//
	// The response Body should not be used as it will be consumed and closed by
	// grab.
	HTTPResponse *http.Response

	// Filename specifies the path where the file transfer is stored in local
	// storage.
	Filename string

	// Start specifies the time at which the file transfer started.
	Start time.Time

	// End specifies the time at which the file transfer completed.
	//
	// This will return zero until the transfer has completed.
	End time.Time

	// CanResume specifies that the remote server advertised that it can resume
	// previous downloads, as the 'Accept-Ranges: bytes' header is set.
	CanResume bool

	// DidResume specifies that the file transfer resumed a previously incomplete
	// transfer.
	DidResume bool

	// Done is closed once the transfer is finalized, either successfully or with
	// errors. Errors are available via Response.Err
	Done chan struct{}

	DownloadParts bool

	// Chunks download file split chunks
	Chunks []*Chunk

	// MultiCPInfo multi parts download checkpoint info
	MultiCPInfo *MultiCPInfo
	// contains filtered or unexported fields
}

Response represents the response to a completed or in-progress download request.

A response may be returned as soon a HTTP response is received from a remote server, but before the body content has started transferring.

All Response method calls are thread-safe.

func Get 

func Get(dst, urlStr string) (*Response, error)

Get sends a HTTP request and downloads the content of the requested URL to the given destination file path. The caller is blocked until the download is completed, successfully or otherwise.

An error is returned if caused by client policy (such as CheckRedirect), or if there was an HTTP protocol or IO error.

For non-blocking calls or control over HTTP client headers, redirect policy, and other settings, create a Client instead.

func (*Response) Bytes 

func (c *Response) Bytes() ([]byte, error)

Bytes blocks the calling goroutine until the underlying file transfer is completed and then reads all bytes from the completed tranafer. If Request.NoStore was enabled, the bytes will be read from memory.

If an error occurred during the transfer, it will be returned.

func (*Response) BytesComplete 

func (c *Response) BytesComplete() int64

BytesComplete returns the total number of bytes which have been copied to the destination, including any bytes that were resumed from a previous download.

func (*Response) BytesPerSecond 

func (c *Response) BytesPerSecond() float64

BytesPerSecond returns the number of bytes per second transferred using a simple moving average of the last five seconds. If the download is already complete, the average bytes/sec for the life of the download is returned.

func (*Response) Cancel 

func (c *Response) Cancel() error

Cancel cancels the file transfer by canceling the underlying Context for this Response. Cancel blocks until the transfer is closed and returns any error - typically context.Canceled.

func (*Response) Duration 

func (c *Response) Duration() time.Duration

Duration returns the duration of a file transfer. If the transfer is in process, the duration will be between now and the start of the transfer. If the transfer is complete, the duration will be between the start and end of the completed transfer process.

func (*Response) ETA 

func (c *Response) ETA() time.Time

ETA returns the estimated time at which the the download will complete, given the current BytesPerSecond. If the transfer has already completed, the actual end time will be returned.

func (*Response) Err 

func (c *Response) Err() error

Err blocks the calling goroutine until the underlying file transfer is completed and returns any error that may have occurred. If the download is already completed, Err returns immediately.

func (*Response) IsComplete 

func (c *Response) IsComplete() bool

IsComplete returns true if the download has completed. If an error occurred during the download, it can be returned via Err.

func (*Response) Open 

func (c *Response) Open() (io.ReadCloser, error)

Open blocks the calling goroutine until the underlying file transfer is completed and then opens the transferred file for reading. If Request.NoStore was enabled, the reader will read from memory.

If an error occurred during the transfer, it will be returned.

It is the callers responsibility to close the returned file handle.

func (*Response) Progress 

func (c *Response) Progress() float64

Progress returns the ratio of total bytes that have been downloaded. Multiply the returned value by 100 to return the percentage completed.

func (*Response) Size 

func (c *Response) Size() int64

Size returns the size of the file transfer. If the remote server does not specify the total size and the transfer is incomplete, the return value is -1.

func (*Response) Wait 

func (c *Response) Wait()

Wait blocks until the download is completed.

type Results 

type Results struct {
	PartNumber int
	// contains filtered or unexported fields
}

type StatusCodeError 

type StatusCodeError int

StatusCodeError indicates that the server response had a status code that was not in the 200-299 range (after following any redirects).

func (StatusCodeError) Error 

func (err StatusCodeError) Error() string

Source Files

View all Source files

Directories

Path Synopsis
pkg
bps
Package bps provides gauges for calculating the Bytes Per Second transfer rate of data streams.
 Package bps provides gauges for calculating the Bytes Per Second transfer rate of data streams.
grabtest

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.